Pedestrian re-identification method based on heterogeneous graph semantic differentiation, computer program product and terminal

Abstract

The invention discloses a pedestrian re-identification method based on heterogeneous graph semantic differentiation, a computer program product and a terminal, and belongs to the technical field of image processing, and the method comprises the following steps: taking a feature block as a node branch and an edge relationship between nodes as an edge branch to establish a heterogeneous graph, and aggregating the nodes of the same or similar classification labels, the semantic association is reflected; performing aggregation processing according to the feature representation of the adjacent nodes, and screening node pairs with closer semantics according to the affinity attribute values; performing aggregation processing on the node features according to the correlation to obtain cross-space-time aggregation features; and executing the Re-ID task according to the cross-space-time aggregation feature. According to the method, the semantic correlation between the nodes is measured, noise, non-target semantics and target semantics are successfully distinguished, on the premise that effective information is not sacrificed, the target semantics are aggregated to the maximum extent, the feasibility of semantic distinguishing and information retention is guaranteed, and the semantic perception ability of an occluded target is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a pedestrian re-identification method based on semantic differentiation of heterogeneous graphs, a computer program product and a terminal. Background Art

[0002] With the rapid development of deep learning and the derivation of large-scale data sets, the application effect of pedestrian re-identification has almost achieved good detection efficiency in the identification of complete pedestrians, but the specific recognition effect will vary greatly due to different application scenarios. Generally speaking, there are two different situations of pedestrian occlusion. One is that pedestrians are inevitably temporarily partially occluded by obstacles (Accidental Partial Occlusion, APO), and the other is that the visible range in the visual domain is continuously occluded by objects with the same moving state and speed (such as vehicles or pedestrians) (Continuous Partial Occlusion, CPO). In particular, for the overlapping parts of the occluded parts, the overlapping features of different pedestrians cause more serious semantic confusion between pedestrian identities. Therefore, it is necessary to distinguish the semantics under occlusion.

[0003] The occlusion of pedestrians has become a major bottleneck affecting recognition performance. To solve this problem, existing research focuses on the discarding of occluded features and occlusion completion. The former discards occluded features by suppressing or reducing the extraction of occluded features, and extracts the feature representation of the visible part using keyframe extraction, local feature capture or attention mechanism. However, the challenge of this method is that while ignoring the occluded area, it may lose the key features for identifying the identity of pedestrians, which is very unfriendly and even destructive for the feature extraction of pedestrians with occlusion. Some scholars use clustering methods to classify images and extract keyframe images with typical features to skip the occluded images, and the remaining images are discarded. Although the occlusion is avoided to a certain extent, the original spatiotemporal information will be weakened or even interrupted, resulting in the masking of its spatial information. The attention mechanism is widely used in pedestrian re-identification, transforming the features of the image from the two dimensions of space and pipeline. Obtain a feature map with higher robustness features. Although the impact of the occluded part is reduced to a certain extent, the lack of mining of temporal information is contrary to the spatiotemporal feature extraction of the video-based pedestrian re-identification task. The latter feature completion strategy combines the human body information structure and local feature distribution to model the correlation between the occluded area and the non-occluded area, and realizes the prediction and repair of the occluded part. The spatiotemporal attention model extracts the occluded attention descriptor and affinity attributes in the video context. However, they rarely explore the feature semantic attributes existing in the local features, and the matching between the noise semantics may lose the key guidance for the pedestrian identity. Occlusion completion relies on the use of rich spatiotemporal information for image / feature completion. Based on 3D convolution, spatiotemporal attention and other technologies, the semantics of the occluded area is restored by learning the contextual information of the occluded area and the prior knowledge of the human body structure.

[0004] In summary, for the above solutions to the pedestrian occlusion problem, it is necessary to weigh the coordination relationship between the lost information and the re-identification performance for different degrees of occlusion. No type of solution can successfully distinguish the confusing semantic features and aggregate the target semantics to the greatest extent. They always balance between losing effective information and improving efficiency, but do not try to distinguish features and aggregate representations while ensuring effective information. Therefore, it is necessary to enhance the model's semantic perception ability of occluded targets from the perspective of semantic distinction. Summary of the invention

[0005] The purpose of the present invention is to overcome the problems of the prior art and provide a pedestrian re-identification method based on semantic differentiation of heterogeneous graphs, a computer program product and a terminal.

[0006] The object of the present invention is to achieve the following technical solution: a pedestrian re-identification method based on semantic differentiation of heterogeneous graphs, the method comprising the following steps:

[0007] Extract features from the original pedestrian image to obtain a feature map, and divide the feature map into multiple feature blocks;

[0008] A heterogeneous graph is established with feature blocks as node branches and edge relationships between nodes as edge branches. The edge relationships are obtained by mapping the affinity metrics of adjacent feature blocks, and nodes with the same or similar classification labels are aggregated.

[0009] Update the feature representation of the node, and then perform aggregation processing based on the feature representation of adjacent nodes; abstract the occlusion mapping relationship between nodes and perform fusion processing to obtain affinity attribute values, and select node pairs with closer semantics based on affinity attribute values;

[0010] Measure the semantic relevance between nodes, calculate the relevance between different nodes, aggregate node features based on the relevance, and obtain aggregate features across time and space;

[0011] The pedestrian re-identification task is performed based on the aggregated features across time and space to obtain the prediction results.

[0012] In one example, the affinity measure is calculated based on the spatial offset factor and the cosine similarity between adjacent feature blocks.

[0013] In one example, the calculation expression of the node classification label is:

[0014]

[0015] Among them, P represents the probability that the feature block belongs to a certain category; x p 、x q Represent different feature blocks respectively; Y p , Y q Represent the feature blocks x p , feature block x q A random variable; c represents the category; E represents the set of feature block pairs; e represents the affinity measure.

[0016] In one example, the calculation expression for performing aggregation processing based on the feature representation of adjacent nodes is:

[0017]

[0018] Among them, S v Represents the feature vector of node v after aggregation; meanpooling represents the feature aggregation operation; represents the set of k-th order neighbor nodes of node u under the considered meta-path φ; Representing a collection All nodes corresponding to index i in x; i Representing a collection The original feature vector of the i-th node in .

[0019] In one example, the calculation expression of the affinity attribute value is:

[0020]

[0021] Among them, α u,v Represents the affinity attribute value between node u and node v; ε v , ε u , ε k They represent the meta-paths of node u, node v, and node k respectively; LeakRelu represents the RLU activation function; and E represents the set of feature block pairs.

[0022] In one example, the measure expression of the semantic relevance between nodes is:

[0023]

[0024] in, Indicates that under a specific operation Φ, the node x feature f x , node y feature f y The semantic correlation measure between them; s represents the background noise; σ represents the activation function; d represents the target semantic feature; and n represents the non-target semantic noise.

[0025] In one example, the calculation expression of the correlation between different nodes is:

[0026]

[0027] in, Indicates that under a specific operation Φ, the feature f of node x x , node y feature f y The correlation between Indicates that under a specific operation Φ, the node x feature f x , node y feature f y Semantic correlation measurement between i Represents the set of nodes associated with index i.

[0028] In one example, the aggregation processing expression of node features is:

[0029]

[0030] Where H represents the aggregation feature; c i,r represents the normalization constant; Represents the weight matrix used in the lth layer of multi-layer convolution, determined according to the correlation between nodes; represents the feature representation of node i in the lth convolutional layer; j represents the number of iterations of traversing the set N.

[0031] It should be further explained that the technical features corresponding to the above examples can be combined or replaced with each other to form a new technical solution.

[0032] The present invention also includes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the pedestrian re-identification method based on heterogeneous graph semantic differentiation formed by any one of the above examples or a combination of multiple examples.

[0033] The present invention also includes a storage medium on which computer instructions are stored. When the computer instructions are executed, the steps of the pedestrian re-identification method based on heterogeneous graph semantic differentiation formed by any one of the above examples or multiple examples are executed.

[0034] The present invention also includes a terminal, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor executes the computer instructions, the steps of the pedestrian re-identification method based on heterogeneous graph semantic distinction formed by any one or more of the above examples are executed.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention establishes a heterogeneous graph with feature blocks as node branches and edge relationships between nodes as edge branches. The edge branches extract affinity clues from patch feature blocks between adjacent feature blocks to completely map the occlusion relationship in the original image. The node branches can dynamically model spatial semantic relationships and aggregate nearby feature blocks with similar or identical attributes to reflect semantic associations. Some original feature units that can express semantic attributes are then extracted and aggregated based on spatial distribution. The affinity attributes reflect the occlusion relationship between semantics. Then, based on the consistent distribution of semantics in the sequence, a cross-temporal and spatial aggregation strategy for semantics with the same attributes is designed to measure the semantic correlation between nodes, successfully distinguish noise features, non-target semantics and target semantics, and maximize the aggregation of target semantics without sacrificing effective information, thereby providing perception targets and ensuring the feasibility of semantic distinction and information retention. Under the guidance of semantic distribution, the target semantic features are effectively screened to enhance the semantic perception ability of occluded targets and ensure the recognition accuracy and effectiveness of the Re-ID task. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0038] Figure 1 A method flow chart provided for an example of the present invention;

[0039] Figure 2 A schematic diagram of the information transmission process between nodes provided by an example of the present invention;

[0040] Figure 3 A schematic diagram of heterogeneous graph sequence modeling provided for a preferred example of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the use of ordinal numbers (e.g., "first and second", "first to fourth", etc.) is to distinguish objects, and is not limited to this order, and cannot be understood as indicating or implying relative importance.

[0043] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0044] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] In one example, if Figure 1 As shown, a pedestrian re-identification method based on semantic differentiation of heterogeneous graphs includes the following steps:

[0046] S1: Extract features from the original pedestrian image to obtain a feature map, and divide the feature map into multiple feature blocks.

[0047] Given a sequence T, define it as having any occlusion scene, including APO and CPO, denoted as I t Represents the image data with occlusion observed at time t, and T represents the length of the sequence. The backbone network extracts the initial feature map from the original pedestrian image of each frame in the video sequence. Among them, H, W, and C represent the height, width, and channel size of the feature map, respectively. Among them, the backbone network can be a convolutional network, a degree-scale feature fusion network, a Transformer-based model, etc. In this example, the Transformer-based PVTv2-B2 is preferred. The feature block is a local area Patch extracted from the pedestrian image. It is a basic unit that can express specific semantic attributes and has spatial coordinate information. It shows different importance and semantic guidance in the original pedestrian image, so that the aggregation degree of the same semantics is inversely proportional to the spatial deviation.

[0048] S2: Create a graph structure, establish a heterogeneous graph with feature blocks as node branches (Node) and edge relationships between nodes as edge branches (Edge). The edge relationships are obtained according to the affinity measurement mapping of adjacent feature blocks, and nodes with the same or similar classification labels are aggregated.

[0049] The present invention urgently needs a structure that can both distinguish outlier features and maximize effective features. In traditional schemes, there are obstacles to information transfer between local features. The iterative process of the Graph Convolutional Network (GCN) fully considers the abstraction of local features and the transmission of global information. Therefore, heterogeneous graphs are the most suitable structure for abstract features. For Re-ID tasks in complex occlusion scenarios, heterogeneous graphs can not only solve the problem of misalignment of outlier features in fixed segmentation schemes, but also the problems of multi-semantic distribution and occlusion relationship abstraction.

[0050] This example designs an Adaptive Heterogeneous Graph Generator (AHGG) to achieve occlusion relationship mapping and multi-semantic spatial distribution modeling. It consists of two branches, one is the edge branch, which extracts affinity clues from feature blocks between adjacent feature blocks to fully map the occlusion relationship in the original image; the other is the node branch, which can dynamically model spatial semantic relationships and aggregate nearby feature blocks with the same attributes. Then, the edge relationship and node classification relationship are fused to form a heterogeneous graph for the Re-ID task with embedded semantic features. It should be noted that both the node branch and the edge branch act on the basic graph structure of the preliminary work. Execute the node branch and the edge branch separately to give the relevant properties of the heterogeneous graph.

[0051] S3: Update the feature representation of the node, and then perform aggregation processing based on the feature representation of adjacent nodes; abstract the occlusion mapping relationship between nodes and perform fusion processing to obtain affinity attribute values, and select node pairs with closer semantics based on the affinity attribute values.

[0052] Among them, the node branch is used to determine the semantic guidance role of the node in the entire heterogeneous graph. In the information transmission process of graph convolution, different semantic nodes express different result-driven processes, which can suppress non-target semantics to the greatest extent and amplify the benefits brought by target semantics. This example infers and expresses the semantics of nodes through the semantic transfer of adjacent nodes in the heterogeneous graph, that is, aggregates the feature representations of adjacent nodes.

[0053] Furthermore, the edge branch is used to determine whether there is a strict dependency relationship or occlusion semantics between nodes. In order to explore the connection relationship between nodes, that is, the high-order semantic attributes of the edges of heterogeneous graphs, it is necessary to abstract the attribute dependency process or occlusion mapping relationship between nodes and perform fusion processing to obtain affinity attribute values ​​that can represent the closeness or similarity of the relationship between nodes. The affinity attribute value is used as a standard or filter, that is, a threshold is designed according to the affinity attribute value. If it is greater than the threshold, it is considered to be severely occluded. The correlation between nodes connects nodes of the same semantic type, thereby achieving semantic guidance and selecting semantically closer or related node pairs.

[0054] For occlusion relationship mapping and node attribute analysis based on multiple semantic relationship abstraction, these two branches adopt complementary strategies in the model, focusing on the meaning construction of edges and nodes in heterogeneous graphs respectively. These two branches can be aggregated in parallel or sequentially.

[0055] S4: Measure the semantic correlation between nodes, calculate the correlation between different nodes, aggregate the node features according to the correlation, and obtain the aggregated features across time and space.

[0056] The present invention constructs a heterogeneous graph sequence for a video sequence, and constructs a heterogeneous graph containing multiple semantic and attribute relationships for each frame feature graph. The purpose of video-based character Re-ID is to abstract a discriminant feature that can uniquely identify the sequence. Due to the semantic diversity of nodes in heterogeneous graphs, the traditional GCN convolution mode cannot obtain image features. For heterogeneous graphs, graph nodes may have different semantics. Blind aggregation of node features is likely to cause more serious semantic confusion and lower recognition effect. Therefore, this example first clusters the nodes of contract class semantics, that is, the node-level feature set, and aggregates the node features of the same class into one class. The aggregation result will obtain the semantic information of the node, which consists of three categories: background noise, target semantic features, and non-target semantic features (semantic noise). Based on these semantic features, semantic attention values ​​are assigned to achieve robust feature extraction (aggregated features across time and space).

[0057] S5: Perform the pedestrian re-identification task based on the aggregated features across time and space to obtain the prediction results.

[0058] Optionally, the cross-temporal and spatial aggregated features are predicted through a fully connected layer or a classification layer, and the Re-ID task is performed to obtain the prediction results of pedestrian re-identification.

[0059] The present invention proposes a semantic distinction method for heterogeneous graph sequences. Aiming at the existing semantic confusion, a heterogeneous graph is designed to extract heterogeneous visual features to uniformly solve such problems. First, the semantic distribution and topological information are deeply analyzed and modeled, and the noise features, target semantic features and non-target semantic features are distinguished for the occluded image, and a heterogeneous graph with different semantic nodes and occlusion mapping edges is constructed. The target semantic aggregation strategy improves the model's perception of positive semantics. Specifically, the present invention first designs an adaptive heterogeneous graph generator to distinguish multiple semantics according to the spatial deviation represented by the semantic distribution. AHGG extracts some original feature units that can express semantic attributes and aggregates them based on spatial distribution. The affinity attribute reflects the occlusion relationship between semantics. Then, based on the consistent distribution of semantics in the sequence, a cross-temporal and spatial aggregation strategy for semantics with the same attributes is designed. Without sacrificing effective information, the target semantics are aggregated to the maximum extent, providing the model with a perceptual target, ensuring the feasibility of semantic distinction and information retention, and cleaning the target semantic features under the guidance of semantic distribution to enhance representation. The present invention has been experimentally verified and analyzed on the authoritative datasets Mars and Occluded-DukeMTMC-ReID. The results show that the method proposed in the present invention has achieved better performance under various experimental settings, verifying the effectiveness of the method in distinguishing semantic structures in pedestrian re-identification tasks in complex scenarios.

[0060] In one example, the affinity measure is calculated based on the spatial bias factor and the cosine similarity (normalized inner product) between adjacent feature blocks. The calculation expression of the affinity measure is:

[0061]

[0062] K = arctan(C*(pp′) η (qq′) 1-η )

[0063] Among them, e(x p,q ,x p′,q′ ) represents the feature block x p,q With feature block x p′,q′ The affinity measure between them; K represents the adjacent spatial bias factor; p, q represent the spatial parameter information (coordinate position) of the feature block; p', q' represent the spatial parameter information of another feature block; C represents the scale factor; η represents the weight used to adjust the influence of the p and q coordinates. The present invention introduces the spatial bias factor K in affinity extraction, which can fully distinguish the semantic spatial meaning between blocks, thereby improving the recognition ability of occlusion and spatial relationship. The edge relationship between the feature blocks is mapped according to the affinity measure, and the edge relationship is represented as the connection between nodes in the heterogeneous graph.

[0064] In one example, the calculation expression of the node classification label is:

[0065]

[0066] Among them, P represents the probability that the feature block belongs to a certain category; x p 、x q Respectively represent different feature blocks; Y p , Y q Represent the feature blocks x p , feature block x q A random variable; c represents the category; E represents the set of feature block pairs; e represents the affinity measure.

[0067] The heterogeneous graph construction process of the present invention can be understood as an improvement on the official base graph (a pre-constructed pedestrian image collection used for retrieval and matching in the Re-ID task) classification scheme. A spatial bias scheme is added, and then a label propagation scheme containing a spatial bias factor is adopted to integrate the original cosine distance into the attribute information and spatial information of the block. Feature blocks with the same or similar semantic labels are aggregated together to form a tight community or cluster, which reflects the process of semantic attribute distribution to a certain extent.

[0068] In one example, updating the feature representation of a node and then performing aggregation processing based on the feature representations of adjacent nodes includes:

[0069] Given a heterogeneous graph, defined as Where V = {v 1 ,v 2 ,···,v m}, represents a set of m graph nodes with background noise, target semantic features or non-target semantic features; ε = {ε 1 ,ε 2 ,···,ε n}, is an edge set with n relationships between m nodes. For each heterogeneous graph node v, it includes k related meta-paths The meta-path is the information source of all attribute factors that connect the generated nodes and can serve as the basis for transferring semantic features. Figure 2 As shown in (a), the semantic information of adjacent nodes is aggregated to represent the semantics of the nodes. The summary calculation method (node ​​representation update method) is as follows:

[0070]

[0071] in, represents the updated node feature representation; Xv and x u Respectively represent the characteristics of node v and its neighbor node u; N v represents the set of neighbor nodes of node v; σ represents the activation function; W φ represents the weight matrix; b φ Represents the bias term.

[0072] Furthermore, for the semantically driven process of each meta-path generation, not only neighbor node relationships but also adjacent relationships are aggregated, and the semantic aggregation expression is calculated as follows:

[0073]

[0074] Among them, S v It represents the feature vector (semantic attribute value) aggregated during the transmission of semantic information of all neighbor nodes v and heterogeneous edge information; meanpooling represents the feature aggregation operation; represents the set of k-th order neighbor nodes of node u under the considered meta-path φ; Representing a collection All nodes corresponding to index i in x; i Representing a collection The original feature vector of the i-th node in .

[0075] In one example, the occlusion mapping relationship between nodes is abstracted and fused to obtain affinity attribute values. Node pairs with closer semantics are selected based on the affinity attribute values, including:

[0076] The attribute dependency process or occlusion mapping relationship between abstract nodes is calculated as follows:

[0077]

[0078] Among them, e i,j represents the attribute of the edge between node i and node j; N represents the set of adjacent nodes of node i; W i represents the weight matrix of node i; b i Represents the bias term of node i. This expression is the calculation formula for the edge relationship between nodes. The value of the edge can be understood as the size of the dependency value between nodes, and its size also reflects the occlusion relationship of the nodes.

[0079] Furthermore, if Figure 2 As shown in (b), the relationship between nodes is connected by calculating the affinity attribute value between nodes, and then fused to obtain the affinity attribute value belonging to, which plays a screening role in the semantic guidance process. Specifically, the calculation expression of the affinity attribute value is:

[0080]

[0081] Among them, α u,v Represents the affinity attribute value between node u and node v; ε v , ε u , ε k They represent the meta-paths of node u, node v, and node k respectively; LeakRelu represents the RLU activation function.

[0082] In one example, the calculated cross-temporal and spatial aggregate features include:

[0083] First, the attention mechanism is used to obtain attention from the occlusion case, and the multi-semantic graph aggregated in the semantic case is convolved to extract the features of the single frame image; the extracted features have various semantic relationships. In this step, the frame-level heterogeneous graph becomes a feature set containing only three semantic features, namely background noise, target semantic features, and non-target semantic noise. In order to aggregate each node of the heterogeneous graph and measure the correlation with adjacent nodes, the semantic correlation between nodes needs to be measured, and the calculation expression is:

[0084]

[0085] in, Indicates that under a specific operation Φ, the node x feature f x , node y feature f y The semantic correlation measure between them; s represents background noise; d represents target semantic features; and n represents non-target semantic noise.

[0086] Furthermore, the importance between different node pairs is calculated, and the softmax value of each connection relationship is recalculated. The softmax value is composed of the elements of the adjacency matrix, and the weight of each connection relationship is recalculated using the softmax function:

[0087]

[0088] in, Indicates that under a specific operation φ, the feature f of node x x , node y feature f y The correlation between them.

[0089] Finally, the semantic nodes of the same category are aggregated, and the multi-layer convolution process is calculated as follows:

[0090]

[0091] Where H represents the aggregation feature; c i,r represents the normalization constant; Represents the weight matrix used in the lth layer of multi-layer convolution; represents the feature representation of node i in the lth convolutional layer; j represents the number of iterations of traversing the set N.

[0092] Due to the semantic diversity of nodes in heterogeneous graphs, the traditional GCN convolution mode cannot obtain image features. For heterogeneous graphs, graph nodes may have different semantics. Blind aggregation of node features can easily cause more serious semantic confusion and lower recognition effect. Therefore, this example first clusters the nodes of contract class semantics, that is, the node-level feature set, and aggregates the node features of the same class into one class. The aggregation result will obtain the semantic information of the node, which consists of three categories: semantic features / background noise / semantic noise. Based on these semantic features, semantic attention values ​​are assigned to achieve robust feature extraction, and then a syntactic semantic aggregation strategy across time and space is created for the same semantic attributes to provide high-order semantic information for the model (the model performing the Re-ID task).

[0093] Combining the above examples, we can obtain a preferred example of the present invention, such as Figure 3 As shown, the method now includes the following steps:

[0094] S10: extracting features from the original pedestrian image to obtain a feature map, and dividing the feature map into multiple feature blocks;

[0095] S20: Establish a heterogeneous graph with feature blocks as node branches and edge relationships between nodes as edge branches, and aggregate nodes with the same or similar classification labels; the edge relationships are obtained by mapping the affinity metrics of adjacent feature blocks, and the affinity metrics are calculated based on the spatial bias factor and the cosine similarity between adjacent feature blocks;

[0096] S30: updating the feature representation of the node, and then performing aggregation processing according to the feature representation of the adjacent nodes; abstracting the occlusion mapping relationship between the nodes and performing fusion processing to obtain affinity attribute values, and selecting node pairs with closer semantics according to the affinity attribute values;

[0097] S40: measuring the semantic correlation between nodes, calculating the correlation between different nodes, and aggregating node features according to the correlation to obtain cross-temporal and spatial aggregate features;

[0098] S50: Perform a person re-identification task based on the cross-temporal and spatial aggregated features to obtain a prediction result.

[0099] The present invention also includes a pedestrian re-identification system based on heterogeneous graph semantic differentiation, which is a heterogeneous graph based semantic differentiation network (Heterogeneous Graph Based Semantic Differentiation Network, HGSD), which is implemented based on a spatiotemporal graph convolutional network and is used to perform the pedestrian re-identification method based on heterogeneous graph semantic differentiation formed by any one of the above examples or a combination of multiple examples. The system includes:

[0100] Feature extraction module: used to extract features from the original pedestrian image, obtain a feature map, and divide the feature map into multiple feature blocks;

[0101] A graph construction module, which is used to establish a heterogeneous graph with feature blocks as node branches and edge relationships between nodes as edge branches. The edge relationships are obtained according to affinity measurement mapping of adjacent feature blocks, and nodes with the same or similar classification labels are aggregated;

[0102] The spatiotemporal convolution module is used to update the feature representation of the node and then perform aggregation processing based on the feature representation of adjacent nodes; the occlusion mapping relationship between abstract nodes is fused to obtain affinity attribute values, and node pairs with closer semantics are screened based on the affinity attribute values; further, the module is also used to measure the semantic correlation between nodes and calculate the correlation between different nodes, and the node features are aggregated based on the correlation to obtain aggregated features across time and space.

[0103] The classification layer is used to output the aggregated feature maps across time and space to obtain the prediction results of pedestrian re-identification

[0104] The performance of the HGSD model in the pedestrian re-identification task is experimentally demonstrated:

[0105] 1.1 Datasets and evaluation metrics used

[0106] The performance of overall person Re-ID is evaluated on Mars. In order to further explore the adaptability of HGSD to occluded pedestrian scenes, the semantic feature confusion problem caused by dense crowds and small targets caused by the HGSD model of the present invention is deeply analyzed on the datasets occlded-duckmtmc and occlded-dukemtmc-reid.

[0107] Among them, Mars is the largest video-based benchmark, which contains 1261 pedestrian targets and 20751 video sequences taken by 6 cameras. iLIDS-VID takes 600 video sequences for 300 pedestrians on two non-overlapping cameras, each video contains 23 to 192 images, with an average of 73. It is worth noting that iLIDS-VID has serious occlusion in the aviation hall environment. Occlud-duckmtmc-reid is a reassembled benchmark based on occlud-dukemtmc, which contains 100% occluded query tracks. All tracks come from the gallery and query set of the original dataset, which means that there is always occlusion in track matching. It can fully verify the ability to handle occlusion problems. In general, the training set contains 1702 video clips of 702 identities, and the query set contains 661 clips of 661 identities.

[0108] (II) Experimental settings and related parameters

[0109] In this invention, the PyTorch framework is used to complete all experiments based on API 1.13.1. PVTv2-B2 is used as the backbone network to extract frame features of the video sequence, and then input video clips with a sampling interval P value of 6 are selected from the original video sequence, because the optimal value of P has been verified in many works. The size of the input image is set to 224*112, and each input image is enhanced by rotation and normalization. HGSD is trained for a total of 100 epochs, the initial learning rate is set to 0.0001, the decay rate is 0.1 every 50 epochs, and the batch size is set to 32.

[0110] (III) Comparative experiment between the method of the present invention and the authoritative method

[0111] As shown in Table 1, compared with three types of schemes such as feature completion and topological space analysis, HGSD is very competitive in the overall pedestrian Re-ID task. HGSD is 1% and 0.6% higher than the best performance (Top-1) on Mars and DukeMTMC-VideoReID, respectively. This result proves that the construction of semantic heterogeneous graphs can further summarize the high-level information representation of the original image. Secondly, compared with the convolutional scheme based on homogeneous graphs, the results are significantly improved, which further shows that heterogeneous graphs can better capture the diverse relationships in complex environments and improve the perceptual depth of the model.

[0112] Table 1 Performance comparison with DukeMTMC-Video ReID on Mars

[0113]

[0114] (IV) Ablation experiment and result analysis of this method

[0115] In order to explore the effectiveness of the features guided by each component in HGSD, the present invention conducts a series of ablation experiments on occlded-duke, occlded-duke-video and UESTC-SD. The present invention uses STGCN as the baseline and cross entropy as the loss function. The training is carried out while keeping the same parameter details as HGSD, and the results are verified at one or more orders of Edge and Node, respectively, and the results under different loss functions are discussed. In order to find the optimal value of the batch size, the present invention also finds that in some cases, a larger batch size value may be more effective.

[0116] In the ablation experiment, the present invention considers implementing only the first-order Edge and multi-order Node branches respectively. In these parts, as shown in Table 2, the present invention finds that the model can obtain the best results when the batch size is set to 64 at certain times, while other conditions remain unchanged. The present invention believes that this is related to the ability of HGSD to perceive model information and the degree of gradient descent, that is, more branches can better reflect the diversity of attribute features in the sample and avoid overfitting problems caused by insufficient range of perceived attributes.

[0117] Table 2 Comparison results with DukeMTMC-VideoReID on Mars

[0118]

[0119] Table 2 shows that HGSD can significantly improve the performance. Compared with the scheme that only uses the spatiotemporal features of homogeneous maps for feature aggregation, the Edge and Node branches improve by 9.217%, 9.381%, 8.71% and 12.007%, 13.564%, 11.879% on DukeMTMC-VideoReID, respectively. These improvements show that in complex occlusion scenes, abstracting and modeling occlusion relations and semantic distributions is more effective and in-depth than simple graph convolution schemes.

[0120] Then the influence of the number and order of the two branches on the structure is discussed. HGSD-E and HGSD-N respectively represent the aggregation of heterogeneous Edge or Node features, HGSD-EN represents the aggregation of Edge and Node features, HGSD-wo-Lce represents the HGSD model is trained without using the cross entropy loss function, HGSD-wo-La represents the HGSD model is trained without using the affinity calculation model, HGSD-wo-Lp represents the HGSD model is trained without using the triplet loss function, HGSD-wi-K1 represents the model trained with K1 as the spatial bias factor, HGSD-wi-K2 represents the model trained with K2 as the spatial bias factor, HGSD-EN represents the HGSD model with modified aggregation of edge to node features, HGSD-NE represents the HGSD model with modified aggregation of node to edge features, and HGSD-64 represents the HGSD model with the batch size set to 64. The results show that the two branches have a particularly significant improvement on DukeMTMC-VideoReID compared with other datasets. At the same time, the results also show that the gain effect of the Node branch is more obvious. The confused semantic information in the data set has a reverse guidance on the convolution process of the model, which is the core contradiction that affects the high-level semantic abstraction in the convolution process. The mapping process of the edge structure to the occlusion problem is the guidance of the convolution weight and feature transfer deviation. The result of the combined use of HGSD-NE and HGSD-NE is higher than that of a single branch.

[0121] An example of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the pedestrian re-identification method based on heterogeneous graph semantic differentiation formed by any one of the above examples or a combination of multiple examples. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0122] An example of the present invention also provides a storage medium, which has the same inventive concept as the pedestrian re-identification method based on heterogeneous graph semantic distinction formed by any one of the above examples or a combination of multiple examples, and stores computer instructions thereon, which, when executed, execute the steps of the pedestrian re-identification method based on heterogeneous graph semantic distinction formed by any one of the above examples or a combination of multiple examples.

[0123] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0124] An example of the present invention further provides a terminal, which has the same inventive concept as any example or a combination of multiple examples corresponding to the above-mentioned pedestrian re-identification method based on heterogeneous graph semantic distinction, including a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and the processor executes the steps of the above-mentioned pedestrian re-identification method based on heterogeneous graph semantic distinction when running the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0125] In one example, the terminal, i.e., the electronic device, is presented in the form of a general-purpose computing device, and the components of the electronic device may include but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0126] The storage unit stores a program code, and the program code can be executed by the processing unit, so that the processing unit performs the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit can perform the above-mentioned pedestrian re-identification method based on heterogeneous graph semantic distinction.

[0127] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0128] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0129] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0130] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0131] Through the above description, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method of the exemplary embodiment of the present application.

[0132] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A person re-identification method based on semantic differentiation of heterogeneous graphs, characterized in that: The following steps are involved: Extract features from the original pedestrian image to obtain a feature map, and divide the feature map into multiple feature blocks; A heterogeneous graph is established with feature blocks as node branches and edge relationships between nodes as edge branches. The edge relationships are obtained by mapping the affinity metrics of adjacent feature blocks, and nodes with the same or similar classification labels are aggregated. Update the feature representation of the node, and then perform aggregation processing based on the feature representation of adjacent nodes; abstract the occlusion mapping relationship between nodes and perform fusion processing to obtain affinity attribute values, and select node pairs with closer semantics based on affinity attribute values; Measure the semantic relevance between nodes, calculate the relevance between different nodes, aggregate node features based on the relevance, and obtain aggregate features across time and space; The pedestrian re-identification task is performed based on the aggregated features across time and space to obtain the prediction results.

2. The pedestrian re-identification method based on heterogeneous graph semantic differentiation according to claim 1 is characterized in that: The affinity measure is calculated based on the spatial bias factor and the cosine similarity between adjacent feature blocks.

3. The pedestrian re-identification method based on heterogeneous graph semantic differentiation according to claim 1 is characterized in that: The calculation expression of the node classification label is: Among them, P represents the probability that the feature block belongs to a certain category; x p 、x q Represent different feature blocks respectively; Y p , Y q Represent the feature blocks x p , feature block x q A random variable; c represents the category; E represents the set of feature block pairs; e represents the affinity measure.

4. The pedestrian re-identification method based on heterogeneous graph semantic differentiation according to claim 1 is characterized in that: The calculation expression for aggregation processing based on the feature representation of adjacent nodes is: Among them, S v Represents the feature vector of node v after aggregation; meanpooling represents the feature aggregation operation; represents the set of k-th order neighbor nodes of node u under the considered meta-path v; Representing a collection All nodes corresponding to index i in x; i Representing a collection The original feature vector of the i-th node in .

5. The pedestrian re-identification method based on heterogeneous graph semantic differentiation according to claim 1 is characterized in that: The calculation expression of the affinity attribute value is: Among them, α u,v Represents the affinity attribute value between node u and node v; v v 、v u 、v k They represent the meta-paths of node u, node v, and node k respectively; LeakRelu represents the RLU activation function; and E represents the set of feature block pairs.

6. The pedestrian re-identification method based on heterogeneous graph semantic differentiation according to claim 1 is characterized in that: The measurement expression of the semantic relevance between nodes is: in, Indicates that under a specific operation Φ, the node x feature f x , node y feature f y The semantic correlation measure between them; s represents the background noise; σ represents the activation function; d represents the target semantic feature; and n represents the non-target semantic noise.

7. The pedestrian re-identification method based on heterogeneous graph semantic differentiation according to claim 1 is characterized in that: The calculation expression of the correlation between different nodes is: in, Indicates that under a specific operation Φ, the feature f of node x x , node y feature f y The correlation between Indicates that under a specific operation Φ, the node x feature f x , node y feature f y Semantic correlation measure between i Represents the set of nodes associated with index i.

8. The pedestrian re-identification method based on heterogeneous graph semantic differentiation according to claim 1 is characterized in that: The aggregation processing expression of node features is: Where H represents the aggregation feature; c i,r represents the normalization constant; Represents the weight matrix used in the lth layer of multi-layer convolution, determined according to the correlation between nodes; represents the feature representation of node i in the lth convolutional layer; j represents the number of iterations of traversing the set N.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the pedestrian re-identification method based on heterogeneous graph semantic differentiation described in any one of claims 1 to 8 are implemented.

10. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, characterized in that: When the processor runs the computer instructions, the processor performs the steps of the pedestrian re-identification method based on heterogeneous graph semantic differentiation according to any one of claims 1 to 8.

Images