Object recognition method based on gated graph neural network and electronic device

Abstract

This application discloses an object re-identification method and electronic device based on a gated graph neural network. The method includes: extracting features from multiple sample images to obtain feature information of the sample images; constructing a knowledge graph based on the multiple sample images; importing the knowledge graph into the gated graph neural network and inputting the feature information of the sample images into the gated graph neural network for iterative training; inputting the feature information of the image to be identified into the trained gated graph neural network; and determining, based on the knowledge graph and weights, the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph, thereby re-identifying the object to be identified in the image. This method can realize the transfer of prototypical representations of categories between similar categories, accelerates the iterative training process, and enables the gated graph neural network to achieve high accuracy even with a small number of training samples.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an object re-identification method and electronic device based on a gated graph neural network. Background Technology

[0002] Re-identification technology primarily relies on computer recognition techniques to determine the presence of specific target objects in images or video sequences. Examples include pedestrian re-identification, vehicle re-identification, luggage re-identification, and other object re-identification, with broad application prospects. Taking vehicle re-identification records as an example, vehicle re-identification technology can be used for vehicle monitoring and tracking at traffic intersections. Re-identifying vehicles at multiple consecutive intersections also helps analyze vehicle travel routes and peak traffic hours in urban areas, thereby providing advance planning for people's travel, management planning for urban traffic, and further improving urban construction planning.

[0003] In the field of vehicle re-identification, the common approach is to directly use convolutional neural networks (CNNs) to extract features; or, to pre-segment the vehicle into several faces using segmentation models or regularization methods, and then use CNNs to extract features from each face; or, to divide the extracted feature maps into global features and local features, and then evaluate them using a loss function.

[0004] In actual recognition processes, even for the same type of vehicle, instances can vary significantly from different viewpoints; different vehicles of the same color can have similar appearances; and in environments with varying lighting, dust, or occlusion, the vehicle's appearance as seen in the camera can differ greatly from its original appearance. However, photos taken in real-world road conditions usually only capture one or a few sides of the vehicle, not its entire appearance. Yet, in actual re-identification tasks, the information obtained about the vehicle to be identified may include aspects that were not captured in the photos. Insufficient training samples for the model can introduce errors into our recognition accuracy.

[0005] Currently, the main methods for re-identification using graph network structures include two approaches: (1) First, a hierarchical spatial structure graph is constructed with global and local regions as nodes and two-layer relationships as edges. Under the constraint of metric learning, the graph convolutional neural network (GCN) module is used to learn the discriminative structural features. Then, the classification loss is combined with the metric learning loss; (2) Based on the graph signature method: the detailed description of a person (attribute labels) and visual features are aggregated into a graph, the graph is used for reasoning, and the person's signature graph is encoded as a more representative feature. However, both of these methods require a large number of samples for training, otherwise it is difficult to achieve a high accuracy. Summary of the Invention

[0006] In view of the above-mentioned problems in the prior art, this application provides an object re-identification method and electronic device based on a gated graph neural network, which can still achieve a high accuracy rate when the number of samples is small.

[0007] To address the above problems, the technical solution adopted in the embodiments of this application is as follows:

[0008] The first aspect of this application provides an object re-identification method based on a gated graph neural network, including:

[0009] Feature extraction is performed on multiple sample images to obtain their feature information; each sample image contains a different type of target object.

[0010] A knowledge graph is constructed based on multiple sample images; wherein the knowledge graph includes multiple nodes corresponding one-to-one with the sample images and edges connecting adjacent nodes, the nodes represent the type of the target object, and the nodes have corresponding weights and hidden states, and the edges represent the correlation between the nodes;

[0011] The knowledge graph is imported into the gated graph neural network, and the feature information of the sample image is input into the gated graph neural network for iterative training; and in each iteration, each node aggregates the hidden state of its neighboring nodes in the previous iteration and its own hidden state in the previous iteration based on the correlation, and updates its own hidden state and weights through the gated recurrent unit.

[0012] The feature information of the image to be identified is input into the trained gated graph neural network. Based on the knowledge graph and the weights, the gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph, so as to re-identify the object to be identified in the image.

[0013] In some embodiments, constructing a knowledge graph based on a plurality of the sample images includes:

[0014] The nodes in the knowledge graph are initialized based on the feature information of the sample images;

[0015] The weights of each node are randomly initialized, and the initial hidden state of each node is configured to be the same as its initial weight.

[0016] In some embodiments, importing the knowledge graph into a gated graph neural network and inputting the feature information of the sample image into the gated graph neural network for iterative training includes:

[0017] The knowledge graph is imported into a gated graph neural network, and the feature information of the sample image is input into the gated graph neural network. The second similarity between the sample image and the type represented by each node in the knowledge graph is determined by the gated graph neural network.

[0018] Based on the second similarity, the vector distance between each node is determined; wherein the vector distance characterizes the correlation between each node;

[0019] Based on the vector distance, each node is driven to aggregate the hidden state of its neighboring nodes from the previous iteration, and update the feature vector of each node itself.

[0020] The hidden state of each node is updated by a gated loop unit based on the feature vector and the hidden state of each node in the previous iteration.

[0021] The weights of the nodes are updated based on the updated hidden state.

[0022] In some embodiments, the step of importing the knowledge graph into a gated graph neural network and inputting the feature information of the sample image into the gated graph neural network for iterative training further includes:

[0023] Based on the second similarity and a predefined loss function, a loss value is determined; wherein the loss value characterizes the difference between the prediction result and the actual result of the gated graph neural network;

[0024] If the change in the loss value meets the first preset condition, the training of the gated graph neural network is determined to be complete.

[0025] In some embodiments, the step of inputting the feature information of the image to be identified into the trained gated graph neural network, and determining a first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph based on the knowledge graph and the weights, so as to re-identify the object to be identified in the image, includes:

[0026] Feature extraction is performed on the image to be identified to obtain its feature information;

[0027] The feature information of the image to be identified is input into the trained gated graph neural network. Based on the knowledge graph and the weights, the gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph.

[0028] If one of the multiple first similarities is greater than a first threshold, the object to be identified is determined to belong to the type represented by the corresponding node in the knowledge graph.

[0029] In some embodiments, the step of inputting the feature information of the image to be identified into the trained gated graph neural network, and determining a first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph based on the knowledge graph and the weights, so as to re-identify the object to be identified in the image, includes:

[0030] Feature extraction is performed on the image to be identified to obtain its feature information;

[0031] The feature information of multiple images to be identified is respectively input into the trained gated graph neural network, and the gated graph neural network outputs a similarity matrix of each image to be identified based on the knowledge graph and the weights; wherein, the similarity matrix includes the first similarity between the object to be identified in the image to be identified and the type represented by each node in the knowledge graph;

[0032] A third similarity is determined among the multiple similarity matrices, and if the third similarity meets a second preset condition, the objects to be identified in the multiple images to be identified are determined to be of the same type.

[0033] In some embodiments, the feature extraction of multiple sample images to obtain feature information of the sample images includes:

[0034] The sample images are preprocessed;

[0035] The feature map of the preprocessed sample image is obtained by extracting features from the preprocessed sample image using a feature extractor.

[0036] In some embodiments, constructing a knowledge graph based on a plurality of the sample images includes:

[0037] Based on the first number of sample images, construct the knowledge graph with the first number of nodes;

[0038] Once the weights of the first number of nodes have been updated, a second number of nodes are added to the knowledge graph based on the second number of sample images to form the knowledge graph with a third number of nodes.

[0039] A second aspect of this application provides an electronic device, including:

[0040] The extraction module is used to extract features from multiple sample images to obtain feature information of the sample images; wherein, the multiple sample images contain different types of target objects;

[0041] A construction module is used to construct a knowledge graph based on multiple sample images; wherein, the knowledge graph includes multiple nodes corresponding one-to-one with the sample images and edges connecting adjacent nodes, the nodes represent the type of the target object, and the nodes have corresponding weights and hidden states, and the edges represent the correlation between the nodes;

[0042] The training module is used to import the knowledge graph into the gated graph neural network and input the feature information of the sample image into the gated graph neural network for iterative training; and in each iteration, each node aggregates the hidden state of its neighboring nodes and the hidden state of the previous iteration based on the correlation, and updates its own hidden state and weights through the gated recurrent unit.

[0043] The recognition module is used to input the feature information of the image to be recognized into the trained gated graph neural network, and through the gated graph neural network based on the knowledge graph and the weights, determine the first similarity between the object to be recognized in the image and the type represented by each node in the knowledge graph, so as to re-recognize the object to be recognized in the image.

[0044] A third aspect of this application provides an electronic device, including at least a memory and a processor, wherein the memory stores an application program, and the processor implements the method described above when executing the application program in the memory.

[0045] The object re-identification method based on a gated graph neural network in this application constructs a knowledge graph based on multiple sample images. Nodes in the knowledge graph represent the type of the target object, and edges represent the correlation between nodes. During the iterative training of the gated graph neural network, each node aggregates the hidden states of its neighboring nodes from the previous iteration and its own hidden state from the previous iteration based on the correlation, and updates its own hidden state through a gated recurrent unit. The node weights are then updated based on the updated hidden states. This enables the transfer of prototypical representations of categories between similar categories, thereby accelerating the iterative training process. Even with a small number of training samples, the gated graph neural network can still achieve high accuracy. After the gated graph neural network is trained, accurate re-identification of the target object in the image to be identified can be performed based on the trained gated graph neural network. Attached Figure Description

[0046] Figure 1 This is a flowchart of the object re-identification method based on gated graph neural networks according to this application;

[0047] Figure 2 This is a schematic diagram of a scenario for the object re-identification method based on gated graph neural networks proposed in this application;

[0048] Figure 3 This is a flowchart of step S110 in the object re-identification method based on gated graph neural network of this application;

[0049] Figure 4 This is a flowchart of step S120 in the object re-identification method based on gated graph neural network of this application;

[0050] Figure 5 This is a flowchart of step S130 in the object re-identification method based on gated graph neural network of this application;

[0051] Figure 6 This is a flowchart of one embodiment of step S140 in the object re-identification method based on gated graph neural network of this application;

[0052] Figure 7 This is a flowchart of another embodiment of step S140 in the object re-identification method based on gated graph neural network of this application;

[0053] Figure 8 This is a structural block diagram of one embodiment of the electronic device of this application;

[0054] Figure 9 This is a structural block diagram of another embodiment of the electronic device of this application. Detailed Implementation

[0055] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0056] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0057] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0058] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0059] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.

[0060] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0061] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0062] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0063] This application provides an object re-identification method based on a gated graph neural network. This method can be applied to vehicle re-identification, pedestrian re-identification, luggage re-identification, or other object re-identification. The object type is not limited here.

[0064] The object re-identification method based on gated graph neural networks in this application includes:

[0065] Feature extraction is performed on multiple sample images to obtain their feature information; each sample image contains a different type of target object.

[0066] A knowledge graph is constructed based on multiple sample images; wherein the knowledge graph includes multiple nodes corresponding one-to-one with the sample images and edges connecting adjacent nodes, the nodes represent the type of the target object, and the nodes have corresponding weights and hidden states, and the edges represent the correlation between the nodes;

[0067] The knowledge graph is imported into the gated graph neural network, and the feature information of the sample image is input into the gated graph neural network for iterative training; and in each iteration, each node aggregates the hidden state of its neighboring nodes in the previous iteration and its own hidden state in the previous iteration based on the correlation, and updates its own hidden state and weights through the gated recurrent unit.

[0068] The feature information of the image to be identified is input into the trained gated graph neural network. Based on the knowledge graph and the weights, the gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph, so as to re-identify the object to be identified in the image.

[0069] The object re-identification method based on a gated graph neural network in this application constructs a knowledge graph based on multiple sample images. Nodes in the knowledge graph represent the type of the target object, and edges represent the correlation between nodes. During the iterative training of the gated graph neural network, each node aggregates the hidden states of its neighboring nodes from the previous iteration and its own hidden state from the previous iteration based on the correlation, and updates its own hidden state through a gated recurrent unit. The node weights are then updated based on the updated hidden states. This enables the transfer of prototypical representations of categories between similar categories, thereby accelerating the iterative training process. Even with a small number of training samples, the gated graph neural network can still achieve high accuracy. After the gated graph neural network is trained, accurate re-identification of the target object in the image to be identified can be performed based on the trained gated graph neural network.

[0070] The steps and principles of the object re-identification method of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] Figure 1 This is a flowchart of the object re-identification method based on a gated graph neural network according to an embodiment of this application. Figure 2 This is a schematic diagram illustrating a scenario of the object re-identification method based on a gated graph neural network according to an embodiment of this application. See also... Figure 1 and Figure 2 As shown, the object re-identification method in this application embodiment may specifically include the following steps.

[0072] S110, perform feature extraction on multiple sample images to obtain feature information of the sample images; wherein, the multiple sample images contain different types of target objects.

[0073] Optionally, the sample images can be images captured in a specific scenario. For example, in vehicle re-identification, sample images may include images captured by traffic monitoring systems or images recorded by vehicle inspection systems. Similarly, in pedestrian re-identification, sample images may include images captured by monitoring systems in specific scenarios, such as traffic monitoring systems, shopping mall monitoring systems, school monitoring systems, or other monitoring systems, or images recorded in, for example, the household registration information management system of the public security department.

[0074] Optional, in combination Figure 3As shown, step S110, which involves extracting features from multiple sample images to obtain feature information of the sample images, may include:

[0075] S111, preprocess the sample image. Optionally, when the sample image is obtained, it can be preprocessed by removing shadows, reducing noise, smoothing, filtering, restoring, normalizing, enhancing, etc., so that the preprocessed sample image meets the requirements of feature extraction.

[0076] S112, the feature extractor extracts features from the preprocessed sample image to obtain the feature map of the sample image.

[0077] The feature extractor can be a component of a gated graph neural network. Optionally, the feature extractor can serve as the backbone of the gated graph neural network. For example, the feature extractor can be based on a residual network (ResNet), such as the ResNet50 network.

[0078] Optionally, the feature map may include information in dimensions such as the height, width, and number of channels of the sample image. Let x represent the sample image, Let X represent the feature extractor and X represent the feature map, then X∈R H*W*C That is, the feature map is a three-dimensional matrix that includes height information, width information, and channel information.

[0079] It should be noted that, in practice, feature information of sample images can be extracted in various ways. Feature information is not limited to feature maps; other data structures or other data content can also be used.

[0080] S120, a knowledge graph is constructed based on the multiple sample images. The knowledge graph includes multiple nodes corresponding one-to-one with each sample image and edges connecting adjacent nodes. Each node represents the type of a target object and has a corresponding weight and hidden state. The edges represent the correlation between the nodes.

[0081] Optionally, a knowledge graph can be represented by G = {V, A}, where V can be represented as {v1, v2, v3, ..., v...}. K}, where K represents the number of sample images, k is the node identifier, k∈(1,K), v k Let A represent the k-th node, and let A denote the adjacency matrix. The adjacency matrix can be represented by a... ij Let $\mathbf{i}$ represent the edge between nodes $i$ and $j$, where $i,j \in (1, K)$. Optionally, the edge length between nodes $i$ and $j$ can be measured, for example, by a vector distance, i.e., by $\mathbf{i}$. ijThis represents the vector distance between node i and node j.

[0082] Optional, in combination Figure 4 As shown, step S120, which involves constructing a knowledge graph based on multiple sample images, may include the following steps.

[0083] S121, Initialize each node in the knowledge graph based on the feature information of the sample image.

[0084] Optionally, when initializing the knowledge graph, 2 can be selected. n Each sample image is used to obtain its feature information, and each node is constructed based on the feature information of each sample image, forming a 2... n A knowledge graph with 64 nodes. For example, when initializing a knowledge graph, 64 sample images can be selected to construct a knowledge graph with 64 nodes.

[0085] S122, randomly initialize the weights of each node, and configure the initial hidden state of the node to be the same as the initial weight.

[0086] Optionally, during knowledge graph initialization, weights can be randomly assigned to each node in the knowledge graph. The weight of a node can be represented by W, and the weight of a node v is... k The initial weights can be expressed as The hidden state can be represented by h, and node v k The initial hidden state can be represented as The initial hidden state can be configured to be the same as the initial weights. That is, the relationship between the initial hidden state and the initial weights can be expressed as: The weights W serve as the learning parameters of the gated graph neural network and need to be updated during each iteration of training.

[0087] S130, the knowledge graph is imported into the gated graph neural network, and the feature information of the sample image is input into the gated graph neural network for iterative training; and in each iteration, each node aggregates the hidden state of the previous iteration of the neighboring nodes and its own hidden state of the previous iteration based on the correlation, and updates its own hidden state and weight through the gated recurrent unit.

[0088] Optional, in combination Figure 5 As shown, step S130, which involves importing the knowledge graph into the gated graph neural network and inputting the feature information of the sample image into the gated graph neural network for iterative training, may specifically include the following steps.

[0089] S131, the knowledge graph is imported into the gated graph neural network, and the feature information of the sample image is input into the gated graph neural network. The second similarity between the sample image and the type represented by each node in the knowledge graph is determined by the gated graph neural network.

[0090] Optionally, feature information of sample images corresponding one-to-one with nodes in the knowledge graph can be input into a gated graph neural network. The gated graph neural network can then predict the second similarity between the target object in the sample image and the type represented by each node based on the weights of the nodes in the knowledge graph.

[0091] Optionally, a second similarity can be measured based on cosine similarity between the target object in the sample image and the category identified by each node. For example, the cosine similarity between the target object in the sample image and the category identified by each node can be calculated using the following formula based on the feature map and weights.

[0092]

[0093] Where X represents the feature map of the sample image; Represents node v k The weights after t iterations; f k (X) represents the feature map X of the sample image and the node v. k The second similarity between the categories represented.

[0094] S132, Based on the second similarity, determine the vector distance between each of the nodes; wherein the vector distance characterizes the correlation between each of the nodes.

[0095] Since there is a one-to-one correspondence between sample images and nodes, the similarity between the categories represented by the sample images and nodes is positively correlated with the correlation between adjacent nodes. The vector distance between nodes can be determined based on the prediction results of the gated graph neural network.

[0096] S133, based on the vector distance, each node is driven to aggregate the hidden state of its neighboring nodes from the previous iteration, and update the feature vector of each node itself.

[0097] Optionally, the node's own feature vector can be calculated based on the following formula.

[0098]

[0099] in, Represents node v k The feature vector after t iterations; k′ is the node identifier, k′∈(1,K); a kk' Represents node v k To node v k’The vector distance, used to characterize node v k To node v k’ Similarity; a k'k Represents node v k’ To node v k The vector distance is used to characterize node v. k’ To node v k Similarity; Represents node v k’ The hidden state of the previous iteration.

[0100] It can be seen that at node v k and node v k’ If the similarity between nodes is high, then node v will be pulled in. k and node v k’ The vector distance between them encourages information to be shared at node v. k and node v k’ Propagation between nodes, at node v k and node v k’ When the similarity between nodes is low, then move node v further away. k and node v k’ The vector distance between them, or keeping the vector distance between them constant, suppresses information at node v. k and node v k’ This allows for the transfer of prototypical representations of categories between similar categories, thereby accelerating iterative training and reducing the number of samples required to train a gated graph neural network.

[0101] S134, the hidden state of each node is updated by the gated loop unit based on the feature vector and the hidden state of each node in the previous iteration.

[0102] Optionally, the process of updating the hidden state of each node by the gated loop unit can be represented by the following formula.

[0103]

[0104]

[0105]

[0106]

[0107] in, The σ() denotes the update gate; σ() denotes the Sigmoid activation function. Represents node v k The hidden state in the (t-1)th iteration; Indicates resetting the door; W z Wr And W is the weight parameter of the hidden layer, U z U r And U is a hyperparameter of the hidden layer, and W z W r 、W、U z U r Both U and U are constants; Represents node v k The candidate hidden states of the tth iteration; tanh() is the hyperbolic tangent function; Θ is the Hadamard product (i.e., element-wise product) operator; Represents node v k The hidden state in the t iterations.

[0108] After t iterations, each node can determine its own hidden state based on the above formula. The hidden state of each node can be recorded, for example, through a hidden state set.

[0109] S135, based on the updated hidden state, update the weight of the node.

[0110] After t iterations, the hidden state set can be obtained, and then the weight of each node can be calculated using the following formula.

[0111]

[0112] in, Represents node v k The weights after t iterations; o() represents the output function.

[0113] Optionally, when initializing the knowledge graph, a metric matrix can be constructed to record the weights of each node.

[0114] In some embodiments, step S130, which involves importing the knowledge graph into a gated graph neural network and inputting the feature information of the sample image into the gated graph neural network for iterative training, may also include the following steps.

[0115] S136, Based on the second similarity and a predefined loss function, determine the loss value; wherein the loss value characterizes the difference between the prediction result and the actual result of the gated graph neural network.

[0116] Optionally, a triplet loss function, for example, can be used as the loss function for this gated graphical neural network. Of course, in specific implementations, it is not limited to using the triplet loss function; other types of loss functions can also be used.

[0117] S137, if the change in the loss value meets the first preset condition, determine that the gating graph neural network training is complete.

[0118] The first preset condition is a condition representing the completion of training of the gated graph neural network, that is, the first preset condition is the iteration termination condition. Optionally, the first preset condition can be that the change in the loss value is less than the threshold of the loss function during the iteration process. For example, if the change between the loss value of generation t-1 and the loss value of generation t is less than the threshold of the loss function, it indicates that the gated graph neural network has met the condition for completion of training, and the training of the gated graph neural network is determined to be complete.

[0119] It should be noted that other iteration termination conditions can also be used in specific implementations. For example, after the gated graph neural network has undergone N iterations, it is considered that the gated graph neural network has completed iterative training.

[0120] S140, the feature information of the image to be identified is input into the trained gated graph neural network. Based on the knowledge graph and the weights, the gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph, so as to re-identify the object to be identified in the image.

[0121] After the gating graph neural network is trained, the feature information of the image to be identified can be obtained. The feature information of the image to be identified is input into the trained gating graph neural network. Based on the weight of each node in the knowledge graph, the gating graph neural network determines the first similarity between the object to be identified in the image and the target object represented by each node in the knowledge graph, so as to identify whether the object to be identified and the target object in the knowledge graph belong to the same object, or to identify whether two or more objects to be identified belong to the same object.

[0122] The object re-identification method based on a gated graph neural network in this application constructs a knowledge graph based on multiple sample images. Nodes in the knowledge graph represent the type of the target object, and edges represent the correlation between nodes. During the iterative training of the gated graph neural network, each node aggregates the hidden states of its neighboring nodes from the previous iteration and its own hidden state from the previous iteration based on the correlation, and updates its own hidden state through a gated recurrent unit. The node weights are then updated based on the updated hidden states. This enables the transfer of prototypical representations of categories between similar categories, thereby accelerating the iterative training process. Even with a small number of training samples, the gated graph neural network can still achieve high accuracy. After the gated graph neural network is trained, accurate re-identification of the target object in the image to be identified can be performed based on the trained gated graph neural network.

[0123] In some embodiments, step S120, constructing a knowledge graph based on the plurality of sample images, may specifically include the following steps.

[0124] S123, Based on the first number of sample images, construct the knowledge graph having the first number of nodes.

[0125] S124, after the weights of the first number of nodes have been updated, a second number of nodes are added to the knowledge graph based on the second number of sample images to form the knowledge graph with a third number of nodes.

[0126] During knowledge graph initialization, a knowledge graph with a first number of nodes can be constructed based on a first number of sample images, where the first number can be denoted as K. base For example, knowledge graphs can be built in batches, with each batch containing 2 nodes (batch size). n For example, the number of nodes constructed in each batch can be 64. When initializing the knowledge graph, a knowledge graph containing 64 nodes can be constructed based on 64 sample images. The gated graph neural network is then iteratively trained, and training for that batch ends when the weights of each node are properly fitted.

[0127] After the weights of the first number of nodes have been iteratively updated, a second number of nodes can be added to the knowledge graph based on a second number of sample images. This second number can be denoted as K. noval Then the current number of nodes in the knowledge graph K = K base +K noval For example, after the weights of the first batch of 64 nodes have been iteratively updated, another 64 nodes can be added to the knowledge graph, and the gated graph neural network can be iteratively trained again. The types represented by the trained nodes can be denoted as the base categories, and the types represented by the newly added nodes can be denoted as the new categories. When the gated graph neural network is iteratively trained again, the nodes representing the new categories can obtain the prototype representation of the category from the nodes representing the base categories based on similarity, thereby greatly enhancing the prototype representation of the new categories. In this way, the training progress can be greatly accelerated and the number of samples required for training can be reduced. Moreover, building the knowledge graph in batches can reduce the computing power requirements of the equipment.

[0128] It should be noted that, in practice, the knowledge graph can be constructed in multiple batches, not just two.

[0129] Cooperate Figure 6As shown, in some embodiments, step S140 involves inputting the feature information of the image to be identified into the trained gated graph neural network. The gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph based on the knowledge graph and the weights, so as to re-identify the object to be identified in the image. Specifically, this may include the following steps.

[0130] S1411, Perform feature extraction on the image to be identified to obtain feature information of the image to be identified.

[0131] S1412, the feature information of the image to be identified is input into the trained gated graph neural network, and the gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph based on the knowledge graph and the weights.

[0132] S1413, if one of the multiple first similarities is greater than a first threshold, determine that the object to be identified belongs to the type represented by the corresponding node in the knowledge graph.

[0133] Optionally, the type of the object to be identified can be determined based on the softmax classification function, the expression of which is shown below.

[0134]

[0135] Where P represents the classification probability; x represents the image to be identified; X represents the feature information of the image to be identified; K represents the number of nodes in the knowledge graph, that is, the number of categories; f k (X) indicates that the object to be identified in the graph belongs to node v. k The confidence level of the category represented, f k (X) can be represented as b k This indicates the deviation term.

[0136] Classification probability P and confidence level f k The following relationship exists between (X) and arg max k P(y=k|x) equals arg max k f k (X), based on this relationship, f can be... k The expression for (X) is reshaped into the following form.

[0137]

[0138] When b k =0 and ||w i ||2=||wj At 2 o'clock, confidence level f k (X) can be considered as the image to be identified x and The similarity measure between them, that is, the confidence level f k (X) is denoted as the first similarity.

[0139] The type of the object to be identified can be determined using the following formula.

[0140]

[0141] This represents the category identifier of the object to be identified, which can be, for example, the category number (ID) represented by a node in the knowledge graph.

[0142] Taking vehicle re-identification as an example, the image to be identified can be an image collected by a traffic monitoring system, and the nodes in the knowledge graph can be constructed based on images recorded by a vehicle inspection system. For example, if a traffic retrieval system collects information about a vehicle that violates traffic laws but does not collect the license plate number, the image of the vehicle can be input into the gated graph neural network of this application embodiment to identify which vehicle in the vehicle inspection system the vehicle violates, thereby achieving the purpose of identifying the vehicle that violates traffic laws.

[0143] Cooperate Figure 7 As shown, in some embodiments, step S140 involves inputting the feature information of the image to be identified into the trained gated graph neural network. The gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph based on the knowledge graph and the weights, so as to re-identify the object to be identified in the image. Specifically, this may include the following steps.

[0144] S1421, Perform feature extraction on the image to be identified to obtain feature information of the image to be identified.

[0145] S1422, the feature information of the multiple images to be identified is respectively input into the trained gated graph neural network, and the gated graph neural network outputs a similarity matrix of each image to be identified based on the knowledge graph and the weights; wherein, the similarity matrix includes the first similarity between the object to be identified in the image to be identified and the type represented by each node in the knowledge graph.

[0146] S1423, determine the third similarity among the multiple similarity matrices, and if the third similarity meets the second preset condition, determine that the objects to be identified in the multiple images to be identified are of the same type.

[0147] Optionally, the second preset condition is used to characterize that the objects to be identified in multiple images to be identified are of the same type. The second preset condition may be, for example, a third similarity among multiple similarity matrices that is less than a second threshold.

[0148] Thus, the object re-identification method of this application embodiment is not limited to identifying the category represented by a certain node in the knowledge graph of the object to be identified, but can also be used to identify whether multiple objects to be identified belong to the same type of object. Taking vehicle re-identification as an example, in the case of a hit-and-run vehicle, monitoring images collected by the traffic monitoring system at multiple monitoring points can be obtained. These multiple monitoring images are input into the gated graph neural network, which identifies which vehicles in the monitoring images belong to the same vehicle. Then, the escape trajectory of the hit-and-run vehicle can be identified based on the location information of the monitoring points.

[0149] See Figure 8 As shown in the illustration, this application also provides an electronic device, including:

[0150] The extraction module 201 is used to extract features from multiple sample images to obtain feature information of the sample images; wherein, the multiple sample images contain different types of target objects;

[0151] The construction module 202 is used to construct a knowledge graph based on multiple sample images; wherein, the knowledge graph includes multiple nodes corresponding one-to-one with the sample images and edges connecting adjacent nodes, the nodes represent the type of the target object, and the nodes have corresponding weights and hidden states, and the edges represent the correlation between the nodes;

[0152] The training module 203 is used to import the knowledge graph into the gated graph neural network and input the feature information of the sample image into the gated graph neural network for iterative training; and in each iteration, each node aggregates the hidden state of the previous iteration and the hidden state of the previous iteration of the neighboring nodes based on the correlation, and updates its own hidden state and weight through the gated loop unit.

[0153] The recognition module 204 is used to input the feature information of the image to be recognized into the trained gated graph neural network, and through the gated graph neural network, based on the knowledge graph and the weights, determine the first similarity between the object to be recognized in the image to be recognized and the type represented by each node in the knowledge graph, so as to re-recognize the object to be recognized in the image to be recognized.

[0154] In some embodiments, the construction module 202 is specifically used for:

[0155] The nodes in the knowledge graph are initialized based on the feature information of the sample images;

[0156] The weights of each node are randomly initialized, and the initial hidden state of each node is configured to be the same as its initial weight.

[0157] In some embodiments, the training module 203 is specifically used for:

[0158] The knowledge graph is imported into a gated graph neural network, and the feature information of the sample image is input into the gated graph neural network. The second similarity between the sample image and the type represented by each node in the knowledge graph is determined by the gated graph neural network.

[0159] Based on the second similarity, the vector distance between each node is determined; wherein the vector distance characterizes the correlation between each node;

[0160] Based on the vector distance, each node is driven to aggregate the hidden state of its neighboring nodes from the previous iteration, and update the feature vector of each node itself.

[0161] The hidden state of each node is updated by a gated loop unit based on the feature vector and the hidden state of each node in the previous iteration.

[0162] The weights of the nodes are updated based on the updated hidden state.

[0163] In some embodiments, the training module 203 is further configured to:

[0164] Based on the second similarity and a predefined loss function, a loss value is determined; wherein the loss value characterizes the difference between the prediction result and the actual result of the gated graph neural network;

[0165] If the change in the loss value meets the first preset condition, the training of the gated graph neural network is determined to be complete.

[0166] In some embodiments, the identification module 204 is specifically used for:

[0167] Feature extraction is performed on the image to be identified to obtain its feature information;

[0168] The feature information of the image to be identified is input into the trained gated graph neural network. Based on the knowledge graph and the weights, the gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph.

[0169] If one of the multiple first similarities is greater than a first threshold, the object to be identified is determined to belong to the type represented by the corresponding node in the knowledge graph.

[0170] In some embodiments, the identification module 204 is specifically used for:

[0171] Feature extraction is performed on the image to be identified to obtain its feature information;

[0172] The feature information of multiple images to be identified is respectively input into the trained gated graph neural network, and the gated graph neural network outputs a similarity matrix of each image to be identified based on the knowledge graph and the weights; wherein, the similarity matrix includes the first similarity between the object to be identified in the image to be identified and the type represented by each node in the knowledge graph;

[0173] A third similarity is determined among the multiple similarity matrices, and if the third similarity meets a second preset condition, the objects to be identified in the multiple images to be identified are determined to be of the same type.

[0174] In some embodiments, the extraction module 201 is specifically used for:

[0175] The sample images are preprocessed;

[0176] The feature map of the preprocessed sample image is obtained by extracting features from the preprocessed sample image using a feature extractor.

[0177] In some embodiments, the construction module 202 is specifically used for:

[0178] Based on the first number of sample images, construct the knowledge graph with the first number of nodes;

[0179] Once the weights of the first number of nodes have been updated, a second number of nodes are added to the knowledge graph based on the second number of sample images to form the knowledge graph with a third number of nodes.

[0180] See Figure 9 As shown, this application embodiment also provides an electronic device, including at least a memory 301 and a processor 302. The memory 301 stores an application program, and the processor 302 implements the method described in any of the above embodiments when executing the application program on the memory 301.

[0181] Those skilled in the art will understand that embodiments of this application can be provided as methods, electronic devices, computer-readable storage media, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Furthermore, this application can take the form of a computer program product implemented on one or more computer-readable storage media containing computer-readable program code. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0182] The aforementioned processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0183] The aforementioned memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0184] The aforementioned readable storage medium may be a magnetic disk, optical disk, DVD, USB, read-only memory (ROM) or random access memory (RAM), etc. This application does not limit the specific form of storage medium.

[0185] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. An object re-identification method based on a gated graph neural network, characterized in that, include: Feature extraction is performed on multiple sample images to obtain their feature information; each sample image contains a different type of target object. A knowledge graph is constructed based on multiple sample images; wherein the knowledge graph includes multiple nodes corresponding one-to-one with the sample images and edges connecting adjacent nodes, the nodes represent the type of the target object, and the nodes have corresponding weights and hidden states, and the edges represent the correlation between the nodes; The knowledge graph is imported into the gated graph neural network, and the feature information of the sample image is input into the gated graph neural network for iterative training; and in each iteration, each node aggregates the hidden state of its neighboring nodes in the previous iteration and its own hidden state in the previous iteration based on the correlation, and updates its own hidden state and weights through the gated recurrent unit. The feature information of the image to be identified is input into the trained gated graph neural network. Based on the knowledge graph and the weights, the gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph, so as to re-identify the object to be identified in the image. The step of importing the knowledge graph into a gated graph neural network and inputting the feature information of the sample images into the gated graph neural network for iterative training includes: The knowledge graph is imported into a gated graph neural network, and the feature information of the sample image is input into the gated graph neural network. The second similarity between the sample image and the type represented by each node in the knowledge graph is determined by the gated graph neural network. Based on the second similarity, the vector distance between each node is determined; wherein the vector distance characterizes the correlation between each node; Based on the vector distance, each node is driven to aggregate the hidden state of its neighboring nodes from the previous iteration, and update the feature vector of each node itself. The hidden state of each node is updated by a gated loop unit based on the feature vector and the hidden state of each node in the previous iteration. The weights of the nodes are updated based on the updated hidden state.

2. The method according to claim 1, characterized in that, The construction of the knowledge graph based on multiple sample images includes: The nodes in the knowledge graph are initialized based on the feature information of the sample images; The weights of each node are randomly initialized, and the initial hidden state of each node is configured to be the same as its initial weight.

3. The method according to claim 1, characterized in that, The step of importing the knowledge graph into the gated graph neural network and inputting the feature information of the sample image into the gated graph neural network for iterative training further includes: Based on the second similarity and a predefined loss function, a loss value is determined; wherein the loss value characterizes the difference between the prediction result and the actual result of the gated graph neural network; If the change in the loss value meets the first preset condition, the training of the gated graph neural network is determined to be complete.

4. The method according to claim 1, characterized in that, The step of inputting feature information of the image to be identified into the trained gated graph neural network, and determining the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph based on the knowledge graph and the weights, so as to re-identify the object to be identified in the image, includes: Feature extraction is performed on the image to be identified to obtain its feature information; The feature information of the image to be identified is input into the trained gated graph neural network. Based on the knowledge graph and the weights, the gated graph neural network determines the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph. If one of the multiple first similarities is greater than a first threshold, the object to be identified is determined to belong to the type represented by the corresponding node in the knowledge graph.

5. The method according to claim 1, characterized in that, The step of inputting feature information of the image to be identified into the trained gated graph neural network, and determining the first similarity between the object to be identified in the image and the type represented by each node in the knowledge graph based on the knowledge graph and the weights, so as to re-identify the object to be identified in the image, includes: Feature extraction is performed on the image to be identified to obtain its feature information; The feature information of multiple images to be identified is respectively input into the trained gated graph neural network, and the gated graph neural network outputs a similarity matrix of each image to be identified based on the knowledge graph and the weights; wherein, the similarity matrix includes the first similarity between the object to be identified in the image to be identified and the type represented by each node in the knowledge graph; A third similarity is determined among the multiple similarity matrices, and if the third similarity meets a second preset condition, the objects to be identified in the multiple images to be identified are determined to be of the same type.

6. The method according to claim 1, characterized in that, The step of extracting features from multiple sample images to obtain feature information of the sample images includes: The sample images are preprocessed; The feature map of the preprocessed sample image is obtained by extracting features from the preprocessed sample image using a feature extractor.

7. The method according to claim 1, characterized in that, The construction of the knowledge graph based on multiple sample images includes: Based on the first number of sample images, construct the knowledge graph with the first number of nodes; Once the weights of the first number of nodes have been updated, a second number of nodes are added to the knowledge graph based on the second number of sample images to form the knowledge graph with a third number of nodes.

8. An electronic device, characterized in that, include: The extraction module is used to extract features from multiple sample images to obtain feature information of the sample images; wherein, the multiple sample images contain different types of target objects; A construction module is used to construct a knowledge graph based on multiple sample images; wherein, the knowledge graph includes multiple nodes corresponding one-to-one with the sample images and edges connecting adjacent nodes, the nodes represent the type of the target object, and the nodes have corresponding weights and hidden states, and the edges represent the correlation between the nodes; The training module is used to import the knowledge graph into the gated graph neural network and input the feature information of the sample image into the gated graph neural network for iterative training; and in each iteration, each node aggregates the hidden state of its neighboring nodes and the hidden state of the previous iteration based on the correlation, and updates its own hidden state and weights through the gated recurrent unit. The recognition module is used to input the feature information of the image to be recognized into the trained gated graph neural network, and through the gated graph neural network based on the knowledge graph and the weights, determine the first similarity between the object to be recognized in the image and the type represented by each node in the knowledge graph, so as to re-recognize the object to be recognized in the image. The training module is specifically used for: The knowledge graph is imported into a gated graph neural network, and the feature information of the sample image is input into the gated graph neural network. The second similarity between the sample image and the type represented by each node in the knowledge graph is determined by the gated graph neural network. Based on the second similarity, the vector distance between each node is determined; wherein the vector distance characterizes the correlation between each node; Based on the vector distance, each node is driven to aggregate the hidden state of its neighboring nodes from the previous iteration, and update the feature vector of each node itself. The hidden state of each node is updated by a gated loop unit based on the feature vector and the hidden state of each node in the previous iteration. The weights of the nodes are updated based on the updated hidden state.

9. An electronic device, characterized in that, It includes at least a memory and a processor, wherein the memory stores an application program, and the processor implements the method as described in any one of claims 1 to 7 when executing the application program in the memory.

Images