Pedestrian re-identification method and device based on posture temporal hypergraph network

Abstract

The present application discloses a person re-identification method and device based on a posture temporal hypergraph network, wherein the method comprises: obtaining a query video and a retrieval video; inputting the query video and the retrieval video into a pre-trained person re-identification network to obtain feature vectors of the query video and the retrieval video respectively; and identifying the same person according to the feature vectors of the query video and the retrieval video, and when the cosine distance between the feature vectors of the query video and the retrieval video is less than a preset threshold. Thus, the problem that the related art only considers how to fuse multiple local features in the spatiotemporal domain, lacks exploration of how to model the correlation between local regions of the image, resulting in poor fusion capability of the temporal and spatial domain features of the video, and low reliability and accuracy is solved.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and pedestrian re-identification, and in particular to a method and device for pedestrian re-identification based on a posture temporal hypergraph network. Background Art

[0002] The pedestrian re-identification task has been widely used in computer vision and surveillance analysis due to its ability to identify pedestrians from different perspectives. Compared with using images to achieve pedestrian re-identification tasks, since videos contain more information, video-based pedestrian re-identification methods and systems have also been explored and applied. However, how to obtain an excellent video representation through temporal and spatial features is one of the challenges in current video-based pedestrian re-identification methods.

[0003] In the related technologies, there are many methods to obtain excellent video representation through temporal and spatial features, such as: using optical flow to describe temporal features, using attention mechanism to fuse temporal and spatial features in the video, and using three-dimensional convolutional neural network or recurrent neural network to simultaneously process feature extraction in temporal and spatial domains.

[0004] However, related technologies only consider how to fuse multiple local features in the spatiotemporal domain, and rarely explore how to model the correlation between local regions of the image, which urgently needs to be improved.

[0005] Application Contents

[0006] The present application provides a pedestrian re-identification method and device based on a posture temporal hypergraph network to solve the problem that the related technology only considers how to fuse multiple local features in the spatiotemporal domain, lacks the exploration of how to model the correlation between local areas of the image, resulting in poor fusion capability of the temporal and spatial domain features of the video, low reliability and accuracy, and other problems.

[0007] The first aspect of the present application provides a method for pedestrian re-identification based on a posture temporal hypergraph network, comprising the following steps: obtaining a query video and a retrieval video; inputting the query video and the retrieval video into a pre-trained pedestrian re-identification network to obtain feature vectors of the query video and the retrieval video, respectively; and identifying the same pedestrian based on the feature vectors of the query video and the retrieval video, and when the cosine distance between the feature vectors of the query video and the retrieval video is less than a preset threshold.

[0008] Optionally, in one embodiment of the present application, before the query video and the retrieval video are input into a pre-trained pedestrian re-identification network, it also includes: training a posture estimation network using image data containing pedestrian key points as labels; based on the trained posture estimation network, screening out key areas in the time domain and spatial domain, constructing a hypergraph neural network, and training the pedestrian re-identification network using video data containing pedestrian identification.

[0009] Optionally, in one embodiment of the present application, the use of image data containing pedestrian key points as labels to train a posture estimation network includes: generating a real key point heat map based on the positions of the labels; constructing a posture estimation deep neural network; inputting a training image into the posture estimation deep neural network to obtain a predicted key point heat map; calculating a first loss function based on the real key point heat map and the predicted key point heat map, and updating the parameters of the posture estimation deep neural network based on the loss function.

[0010] Optionally, in one embodiment of the present application, the posture estimation network after training screens out key areas in the time domain and the spatial domain, constructs a hypergraph neural network, and uses video data containing pedestrian identification to train a pedestrian re-identification network, including: inputting each frame in the pedestrian re-identification training video into the posture estimation deep neural network to obtain predicted key point positions; constructing a pedestrian re-identification deep neural network based on a posture timing hypergraph; calculating a second loss function based on the predicted pedestrian class label and the true class label, and updating the parameters of the pedestrian re-identification deep neural network based on the posture timing hypergraph based on the second loss function.

[0011] Optionally, in one embodiment of the present application, the step of inputting the query video and the retrieval video into a pre-trained person re-identification network to obtain feature vectors of the query video and the retrieval video, respectively, includes: loading the trained parameters of the person re-identification deep neural network; inputting the query video into the loaded person re-identification network to obtain the feature vector of the query video; and inputting the retrieval video into the person re-identification network to obtain the feature vector of the retrieval video.

[0012] Optionally, in one embodiment of the present application, the identification as the same pedestrian based on the feature vectors of the query video and the retrieval video, and when the cosine distance between the feature vectors of the query video and the retrieval video is less than a preset threshold, includes: calculating the cosine distance between the feature vectors of the query video and the retrieval video; if the cosine distance is less than the preset threshold, determining that the query video and the retrieval video are the same pedestrian, otherwise determining that they are different pedestrians.

[0013] The second aspect of the present application provides a pedestrian re-identification device based on a posture temporal hypergraph network, including: an acquisition module, used to acquire a query video and a retrieval video; a calculation module, used to input the query video and the retrieval video into a pre-trained pedestrian re-identification network to obtain feature vectors of the query video and the retrieval video respectively; and an identification module, used to identify the same pedestrian based on the feature vectors of the query video and the retrieval video, and when the cosine distance between the feature vectors of the query video and the retrieval video is less than a preset threshold.

[0014] Optionally, in one embodiment of the present application, the pedestrian re-identification device based on the posture temporal hypergraph network further includes: an estimation module for training a posture estimation network using image data containing pedestrian key points as labels; a construction module for screening out key areas in the time domain and the spatial domain based on the trained posture estimation network, constructing a hypergraph neural network, and training a pedestrian re-identification network using video data containing pedestrian identification;

[0015] Optionally, in one embodiment of the present application, the image data trains a posture estimation network, and is further used to: generate a real key point heat map based on the position of the label; construct a posture estimation deep neural network; input the training image into the posture estimation deep neural network to obtain a predicted key point heat map; calculate a first loss function based on the real key point heat map and the predicted key point heat map, and update the parameters of the posture estimation deep neural network based on the loss function.

[0016] Optionally, in one embodiment of the present application, the construction module includes: a first acquisition unit, used to input each frame in the pedestrian re-identification training video into the posture estimation deep neural network to obtain the predicted key point position; a construction unit, used to construct a pedestrian re-identification deep neural network based on a posture timing hypergraph; an update unit, used to calculate a second loss function based on the predicted pedestrian class label and the true class label, and update the parameters of the pedestrian re-identification deep neural network based on the posture timing hypergraph based on the second loss function.

[0017] Optionally, in one embodiment of the present application, the computing module includes: a loading unit for loading the parameters of the trained person re-identification deep neural network; a retrieval unit for inputting the query video into the loaded person re-identification network to obtain a feature vector of the query video; and a second acquisition unit for inputting the retrieval video into the person re-identification network to obtain a feature vector of the retrieval video.

[0018] Optionally, in one embodiment of the present application, the recognition module includes: a calculation unit, used to calculate the cosine distance between the feature vectors of the query video and the retrieval video; a judgment unit, used to determine whether the query video and the retrieval video are the same person, if the cosine distance is less than the preset threshold, then the query video and the retrieval video are determined to be the same pedestrian, otherwise they are determined to be different pedestrians.

[0019] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pedestrian re-identification method based on a posture temporal hypergraph network as described in the above embodiment.

[0020] The fourth aspect of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the pedestrian re-identification method based on the posture temporal hypergraph network as described in the above embodiment.

[0021] The pedestrian re-identification method based on the posture time series hypergraph network of the embodiment of the present application uses the posture estimation and hypergraph neural network method to construct the temporal and spatial domain feature association of the video sequence, and determines the result of pedestrian re-identification by querying and retrieving the feature vector of the video, thereby improving the fusion capability of the temporal and spatial domain features of the video, and further ensuring the reliability and accuracy of the pedestrian re-identification method. Thus, the problem that the related art only considers how to fuse multiple local features in the temporal and spatial domains, lacks the exploration of how to model the correlation between local regions of the image, resulting in poor fusion capability of the temporal and spatial domain features of the video, and low reliability and accuracy is solved.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a pedestrian re-identification method based on a posture temporal hypergraph network provided according to an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for constructing a deep neural network for pedestrian re-identification based on a posture temporal hypergraph according to an embodiment of the present application;

[0026] Figure 3Schematic diagram of a pedestrian re-identification method based on a posture temporal hypergraph network according to an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of a pedestrian re-identification device based on a posture temporal hypergraph network provided according to an embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The following describes the pedestrian re-identification method and device based on the posture time series hypergraph network of the embodiment of the present application with reference to the accompanying drawings. In view of the related technology mentioned in the background technology center, which only considers how to fuse multiple local features in the spatiotemporal domain, lacks the exploration of how to model the correlation between local regions of the image, resulting in poor fusion ability of the temporal and spatial features of the video, and low reliability and accuracy, the present application provides a method using posture estimation and hypergraph neural network, which uses the posture estimation and hypergraph neural network method to construct the spatiotemporal feature association of the video sequence, and determines the result of pedestrian re-identification by querying and retrieving the feature vector of the video, thereby improving the fusion ability of the temporal and spatial features of the video, and then ensuring the reliability and accuracy of the pedestrian re-identification method. As a result, the problem that the related technology only considers how to fuse multiple local features in the spatiotemporal domain, lacks the exploration of how to model the correlation between local regions of the image, resulting in poor fusion ability of the temporal and spatial features of the video, and low reliability and accuracy is solved.

[0031] Specifically, Figure 1 A flowchart of a pedestrian re-identification method based on a posture temporal hypergraph network provided in an embodiment of the present application.

[0032] like Figure 1 As shown, the pedestrian re-identification method based on the posture temporal hypergraph network includes the following steps:

[0033] In step S101, a query video and a search video are obtained.

[0034] It should be noted that there are many ways to obtain it, and no specific restrictions are made here.

[0035] In step S102, the query video and the retrieval video are input into a pre-trained person re-identification network to obtain feature vectors of the query video and the retrieval video respectively.

[0036] It can be understood that pedestrian re-identification, also known as pedestrian re-recognition, is a technology that uses computer vision technology to determine whether there is a specific pedestrian in an image or video sequence. The embodiment of the present application uses a pre-trained pedestrian re-identification network to obtain the corresponding feature vector. The training method of the pedestrian re-identification network is described in detail below.

[0037] Optionally, in one embodiment of the present application, before the query video and the retrieval video are input into a pre-trained pedestrian re-identification network, it also includes: training a pose estimation network using image data containing pedestrian key points as labels; based on the trained pose estimation network, screening out key areas in the time domain and spatial domain, constructing a hypergraph neural network, and training the pedestrian re-identification network using video data containing pedestrian identification.

[0038] For example, the embodiment of the present application can use image data containing pedestrian key points as labels to train the posture estimation network, and combine the key areas screened in the time domain and the spatial domain to build a hypergraph neural network. In order to ensure the reliability of the pedestrian re-identification network, the embodiment of the present application can use video data containing pedestrian identification to train the pedestrian re-identification network, so as to achieve the purpose of pre-training the pedestrian re-identification network.

[0039] Optionally, in one embodiment of the present application, a posture estimation network is trained using image data containing pedestrian key points as labels, including: generating a real key point heat map based on the position of the labels; constructing a posture estimation deep neural network; inputting the training image into the posture estimation deep neural network to obtain a predicted key point heat map; calculating a first loss function based on the real key point heat map and the predicted key point heat map, and updating the parameters of the posture estimation deep neural network based on the loss function.

[0040] As a possible implementation method, the embodiment of the present application can generate a real key point heat map according to the position of the key point label, and input the training image into a pre-built posture estimation neural network to obtain a predicted key point heat map, use the real key point heat map and the predicted key point heat map to calculate the first loss function, and update the parameters of the posture estimation deep network based on the first loss function. The embodiment of the present application can use the training image to update the parameters of the posture estimation deep neural network, so that the posture estimation deep neural network is in a learnable and evolvable state, which increases the practicality and reliability of the embodiment of the present application.

[0041] Optionally, in one embodiment of the present application, based on the trained posture estimation network, key areas are screened out in the time domain and the spatial domain, a hypergraph neural network is constructed, and the pedestrian re-identification network is trained using video data containing pedestrian identification, including: inputting each frame in the pedestrian re-identification training video into the posture estimation deep neural network to obtain the predicted key point position; constructing a pedestrian re-identification deep neural network based on the posture timing hypergraph; calculating a second loss function based on the predicted pedestrian class label and the true class label, and updating the parameters of the pedestrian re-identification deep neural network based on the posture timing hypergraph based on the second loss function.

[0042] Specifically, Figure 2 As shown, the main construction method of the deep neural network for pedestrian re-identification based on the posture time series hypergraph in the embodiment of the present application includes the following steps:

[0043] Step S201: Feature extraction. The embodiment of the present application uses a basic convolutional neural network to extract a feature map of each input frame image. The basic convolutional neural network is a widely used deep neural network front-end framework, including: convolution layer, maximum pooling layer and average pooling layer and other structures.

[0044] Step S202: Extracting key areas. Centered on the key point positions predicted for each frame of image obtained in step S201, the embodiment of the present application crops key areas of the same size on the feature map of each frame of image, and the key areas include: a head key area, a torso key area, and a leg key area.

[0045] Step S203: Hyperedge construction. The embodiment of the present application constructs a hyperedge set based on the feature similarity of the key area, wherein the hyperedge set includes: a global hyperedge set and a local hyperedge set, and the local hyperedge set includes: a head hyperedge set, a torso hyperedge set, and a leg hyperedge set. The embodiment of the present application uses the features of the key area as the nodes of the hypergraph, and constructs a global hypergraph and a local hypergraph based on the hyperedge set. The global hypergraph is constructed by the global hyperedge set, and the local hypergraph is constructed by the head hyperedge set, the torso hyperedge set, and the leg hyperedge set.

[0046] Step S204: Hypergraph propagation. The embodiment of the present application uses a hypergraph convolution operation to perform hypergraph propagation on the global hypergraph and the local hypergraph respectively to obtain updated node features.

[0047] For example, the calculation formula for the hypergraph convolution operation is:

[0048]

[0049] Among them, V (l) is the feature vector of the l-th layer node in the hypergraph network, V (l+1) is the feature vector of the l+1th layer node in the hypergraph network, σ is the nonlinear activation function, H is the association matrix describing the hypergraph, D y and D e are the degrees of nodes and hyperedges in the hypergraph, Θ is the first hyperparameter vector, W e is the second hyperparameter vector.

[0050] Step S205: Hypergraph feature fusion. The embodiment of the present application uses an attention mechanism to weighted average the features of all nodes in the global hypergraph to obtain a global feature vector, and to weighted average the features of all nodes in the local hypergraph to obtain a local feature vector.

[0051] For example, the attention mechanism can be expressed as: assigning different weights to the nodes of the global hypergraph and the local hypergraph for weighted averaging. The calculation formula for weighted averaging is:

[0052]

[0053] in, is the feature vector of the node, and f is the global feature vector obtained from the global hypergraph or the local feature vector obtained from the local hypergraph.

[0054] Step S206: Video feature fusion. In the embodiment of the present application, the global feature vector and the local feature vector are concatenated to obtain a feature vector describing the input video.

[0055] Furthermore, the predicted pedestrian class labels can be obtained by selecting key frames from the training video in the embodiment of the present application, and inputting them together with the predicted key point positions into the pedestrian re-identification deep neural network based on the posture time series hypergraph. Among them, the key frames are a set of fixed-length video frames randomly sampled according to the obtained posture information. It can be understood that video frames containing posture information have a higher probability of being sampled.

[0056] The embodiment of the present application can use training videos to update the parameters of the pedestrian re-identification deep neural network, so that the pedestrian re-identification deep neural network is in a learnable and evolvable state, which increases the practicality and reliability of the embodiment of the present application.

[0057] Optionally, in one embodiment of the present application, the query video and the retrieval video are input into a pre-trained person re-identification network to obtain feature vectors of the query video and the retrieval video, respectively, including: loading the parameters of the trained person re-identification deep neural network; inputting the query video into the loaded person re-identification network to obtain the feature vector of the query video; inputting the retrieval video into the person re-identification network to obtain the feature vector of the retrieval video.

[0058] In the actual implementation process, after loading the trained person re-identification network parameters, the embodiment of the present application can input the query video and the search video into the person re-identification network respectively, and obtain the feature vector of the query video and the feature vector of the search video respectively. The embodiment of the present application updates and improves the person re-identification network by updating the parameters of the person re-identification network, thereby ensuring the reliability of the person re-identification network and making the recognition results more accurate.

[0059] In step S103, based on the feature vectors of the query video and the retrieval video, when the cosine distance between the feature vectors of the query video and the retrieval video is less than a preset threshold, the pedestrian is identified as the same person.

[0060] It is understandable that the embodiment of the present application can use the cosine distance between the feature vector of the query video obtained using the pedestrian re-identification network and the feature vector of the retrieval video obtained using the pedestrian re-identification network to determine whether the two are the same person. Specifically, when the cosine distance between the feature vector of the query video and the feature vector of the retrieval video is less than a preset threshold, it can be identified that the query video and the retrieval video are the same person. Among them, the preset threshold can be set by those skilled in the art according to actual conditions, and no specific limitation is made here.

[0061] Optionally, in one embodiment of the present application, based on the feature vectors of the query video and the retrieval video, and when the cosine distance between the feature vectors of the query video and the retrieval video is less than a preset threshold, identifying them as the same pedestrian includes: calculating the cosine distance between the feature vectors of the query video and the retrieval video; if the cosine distance is less than the preset threshold, determining that the query video and the retrieval video are the same pedestrian, otherwise determining them as different pedestrians.

[0062] For example, the embodiment of the present application can obtain the cosine distance between the feature vectors of the query video and the retrieved video through vector calculation. If the cosine distance is less than a certain threshold, it is determined that the query video and the retrieved video are the same pedestrian; if the cosine distance is greater than the preset threshold, it is determined that the query video and the retrieved video are not the same pedestrian.

[0063] Combination Figure 3 The principle of a pedestrian re-identification method based on a posture temporal hypergraph network of a specific embodiment of the present application is explained in detail.

[0064] like Figure 3 As shown, the embodiment of the present application includes the following steps:

[0065] Step S301: Construct a posture estimation network based on the posture estimation training image. The embodiment of the present application uses image data containing pedestrian key points as labels to train the posture estimation network. Specifically, the embodiment of the present application generates a real key point heat map based on the position of the key point labels, and constructs a posture estimation deep neural network. The training image is then input into the posture estimation deep neural network to obtain a predicted key point heat map. Finally, based on the real key point heat map and the predicted key point heat map, the loss function is calculated to update the parameters of the posture estimation deep neural network. The embodiment of the present application uses training images to update the parameters of the posture estimation deep neural network, so that the posture estimation deep neural network is in a learnable and evolvable state, which increases the practicality and reliability of the embodiment of the present application.

[0066] Step S302: constructing a person re-identification network based on the person re-identification training video. The embodiment of the present application selects key areas in the time domain and the spatial domain, constructs a hypergraph neural network, and uses video data containing pedestrian identification to train the person re-identification network.

[0067] Furthermore, the predicted pedestrian class labels can be obtained by selecting key frames from the training video in the embodiment of the present application, and inputting them together with the predicted key point positions into the pedestrian re-identification deep neural network based on the posture time series hypergraph. Among them, the key frames are a set of fixed-length video frames randomly sampled according to the obtained posture information. It can be understood that video frames containing posture information have a higher probability of being sampled.

[0068] The embodiment of the present application uses training videos to update the parameters of the pedestrian re-identification deep neural network, so that the pedestrian re-identification deep neural network is in a learnable and evolvable state, which increases the practicality and reliability of the embodiment of the present application.

[0069] Step S303: feature vector calculation. The embodiment of the present application inputs the query video and the retrieval video into the pedestrian re-identification network to obtain the feature vectors of the query video and the retrieval video. Specifically, the embodiment of the present application first loads the parameters of the trained pedestrian re-identification network; secondly, the query video is input into the pedestrian re-identification network to obtain the feature vector of the query video; finally, the retrieval video is input into the pedestrian re-identification network to obtain the feature vector of the retrieval video. The embodiment of the present application updates and improves the pedestrian re-identification network by updating the parameters of the pedestrian re-identification network, thereby ensuring the reliability of the pedestrian re-identification network and making the recognition results more accurate.

[0070] Step S304: Determine the recognition result. The embodiment of the present application determines whether the query video and the retrieved video belong to the same pedestrian based on the feature vectors of the query video and the retrieved video by using a threshold. Specifically, the embodiment of the present application first calculates the cosine distance between the feature vectors of the query video and the retrieved video; secondly, sets a threshold, and if the cosine distance is less than the threshold, the query video and the retrieved video are determined to be the same pedestrian, otherwise they are determined to be different pedestrians.

[0071] According to the pedestrian re-identification method based on the posture time series hypergraph network proposed in the embodiment of the present application, the time and space domain feature association of the video sequence is constructed using the posture estimation and hypergraph neural network method, and the result of pedestrian re-identification is determined by querying and retrieving the feature vector of the video, thereby improving the fusion capability of the video time and space domain features, and further ensuring the reliability and accuracy of the pedestrian re-identification method. Thus, the problem that the related technology only considers how to fuse multiple local features in the time and space domain, lacks the exploration of how to model the correlation between local areas of the image, resulting in poor fusion capability of the video time and space domain features, low reliability and accuracy, etc.

[0072] Next, a pedestrian re-identification device based on a posture temporal hypergraph network proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0073] Figure 4 It is a block diagram of a pedestrian re-identification device based on a posture temporal hypergraph network according to an embodiment of the present application.

[0074] like Figure 4 As shown, the pedestrian re-identification device 10 based on the posture temporal hypergraph network includes: an acquisition module 100, a calculation module 200 and a recognition module 300.

[0075] Specifically, the acquisition module 100 is used to acquire query videos and retrieve videos.

[0076] The calculation module 200 is used to input the query video and the retrieval video into a pre-trained pedestrian re-identification network to obtain feature vectors of the query video and the retrieval video respectively.

[0077] The identification module 300 is used to identify the same pedestrian based on the feature vectors of the query video and the search video, and when the cosine distance between the feature vectors of the query video and the search video is less than a preset threshold.

[0078] Optionally, in one embodiment of the present application, the pedestrian re-identification device 10 based on the posture temporal hypergraph network further includes: an estimation module and a construction module.

[0079] Among them, the estimation module uses image data containing pedestrian key points as labels to train the posture estimation network.

[0080] The construction module selects key areas in the time and space domains based on the trained posture estimation network, builds a hypergraph neural network, and uses video data containing pedestrian identification to train a pedestrian re-identification network;

[0081] Optionally, in one embodiment of the present application, the estimation module is further used to: generate a real key point heat map based on the position of the label; construct a posture estimation deep neural network; input the training image into the posture estimation deep neural network to obtain a predicted key point heat map; calculate a first loss function based on the real key point heat map and the predicted key point heat map, and update the parameters of the posture estimation deep neural network based on the loss function.

[0082] Optionally, in one embodiment of the present application, the construction module includes: a first acquisition unit, a construction unit and an update unit.

[0083] Among them, the first acquisition unit is used to input each frame in the pedestrian re-identification training video into the posture estimation deep neural network to obtain the predicted key point position.

[0084] A construction unit for constructing a deep neural network for person re-identification based on a pose temporal hypergraph.

[0085] An updating unit is used to calculate a second loss function according to the predicted pedestrian class label and the true class label, and to update the parameters of the deep neural network for pedestrian re-identification based on the posture temporal hypergraph based on the second loss function.

[0086] Optionally, in one embodiment of the present application, the calculation module 200 includes: a loading unit, a retrieval unit and a second acquisition unit.

[0087] Among them, the loading unit is used to load the parameters of the trained pedestrian re-identification deep neural network.

[0088] The retrieval unit is used to input the query video into the loaded pedestrian re-identification network to obtain the feature vector of the query video.

[0089] The second acquisition unit is used to input the search video into the pedestrian re-identification network to obtain a feature vector of the search video.

[0090] Optionally, in one embodiment of the present application, the identification module 300 includes: a calculation unit and a determination unit.

[0091] The calculation unit is used to calculate the cosine distance between the feature vectors of the query video and the retrieved video.

[0092] The judgment unit is used to judge whether the query video and the retrieved video are the same person. If the cosine distance is less than a preset threshold, the query video and the retrieved video are judged to be the same pedestrian, otherwise they are judged to be different pedestrians.

[0093] It should be noted that the aforementioned explanation of the embodiment of the pedestrian re-identification method based on the posture temporal hypergraph network is also applicable to the pedestrian re-identification device based on the posture temporal hypergraph network of this embodiment, and will not be repeated here.

[0094] According to the pedestrian re-identification device based on the posture time series hypergraph network proposed in the embodiment of the present application, the time and space domain feature association of the video sequence is constructed using the posture estimation and hypergraph neural network method, and the result of pedestrian re-identification is determined by querying and retrieving the feature vector of the video, thereby improving the fusion capability of the video time and space domain features, and further ensuring the reliability and accuracy of the pedestrian re-identification method. Thus, the problem that the related technology only considers how to fuse multiple local features in the time and space domain, lacks the exploration of how to model the correlation between local areas of the image, resulting in poor fusion capability of the video time and space domain features, low reliability and accuracy, etc.

[0095] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0096] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0097] When the processor 502 executes the program, the pedestrian re-identification method based on the posture temporal hypergraph network provided in the above embodiment is implemented.

[0098] Furthermore, the electronic device further comprises:

[0099] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0100] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0101] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0102] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0103] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0104] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0105] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned pedestrian re-identification method based on the posture temporal hypergraph network is implemented.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0107] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0108] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0109] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0110] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0113] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A pedestrian re-identification method based on posture temporal hypergraph network, characterized in that: The following steps are involved: Get query videos and retrieve videos; Inputting the query video and the retrieval video into a pre-trained person re-identification network to obtain feature vectors of the query video and the retrieval video respectively; as well as According to the feature vectors of the query video and the retrieval video, when the cosine distance between the feature vectors of the query video and the retrieval video is less than a preset threshold, the same pedestrian is identified; before the query video and the retrieval video are input into a pre-trained pedestrian re-identification network, the method further includes: Train the pose estimation network using image data labeled with key points of pedestrians; The method of training a posture estimation network using image data containing pedestrian key points as labels includes: Generate a real key point heat map according to the position of the label; Build a deep neural network for pose estimation; Based on the trained pose estimation network, key areas are screened in the time and space domains, a hypergraph neural network is constructed, and the pedestrian re-identification network is trained using video data containing pedestrian identification. The method of filtering out key areas in the time domain and the space domain based on the trained posture estimation network, constructing a hypergraph neural network, and training a pedestrian re-identification network using video data containing pedestrian identification includes: Input each frame in the person re-identification training video into the posture estimation deep neural network to obtain the predicted key point positions; Construct a deep neural network for person re-identification based on posture temporal hypergraph; Calculating a second loss function according to the predicted pedestrian class label and the true class label, and updating parameters of the pedestrian re-identification deep neural network based on the posture temporal hypergraph based on the second loss function; The construction of a deep neural network for person re-identification based on a pose temporal hypergraph includes the following steps: Step 1: Feature extraction; Step 2: Extract key areas, taking the key point position predicted for each frame image obtained in step 1 as the center, cropping key areas of the same size on the feature map of each frame image, the key areas include: head key area, torso key area and leg key area; Step 3: Hyperedge construction: The features of the key areas are used as nodes of the hypergraph. The global hypergraph and local hypergraph are constructed based on the hyperedge set. The global hypergraph is constructed by the global hyperedge set, and the local hypergraph is constructed by the head hyperedge set, the torso hyperedge set, and the leg hyperedge set. Step 4: Hypergraph propagation: Use hypergraph convolution operation to propagate the global hypergraph and the local hypergraph respectively to obtain the updated node features; the calculation formula of the hypergraph convolution operation is: Among them, V (l) is the feature vector of the l-th layer node in the hypergraph network, V (l+1) is the feature vector of the l+1th layer node in the hypergraph network, σ is the nonlinear activation function, H is the association matrix describing the hypergraph, D v and D e are the degrees of nodes and hyperedges in the hypergraph, Θ is the first hyperparameter vector, W e is the second hyperparameter vector; Step 5: Hypergraph feature fusion: Use the attention mechanism to weighted average the features of all nodes in the global hypergraph to obtain the global feature vector, and weighted average the features of all nodes in the local hypergraph to obtain the local feature vector. The attention mechanism can be expressed as: assigning different weights to the nodes of the global hypergraph and the local hypergraph for weighted averaging. The calculation formula for weighted averaging is: in, is the feature vector of the node, f is the global feature vector obtained from the global hypergraph or the local feature vector obtained from the local hypergraph; Step 6: Video feature fusion, concatenate the global feature vector with the local feature vector to obtain a feature vector that describes the input video.

2. The method according to claim 1, characterized in that: The method further comprises: training a pose estimation network using image data containing pedestrian key points as labels; Inputting the training image into the posture estimation deep neural network to obtain a predicted key point heat map; A first loss function is calculated according to the real key point heat map and the predicted key point heat map, and parameters of the posture estimation deep neural network are updated based on the loss function.

3. The method according to claim 1, characterized in that The step of inputting the query video and the retrieval video into a pre-trained person re-identification network to obtain feature vectors of the query video and the retrieval video respectively includes: Loading the trained parameters of the pedestrian re-identification deep neural network; Inputting the query video into the loaded person re-identification network to obtain a feature vector of the query video; The retrieval video is input into the pedestrian re-identification network to obtain a feature vector of the retrieval video.

4. The method according to claim 3, characterized in that: The step of identifying the same person according to the feature vectors of the query video and the search video and when the cosine distance between the feature vectors of the query video and the search video is less than a preset threshold value includes: Calculating a cosine distance between feature vectors of the query video and the retrieved video; If the cosine distance is less than the preset threshold, the query video and the retrieved video are determined to be the same pedestrian, otherwise they are determined to be different pedestrians.

5. A pedestrian re-identification device based on posture temporal hypergraph network, characterized in that: The pedestrian re-identification device is used to execute the method according to any one of claims 1 to 4, and comprises: An acquisition module, used to acquire query videos and retrieve videos; a computing module, configured to input the query video and the retrieval video into a pre-trained person re-identification network to obtain feature vectors of the query video and the retrieval video respectively; and The recognition module is used to recognize the same pedestrian based on the feature vectors of the query video and the retrieval video and when the cosine distance between the feature vectors of the query video and the retrieval video is less than a preset threshold.

6. The device according to claim 5, characterized in that Also includes: The estimation module trains the pose estimation network using image data containing pedestrian key points as labels; The building module screens out key areas in the time and space domains based on the trained posture estimation network, builds a hypergraph neural network, and uses video data containing pedestrian identification to train the pedestrian re-identification network.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pedestrian re-identification method based on a posture temporal hypergraph network as described in any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the pedestrian re-identification method based on a posture temporal hypergraph network as described in any one of claims 1 to 4.

Images