Pedestrian re-identification method and device, intelligent device and storage medium

Abstract

The application relates to the technical field of target identification, in particular to a pedestrian re-identification method and device, an intelligent device and a storage medium. The method comprises the following steps: pre-processing a monitoring video collected at a current moment to obtain a target image and target audio corresponding to the target image; performing human body identification on the target image to obtain a target human body; performing footstep sound feature extraction on the target audio to obtain a target footstep sound feature; matching the target human body and the target footstep sound feature with historical human bodies and historical footstep sound features in a historical feature database, and performing pedestrian re-identification according to a matching result. The method can improve the identification accuracy of pedestrian re-identification and reduce misjudgment.

Description

Technical Field

[0001] This application relates to the field of target recognition technology, and in particular to a pedestrian re-identification method, apparatus, smart device, and storage medium. Background Technology

[0002] Pedestrian re-identification refers to the identification of target pedestrians in camera footage without overlapping scenes. It is a key technology for identifying specific pedestrians, utilizing computer vision techniques to determine the presence of a particular pedestrian in an image or video sequence.

[0003] In existing technologies, pedestrian re-identification is prone to misjudgment due to factors such as the single feature of the image and the difference in shooting angle, resulting in low accuracy and affecting the practicality of pedestrian re-identification.

[0004] Improving the accuracy of pedestrian re-identification and reducing misjudgments are issues that need to be considered. Summary of the Invention

[0005] In view of this, embodiments of this application provide a pedestrian re-identification method, apparatus, smart device, and storage medium, which can improve the accuracy of pedestrian re-identification and reduce misjudgments.

[0006] The first aspect of this application provides a pedestrian re-identification method, including:

[0007] The surveillance video captured at the current moment is preprocessed to obtain the target image and the target audio corresponding to the target image;

[0008] Perform human figure recognition on the target image to obtain the target human figure;

[0009] Footstep sound features are extracted from the target audio to obtain target footstep sound features;

[0010] The target human figure and the target footstep sound features are matched with historical human figures and historical footstep sound features in the historical feature database, and pedestrian re-identification is performed based on the matching results.

[0011] In one possible implementation of the first aspect, before matching the target human figure and the target footstep sound features with historical human figures and historical footstep sound features in a historical feature database, and performing pedestrian re-identification based on the matching result, the method includes:

[0012] Historical surveillance videos are preprocessed to obtain historical image sequences and historical audio sequences, wherein the historical surveillance videos are surveillance videos of a preset duration prior to the current moment;

[0013] Human features are identified in the historical images in the historical image sequence to obtain the historical human figures in each historical image in the historical image sequence;

[0014] Footstep sound features are extracted from the historical audio sequence to obtain the historical footstep sound features of each historical audio in the historical audio sequence.

[0015] Establish a correspondence between the historical humanoid figures and the characteristics of the historical footsteps;

[0016] Based on the historical human figures, the historical footsteps, and the corresponding relationships, a historical feature database is established.

[0017] In one possible implementation of the first aspect, the preprocessing of historical surveillance video to obtain historical image sequences and historical audio sequences includes:

[0018] The historical surveillance video is split into audio and video images to obtain separate audio and video images;

[0019] The video images are split into frames according to a preset frame rate to obtain a historical image sequence;

[0020] The audio is split according to a preset inter-frame interval to obtain a historical audio sequence.

[0021] In one possible implementation of the first aspect, the historical footstep features include historical frequency domain features and historical time domain features. The step of extracting footstep features from the historical audio sequence to obtain the historical footstep features of each historical audio segment in the historical audio sequence includes:

[0022] Obtain the historical energy spectrum of each historical footstep feature in the historical audio sequence;

[0023] Based on the historical energy spectrum, feature extraction is performed to obtain a historical frequency domain feature set, which includes the historical frequency domain features of each historical footstep sound feature separated from the historical audio sequence.

[0024] The historical energy spectrum is transformed in the time domain to obtain a historical time domain feature set, which includes the historical time domain features of each historical footstep sound feature separated from the historical audio sequence.

[0025] In one possible implementation of the first aspect, the step of performing human feature recognition on historical images in the historical image sequence to obtain historical human figures in each historical image in the historical image sequence includes:

[0026] Extract human features from each historical image in the historical image sequence;

[0027] Based on the similarity of human features and movement trends, the same human features in the historical image sequence are merged to obtain the historical human figures in the historical image sequence.

[0028] In one possible implementation of the first aspect, establishing the correspondence between the historical human figure and the historical footstep characteristics includes:

[0029] Obtain key point features of the historical human figure in each of the historical images in the historical image sequence;

[0030] Based on the key point features, determine the human posture features corresponding to the historical human figure;

[0031] Based on the human posture features corresponding to each historical human figure in the historical image sequence, determine the human posture change features corresponding to the historical image sequence.

[0032] Calculate the historical feature distance value between the historical footstep sound feature and each human posture feature in the human posture change feature, and determine the minimum historical feature distance value corresponding to the historical footstep sound feature;

[0033] If the minimum historical feature distance value is less than or equal to a preset historical distance threshold, then it is determined that the historical footstep feature matches the human posture feature corresponding to the minimum historical feature distance value.

[0034] Establish a correspondence between the historical footstep sound features and the historical human figures corresponding to the matched human posture features.

[0035] In one possible implementation of the first aspect, the step of matching the target human figure and the target footstep sound features with historical human figures and historical footstep sound features in a historical feature database, and performing pedestrian re-identification based on the matching results, includes:

[0036] A preset number of historical human figures are selected from the historical feature database, and the similarity between the preset number of historical human figures and the target human figure is within a preset similarity threshold range;

[0037] Obtain the minimum feature distance value, where the minimum feature distance value is the smallest among the feature distance values ​​between the target footstep feature and the historical footstep features corresponding to the preset number of historical human figures.

[0038] If the minimum feature distance value is less than or equal to a preset feature distance threshold, then it is determined that the historical feature database contains a historical humanoid that matches the target humanoid.

[0039] A second aspect of this application provides a pedestrian re-identification device, the device comprising:

[0040] The audio and video processing unit is used to preprocess the surveillance video collected at the current moment to obtain the target image and the target audio corresponding to the target image;

[0041] The target human figure acquisition unit is used to perform human figure recognition on the target image to obtain the target human figure;

[0042] The target footstep sound feature acquisition unit is used to extract footstep sound features from the target audio to obtain target footstep sound features;

[0043] The pedestrian re-identification unit is used to match the target human figure and the target footstep sound features with the historical human figure and historical footstep sound features in the historical feature database, and to perform pedestrian re-identification based on the matching results.

[0044] A third aspect of this application provides an intelligent device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the pedestrian re-identification method provided in the first aspect of this application.

[0045] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the pedestrian re-identification method provided in the first aspect of this application.

[0046] The fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the steps of the pedestrian re-identification method described in the first aspect of this application.

[0047] In this embodiment, the surveillance video captured at the current moment is preprocessed to obtain a target image and its corresponding target audio. Human figure recognition is performed on the target image to obtain a target human figure. Footstep sound features are extracted from the target audio to obtain target footstep sound features. The target human figure and target footstep sound features are matched with historical human figures and historical footstep sound features in a historical feature database, and pedestrian re-identification is performed based on the matching results. This application's solution combines human figure and footstep sound features for pedestrian re-identification, which can improve the accuracy of pedestrian re-identification in different scenarios (such as different shooting angles, different shooting locations, and different shooting lighting), expand the application scenarios of pedestrian re-identification, and reduce false positives. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the implementation of the pedestrian re-identification method provided in the embodiments of this application;

[0050] Figure 2 This is a flowchart illustrating a specific implementation of the pedestrian re-identification method provided in this application for establishing the historical feature database;

[0051] Figure 3 This is a flowchart illustrating a specific implementation of the pedestrian re-identification method provided in this application, which obtains historical image sequences and historical audio sequences.

[0052] Figure 4 This is a flowchart illustrating a specific implementation of the pedestrian re-identification method provided in this application for obtaining historical human figures;

[0053] Figure 5 This is a flowchart illustrating a specific implementation of the pedestrian re-identification method provided in this application for extracting historical footstep features;

[0054] Figure 6 This is a flowchart illustrating a specific implementation of the pedestrian re-identification method provided in this application, which establishes the correspondence between the historical human figure and the historical footstep sound features.

[0055] Figure 7 This is a flowchart illustrating a specific implementation of step S104 in the pedestrian re-identification method provided in this application embodiment;

[0056] Figure 8 This is a structural block diagram of the pedestrian re-identification device provided in the embodiments of this application;

[0057] Figure 9 This is a schematic diagram of a smart device provided in an embodiment of this application. Detailed Implementation

[0058] In the following description, specific details such as particular device structures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0059] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0060] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0061] It should be understood that the various method embodiments of this application provide a pedestrian re-identification method applicable to various types of smart devices that require pedestrian re-identification, such as personal computers, laptops, smartphones, tablets, servers, and other smart devices.

[0062] The pedestrian re-identification method provided in this application will be described exemplarily below with reference to specific embodiments.

[0063] Figure 1 The implementation flow of the pedestrian re-identification method provided in this application embodiment is illustrated. The execution end of this application embodiment can be a smart device. The method flow may include the following steps S101 to S104.

[0064] Step S101: Preprocess the surveillance video collected at the current moment to obtain the target image and the target audio corresponding to the target image.

[0065] The above preprocessing includes performing audio-video splitting on the surveillance video, separating the audio and video images to obtain the target image at the current moment and its corresponding target audio.

[0066] Generally, human walking speed ranges from approximately 0.3 m / s to 1.5 m / s, and stride length ranges from approximately 0.3 m to 0.8 m; therefore, the stride frequency can be roughly considered to be 1 step / s to 3 steps / s. The target image includes at least one target human figure, and the target audio includes at least one complete footstep sound.

[0067] Step S102: Perform human figure recognition on the target image to obtain the target human figure.

[0068] In this embodiment, the target human figure can be determined by extracting human features from the target image. The algorithm for human figure recognition can refer to existing technologies and will not be elaborated here.

[0069] Step S103: Extract footstep features from the target audio to obtain target footstep features.

[0070] The target footstep sound features correspond to the target human figure. The target footstep sound features include target frequency domain features and target time domain features. In one possible implementation, the energy spectrum of the target footstep sound is separated from the target audio, and based on the energy spectrum, the spectral features of the energy spectrum are extracted to obtain the target frequency domain features of the target footstep sound features; the frequency domain features are then transformed into the time domain to obtain the target time domain features of the target footstep sound.

[0071] Step S104: Match the target human figure and the target footstep sound features with the historical human figures and historical footstep sound features in the historical feature database, and perform pedestrian re-identification based on the matching results.

[0072] The historical feature database stores historical human figures and their corresponding historical footstep sound features. In this embodiment, the target human figure is matched with historical human figures in the historical feature database, and the target footstep sound features are matched with the historical footstep sound features of the matched historical human figures. Pedestrian re-identification is then performed based on the matching results.

[0073] In this embodiment of the application, by combining human figure and footstep sound features for pedestrian re-identification, the accuracy of pedestrian re-identification can be improved, the application scenarios of pedestrian re-identification can be expanded, and false judgments can be reduced.

[0074] As one possible implementation of this application, before the step of matching the target human figure and the target footstep sound features with historical human figures and historical footstep sound features in the historical feature database, and performing pedestrian re-identification based on the matching results, the historical feature database is established in advance. Figure 2 The following is a detailed implementation process of establishing the historical feature database in the method provided in the embodiments of this application:

[0075] S201: Preprocess the historical surveillance video to obtain a historical image sequence and a historical audio sequence, wherein the historical surveillance video is a surveillance video of a preset duration prior to the current moment.

[0076] In this embodiment, historical surveillance video is acquired and preprocessed to obtain historical image sequences and historical audio sequences. To extract footstep features, the historical surveillance video includes at least one complete footstep sound; generally, the preset duration is not less than 2 seconds.

[0077] As one possible implementation of this application Figure 3 The following is a detailed implementation process of preprocessing historical surveillance video to obtain historical image sequences and historical audio sequences, as shown in the method embodiment provided in this application:

[0078] A1: Perform audio and video splitting processing on the historical surveillance video to obtain separate audio and video images.

[0079] A2: The video images are split into frames according to a preset frame rate to obtain a historical image sequence. The preset frame rate can be 25fps.

[0080] A3: The audio is split according to a preset inter-frame interval to obtain a historical audio sequence. The preset inter-frame interval is associated with the preset frame rate.

[0081] In some implementations, the preset inter-frame interval is 1 / f, where f is the preset frame rate. For example, when the preset frame rate is 25fps, the preset inter-frame interval is 1 / 25fps = 40ms.

[0082] For example, the historical surveillance video two seconds prior to the current moment is split into audio and video segments, resulting in two seconds of video images and two seconds of audio. The video images are then divided into frames at a preset frame rate f = 25fps to form a historical image sequence containing 2f historical images. The two seconds of audio are then divided into historical audio segments at a preset inter-frame interval of 1 / 25fps = 40ms to form a historical audio sequence containing 2f audio segments.

[0083] S202: Perform human feature recognition on the historical images in the historical image sequence to obtain the historical human figures in each historical image in the historical image sequence.

[0084] As one possible implementation of this application Figure 4 This application provides an embodiment of a method for performing human feature recognition on historical images in a historical image sequence to obtain historical human figures in each historical image in the historical image sequence. A detailed description follows:

[0085] B1: Extract human features from each historical image in the historical image sequence. The specific algorithm for human feature extraction can be found in existing technologies and will not be elaborated here.

[0086] B2: Based on the similarity of human features and movement trends, merge the same human features in the historical image sequence to obtain the historical human figures in the historical image sequence.

[0087] For example, a publicly available human figure recognition algorithm is used on a single historical image to separate and extract each human figure feature. Based on the similarity of human figure features and movement trends, the entire historical image sequence I is merged. T The same humanoid features are denoted as H. t = [h1, ..., h n ], where n is the historical image sequence I T The number of different humanoid figures included.

[0088] In this embodiment, human features are merged based on similarity and movement trends to determine the historical human figures and their number in the historical image sequence.

[0089] S203: Extract footstep features from the historical audio sequence to obtain the historical footstep features of each historical audio in the historical audio sequence.

[0090] As one possible implementation of this application, the historical footstep features include historical frequency domain features and historical time domain features. Figure 5 The following is a detailed implementation process of extracting footstep features from the historical audio sequence to obtain the historical footstep features of each historical audio segment in the historical audio sequence, as provided in the method embodiment of this application:

[0091] C1: Obtain the historical energy spectrum of each historical footstep feature in the historical audio sequence.

[0092] C2: Based on the historical energy spectrum, feature extraction is performed to obtain a historical frequency domain feature set, which includes the historical frequency domain features of each historical footstep sound feature separated from the historical audio sequence.

[0093] In this embodiment, the frequency domain features include spectral features. For the historical audio sequence, the historical energy spectrum (CP) of each historical footstep is separated. t = [cp1, ..., cp k ], extract the historical spectral features F of each historical energy spectrum respectively. t = [f1, ..., f k ], where k is the number of spectral features contained in the historical audio sequence.

[0094] For example, in this embodiment of the application, a combination of gait frequency, linear prediction cepstral coefficients, loudness, and Mel cepstral coefficients can be selected as the spectral features of the historical footsteps. Including different features helps to improve the distinguishability.

[0095] C3: Perform time-domain transformation on the historical energy spectrum to obtain a historical time-domain feature set, which includes the historical time-domain features of each historical footstep sound feature separated from the historical audio sequence.

[0096] In this embodiment, the historical energy spectrum is transformed in the time domain to extract the time-domain waveform energy changes of historical footsteps (reflecting the intervals and durations of footsteps) and quantize them into the time-domain features t of historical footsteps. i =[e i,1 …,e i,2f If the i-th historical footstep does not exist in the j-th audio segment, then e i,j The value is 0. If the historical footstep exists, e is quantized according to the preset interval. i,j Assign a value. For example, if the quantization interval is 2 and the current energy is 7, e i,j The value is 7 divided by 2, which is e i,j The specific value is 3.

[0097] In this embodiment, the quantization interval can be adjusted according to the business scenario. For example, it can be differentiated by indoor and outdoor dimensions, or further subdivided into residential buildings, office buildings, supermarkets, warehouses, shopping malls, squares, stations, parks, streets, etc. Indoor monitoring distances are usually shorter and footstep energy is stronger, while outdoor distances are the opposite. The quantization interval can be selected appropriately based on the statistical patterns of different scenarios and the microphone's own pickup performance to maximize the distinguishability and robustness of footstep characteristics.

[0098] In this embodiment, the historical footstep features consist of the historical frequency domain features and the historical time domain features, that is, the historical footstep feature set S corresponding to the historical audio sequence. T = [S1, ..., S k ], S k =[f k , t k ].

[0099] S204: Establish the correspondence between the historical humanoid figure and the historical footstep characteristics.

[0100] As one possible implementation of this application Figure 6 The following is a detailed implementation process for establishing the correspondence between the historical human figure and the historical footstep characteristics in the method embodiment provided in this application:

[0101] D1: Obtain the key point features of the historical human figure in each of the historical images in the historical image sequence.

[0102] In this embodiment, publicly available human pose estimation methods such as Direct Pose and PifPaf can be used to extract key point features. The key point features include the coordinates of key points of the human leg and arm joints (more key point features can be added depending on the specific application scenario).

[0103] D2: Based on the key point features, determine the human posture features corresponding to the historical human figure.

[0104] D3: Based on the human posture features corresponding to each historical human figure in the historical image sequence, determine the human posture change features corresponding to the historical image sequence.

[0105] Starting from the second image, calculate the Euclidean distance between the coordinates of the leg and arm keypoints in the current image and the corresponding keypoint coordinates in the previous image. Similar to footstep feature extraction, quantify the leg and arm changes of the i-th human figure in the j-th image according to a preset scale (adjustable according to the business scenario) to determine the human posture feature b of the i-th human figure in the j-th image. i,j =[l i,j a i,j If ], then the human posture characteristics of the i-th human figure are b i =[b i,1 , ..., b i,2f Historical image sequence I T The characteristics of human posture changes are B t = [b1, ..., b n ].

[0106] D4: Calculate the historical feature distance value between the historical footstep sound feature and each human posture feature in the human posture change feature, and determine the minimum historical feature distance value corresponding to the historical footstep sound feature.

[0107] In one possible implementation, the historical temporal features of the historical footsteps are calculated, and the historical feature distance values ​​of the leg and arm changes in the human posture change features are determined, and the minimum historical feature distance value is determined.

[0108] For example, the Dynamic Time Warping (DTW) algorithm can be used to calculate the minimum historical feature distance value between each footstep time-domain change feature and each leg and arm change feature in the human posture change features. The human posture change feature corresponding to the minimum historical feature distance value is matched with the historical matching feature of the historical footstep feature.

[0109] In some implementations, the DTW algorithm is used to calculate the first minimum Euclidean distance between the historical footstep sound features and the leg change features, and then the DTW algorithm is used to calculate the second minimum Euclidean distance between the footstep sound features and the arm change features. The sum of the first minimum Euclidean distance and the second minimum Euclidean distance is calculated, and the sum of this distance is used as the minimum historical feature distance value between the historical footstep sound features and the human posture change features. The human posture change features corresponding to the determined minimum historical feature distance value are matched with the historical matching features of the historical footstep sound features.

[0110] D5: If the minimum historical feature distance value is less than or equal to the preset historical distance threshold, then it is determined that the historical footstep feature matches the human posture feature corresponding to the minimum historical feature distance value.

[0111] D6: Establish the correspondence between the historical footstep sound features and the historical human figures corresponding to the matched human posture features.

[0112] In this embodiment, based on the historical feature distance values ​​between various human posture features in the historical footstep sound features and human posture change features, the human posture features that match the historical footstep sound features are determined, which can improve the accuracy of historical footstep sound feature matching and human posture feature matching, and enhance the effectiveness of establishing the correspondence between historical footstep sound features and historical human figures.

[0113] S205: Based on the historical human figures, the historical footsteps features, and the corresponding relationships, establish the historical feature database.

[0114] In this embodiment, the historical humanoid h is used. j As the key, based on historical frequency domain features f j Historical temporal characteristics t j Store it in the database as a value.

[0115] As one possible implementation of this application Figure 7 The present application illustrates a specific implementation process in which the target human figure and the target footstep sound features are matched with historical human figures and historical footstep sound features in a historical feature database, and pedestrian re-identification is performed based on the matching results. The details are as follows:

[0116] E1: Select a preset number of historical human figures from the historical feature database, and the similarity between the preset number of historical human figures and the target human figure is within a preset similarity threshold range.

[0117] E2: Obtain the minimum feature distance value, where the minimum feature distance value refers to the minimum feature distance value among the target footstep feature and the historical footstep features corresponding to the preset number of historical human figures.

[0118] E3: If the minimum feature distance value is less than or equal to the preset feature distance threshold, then it is determined that there is a historical humanoid in the historical feature database that matches the target humanoid.

[0119] In one possible implementation, if the minimum feature distance value is greater than the preset feature distance threshold, the target human figure and its target footstep sound features are stored in the historical feature database.

[0120] For example, suppose the human figure re-identification algorithm detects human figure p at the current moment. i Then, based on the preset similarity threshold range, select the pair with p. i Similar historical doll group P, including a preset number (no more than 10) of historical dolls P j Then read the humanoid p respectively i Footstep characteristics [f i , t i ], and the characteristics of historical footsteps in historical humanoid group P [[f j , t j ],…[f k , t k Similarly, the DTW algorithm is used to find the match with f. i t i The f with the smallest sum of distances j t j If the sum of the minimum distances is less than or equal to a preset feature distance threshold, then a matching human figure is considered to exist, and the human figure p is set to... i Classified as a historical figure P j If the sum of the minimum distances is greater than a preset feature distance threshold, then it is considered that there is no matching human figure, and the human figure p is removed. i Add to the historical doll group P and store p. i The characteristics of footsteps.

[0121] In one possible implementation, human figure recognition is performed on the target image to obtain more than one target human figure, and / or footstep sound feature extraction is performed on the target audio to obtain more than one target footstep sound feature. In this scenario, a correspondence between the target human figure and the target footstep sound feature needs to be established. The establishment of this correspondence can refer to the method for establishing the correspondence between historical human figures and historical footstep sound features in step S204, which will not be elaborated here. In this scenario, the corresponding target human figure and target footstep sound feature are taken as a set of human figure footstep sound features. Each set of human figure footstep sound features is matched with historical human figures and historical footstep sound features in the historical feature database, and pedestrian re-identification is performed based on the matching results, thereby realizing the re-identification of multiple pedestrians.

[0122] In this embodiment, the surveillance video captured at the current moment is preprocessed to obtain a target image and its corresponding target audio. Human figure recognition is performed on the target image to obtain a target human figure. Footstep sound features are extracted from the target audio to obtain target footstep sound features. The target human figure and target footstep sound features are matched with historical human figures and historical footstep sound features in a historical feature database, and pedestrian re-identification is performed based on the matching results. This application's solution combines human figure and footstep sound features for pedestrian re-identification, which can improve the accuracy of pedestrian re-identification in different scenarios (such as different shooting angles, different shooting locations, and different shooting lighting), expand the application scenarios of pedestrian re-identification, and reduce false positives.

[0123] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0124] Corresponding to the pedestrian re-identification method described in the above embodiments, Figure 8 The diagram shows a structural block diagram of the pedestrian re-identification device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0125] Reference Figure 8 The pedestrian re-identification device is applied to smart devices. The device includes: an audio / video processing unit 81, a target human figure acquisition unit 82, a target footstep sound feature acquisition unit 83, and a pedestrian re-identification unit 84, wherein:

[0126] The audio and video processing unit 81 is used to preprocess the surveillance video collected at the current moment to obtain the target image and the target audio corresponding to the target image;

[0127] The target human figure acquisition unit 82 is used to perform human figure recognition on the target image to obtain the target human figure;

[0128] The target footstep sound feature acquisition unit 83 is used to extract footstep sound features from the target audio to obtain target footstep sound features;

[0129] The pedestrian re-identification unit 84 is used to match the target human figure and the target footstep sound features with the historical human figure and historical footstep sound features in the historical feature database, and to perform pedestrian re-identification based on the matching results.

[0130] As one possible implementation of this application, the pedestrian re-identification device further includes:

[0131] The historical audio and video processing unit is used to preprocess historical surveillance videos to obtain historical image sequences and historical audio sequences. The historical surveillance videos are surveillance videos of a preset duration prior to the current moment.

[0132] The historical human figure acquisition unit is used to perform human figure feature recognition on historical images in the historical image sequence to obtain the historical human figures of each historical image in the historical image sequence;

[0133] The historical footstep feature acquisition unit is used to extract footstep features from the historical audio sequence to obtain the historical footstep features of each historical audio in the historical audio sequence.

[0134] A correspondence establishment unit is used to establish a correspondence between the historical human figure and the historical footstep characteristics;

[0135] The historical database establishment unit is used to establish the historical feature database based on the historical human figures, the historical footstep characteristics, and the corresponding relationships.

[0136] As one possible implementation of this application, the aforementioned historical audio and video processing unit includes:

[0137] The historical audio and video splitting module is used to split the historical surveillance video into separate audio and video images.

[0138] The first sequence acquisition module is used to split the video image into frames according to a preset frame rate to obtain a historical image sequence;

[0139] The second sequence acquisition module is used to split the audio according to a preset inter-frame interval to obtain a historical audio sequence.

[0140] As one possible implementation of this application, the aforementioned historical footstep feature acquisition unit includes:

[0141] The historical energy spectrum acquisition module is used to acquire the historical energy spectrum of each historical footstep feature in the historical audio sequence;

[0142] The first feature acquisition module is used to extract features based on the historical energy spectrum and acquire a historical frequency domain feature set, wherein the historical frequency domain feature set includes the historical frequency domain features of each historical footstep feature separated from the historical audio sequence;

[0143] The second feature acquisition module is used to perform time-domain conversion on the historical energy spectrum to obtain a historical time-domain feature set, which includes the historical time-domain features of each historical footstep sound feature separated from the historical audio sequence.

[0144] As one possible implementation of this application, the aforementioned historical humanoid acquisition unit includes:

[0145] The first feature extraction module is used to extract human features from each historical image in the historical image sequence;

[0146] The historical human figure acquisition module is used to merge the same human figure features in the historical image sequence based on the similarity of human figure features and movement trends, so as to obtain the historical human figures in the historical image sequence.

[0147] As one possible implementation of this application, the above-mentioned correspondence establishment unit includes:

[0148] The second feature acquisition module is used to acquire key point features of the historical human figure in each of the historical images in the historical image sequence;

[0149] The posture feature determination module is used to determine the human posture features corresponding to the historical human figure based on the key point features.

[0150] The posture change feature determination module is used to determine the human posture change features corresponding to the historical image sequence based on the human posture features corresponding to each historical human figure in the historical image sequence.

[0151] The minimum historical value determination module is used to calculate the historical feature distance value between the historical footstep sound feature and each human posture feature in the human posture change feature, and to determine the minimum historical feature distance value corresponding to the historical footstep sound feature.

[0152] The historical feature matching module is used to determine whether the historical footstep feature matches the human posture feature corresponding to the minimum historical feature distance value if the minimum historical feature distance value is less than or equal to a preset historical distance threshold.

[0153] The correspondence establishment module is used to establish the correspondence between the historical footstep sound features and the historical human figures corresponding to the matched human posture features.

[0154] As one possible implementation of this application, the pedestrian re-identification unit 84 includes:

[0155] The historical doll filtering module is used to filter out a preset number of historical dolls from the historical feature database, wherein the similarity between the preset number of historical dolls and the target doll is within a preset similarity threshold range;

[0156] The minimum feature distance value determination module is used to obtain the minimum feature distance value, which refers to the minimum feature distance value among the target footstep sound features and the historical footstep sound features corresponding to the preset number of historical human figures.

[0157] The human figure matching module is used to determine that if the minimum feature distance value is less than or equal to a preset feature distance threshold, the historical feature database contains a historical human figure that matches the target human figure.

[0158] In this embodiment, the surveillance video captured at the current moment is preprocessed to obtain a target image and its corresponding target audio. Human figure recognition is performed on the target image to obtain a target human figure. Footstep sound features are extracted from the target audio to obtain target footstep sound features. The target human figure and target footstep sound features are matched with historical human figures and historical footstep sound features in a historical feature database, and pedestrian re-identification is performed based on the matching results. This application's solution combines human figure and footstep sound features for pedestrian re-identification, which can improve the accuracy of pedestrian re-identification in different scenarios (such as different shooting angles, different shooting locations, and different shooting lighting), expand the application scenarios of pedestrian re-identification, and reduce false positives.

[0159] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figures 1 to 7 The steps of any pedestrian re-identification method are represented.

[0160] This application embodiment also provides a smart device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements... Figures 1 to 7 The steps of any pedestrian re-identification method are represented.

[0161] This application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute the implementation of... Figures 1 to 7 The steps of any pedestrian re-identification method are represented.

[0162] Figure 9 This is a schematic diagram of a smart device provided in an embodiment of this application. Figure 9 As shown, the smart device 9 of this embodiment includes: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90. When the processor 90 executes the computer program 92, it implements the steps in the embodiments of the various pedestrian re-identification methods described above, for example... Figure 1 Steps S101 to S104 are shown. Alternatively, when the processor 90 executes the computer program 92, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 8 The functions of units 81 to 84 are shown.

[0163] The computer program 92 can be divided into one or more modules / units, which are stored in the memory 91 and executed by the processor 90 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 92 in the smart device 9.

[0164] The processor 90 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0165] The memory 91 can be an internal storage unit of the smart device 9, such as a hard drive or memory of the smart device 9. The memory 91 can also be an external storage device of the smart device 9, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the smart device 9. Furthermore, the memory 91 can include both internal and external storage units of the smart device 9. The memory 91 is used to store the computer program and other programs and data required by the smart device. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0167] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0168] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0169] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0174] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A pedestrian re-identification method, characterized in that, The method comprises: preprocessing a monitoring video collected at a current moment to obtain a target image and a target audio corresponding to the target image; performing human shape recognition on the target image to obtain a target human shape; extracting a footstep sound feature from the target audio to obtain a target footstep sound feature; preprocessing a historical monitoring video to obtain a historical image sequence and a historical audio sequence, the historical monitoring video being a monitoring video of a preset time length before the current moment; performing human shape feature recognition on a historical image in the historical image sequence to obtain a historical human shape of each historical image in the historical image sequence; extracting a footstep sound feature from the historical audio sequence to obtain a historical footstep sound feature of each historical audio in the historical audio sequence; establishing a corresponding relationship between the historical human shape and the historical footstep sound feature, comprising: obtaining a key point feature of the historical human shape in each historical image in the historical image sequence; determining a human posture feature corresponding to the historical human shape according to the key point feature; determining a human posture change feature corresponding to the historical image sequence based on the human posture feature corresponding to each historical human shape in the historical image sequence; calculating a historical feature distance value between the historical footstep sound feature and each human posture feature in the human posture change feature, and determining a minimum historical feature distance value corresponding to the historical footstep sound feature; if the minimum historical feature distance value is less than or equal to a preset historical distance threshold, it is determined that the historical footstep sound feature matches the human posture feature corresponding to the minimum historical feature distance value; establishing a corresponding relationship between the historical footstep sound feature and the historical human shape corresponding to the matched human posture feature; and establishing the historical feature database based on the historical human shape, the historical footstep sound feature and the corresponding relationship; matching the target human shape and the target footstep sound feature with the historical human shape and the historical footstep sound feature in the historical feature database, and performing pedestrian re-identification according to the matching result.

2. The method of claim 1, wherein, The preprocessing of the historical monitoring video to obtain the historical image sequence and the historical audio sequence comprises: performing audio and video splitting processing on the historical monitoring video to obtain separated audio and video images; frame splitting the video images at a preset frame rate to obtain a historical image sequence; splitting the audio at a preset interframe interval to obtain a historical audio sequence.

3. The method of claim 1, wherein, The historical footstep sound feature comprises a historical frequency domain feature and a historical time domain feature, and the extraction of the footstep sound feature from the historical audio sequence to obtain the historical footstep sound feature of each historical audio in the historical audio sequence comprises: obtaining a historical energy spectrum of each historical footstep sound feature in the historical audio sequence; performing feature extraction based on the historical energy spectrum to obtain a historical frequency domain feature set, the historical frequency domain feature set comprising historical frequency domain features of each historical footstep sound feature separated from the historical audio sequence; The historical energy spectrum is time domain converted to obtain a historical time domain feature set, which includes historical time domain features of each historical footstep sound feature separated from the historical audio sequence.

4. The method of claim 1, wherein, The historical image in the historical image sequence is subjected to human body feature recognition to obtain a historical human body of each historical image in the historical image sequence, including: Human body features in each historical image in the historical image sequence are extracted; The same human body features in the historical image sequence are merged according to human body feature similarity and motion trend to obtain historical human bodies in the historical image sequence.

5. The method according to any one of claims 1 to 4, characterized in that, The target human body and the target footstep sound feature are matched with historical human bodies and historical footstep sound features in a historical feature database, and pedestrian re-identification is performed according to a matching result, including: A preset number of historical human bodies are screened out from the historical feature database, and the preset number of historical human bodies have a similarity to the target human body within a preset similarity threshold range; A minimum feature distance value is obtained, which is a minimum feature distance value of the target footstep sound feature and historical footstep sound features corresponding to the preset number of historical human bodies; If the minimum feature distance value is less than or equal to a preset feature distance threshold value, it is determined that the historical feature database has a historical human body matching the target human body.

6. A pedestrian re-identification apparatus characterized by comprising: The device includes: An audio and video processing unit configured to pre-process a monitoring video collected at a current time to obtain a target image and a target audio corresponding to the target image; A target human body acquisition unit configured to perform human body recognition on the target image to obtain a target human body; A target footstep sound feature acquisition unit configured to extract a footstep sound feature from the target audio to obtain a target footstep sound feature; A historical audio and video processing unit configured to pre-process a historical monitoring video to obtain a historical image sequence and a historical audio sequence, the historical monitoring video being a monitoring video of a preset time length before the current time; A historical human body acquisition unit configured to perform human body feature recognition on a historical image in the historical image sequence to obtain a historical human body of each historical image in the historical image sequence; A historical footstep sound feature acquisition unit configured to extract a footstep sound feature from the historical audio sequence to obtain a historical footstep sound feature of each historical audio in the historical audio sequence; A corresponding relationship establishment unit configured to establish a corresponding relationship between the historical human body and the historical footstep sound feature, the corresponding relationship establishment unit including: The second feature acquisition module is configured to acquire key point features of the historical human form in each historical image of the historical image sequence; the posture feature determination module is configured to determine human posture features corresponding to the historical human form according to the key point features; the posture change feature determination module is configured to determine human posture change features corresponding to the historical image sequence based on the human posture features corresponding to each historical human form in the historical image sequence; the minimum historical value determination module is configured to calculate historical feature distance values between the historical footstep sound features and each human posture feature in the human posture change features, and determine minimum historical feature distance values corresponding to the historical footstep sound features; the historical feature matching module is configured to determine that the historical footstep sound features match the human posture features corresponding to the minimum historical feature distance values if the minimum historical feature distance values are less than or equal to a preset historical distance threshold; the corresponding relationship establishment module is configured to establish a corresponding relationship of the historical human form between the historical footstep sound features and the matched human posture features; and the historical database establishment unit is configured to establish the historical feature database based on the historical human form, the historical footstep sound features, and the corresponding relationship. The pedestrian re-identification unit is configured to match the target human form and the target footstep sound features with historical human forms and historical footstep sound features in the historical feature database, and perform pedestrian re-identification according to a matching result.

7. An intelligent device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the pedestrian re-identification method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the steps of the pedestrian re-identification method according to any one of claims 1 to 5.

Images