Pedestrian re-identification device and method

Abstract

To provide a pedestrian re-identification device, method, and computer storage medium, which improve accuracy of pedestrian re-identification in scenes where a pedestrian changed his / her posture significantly or is identified by different cameras.SOLUTION: A pedestrian re-identification device 100 is provided, comprising: a pedestrian detection unit configured to detect pedestrians from each video frame of a video sequence; a feature extraction unit configured to extract appearance features of each pedestrian detected in the video frames; and a pedestrian identification unit configured to match extracted appearance features of each pedestrian with target appearance features of a target pedestrian, and identify the target pedestrian in the video frames based on a matching result. The target appearance features of the target pedestrian are generated based on initial appearance features and most recent appearance features of the target pedestrian.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to image processing, and in particular to pedestrian re-identification devices, methods and computer storage media. [Background technology]

[0002] Pedestrian re-identification is a technique for determining whether a specific target pedestrian exists among image or video sequences from multiple overlapping or non-overlapping cameras of different fields of view. Different from normal pedestrian tracking, pedestrian re-identification can identify a specific target pedestrian in image or video sequences captured by different cameras, so as to enable long-term tracking and monitoring of the specific target pedestrian, and therefore has great application prospects in surveillance fields, crime detection tasks, etc. Summary of the Invention [Problem to be solved by the invention]

[0003] Currently, pedestrian re-identification generally involves matching the features of each pedestrian detected from an image or video sequence with the features of a known target pedestrian in a pre-constructed target pedestrian library to determine whether the detected pedestrian corresponds to a known target pedestrian. However, this is often ineffective, mainly because the posture and appearance of the same pedestrian in images captured by different cameras may be significantly different due to differences in the shooting angles, field of view, lighting conditions, etc. between different cameras; pedestrians are often highly mobile, so their posture and appearance will also change slightly over time; and the features of a pre-constructed target pedestrian are uniform, making it easy to cause misidentification when the posture or appearance of the same pedestrian changes. [Means for solving the problem]

[0004] According to one aspect of the present disclosure, a pedestrian re-identification device is provided, comprising: a pedestrian detection unit configured to detect a pedestrian in each video frame of a video sequence; a feature extraction unit configured to extract, for each pedestrian detected from the video frames, appearance features; and a pedestrian identification unit configured to match the extracted appearance features of each pedestrian with target appearance features of a target pedestrian and identify the target pedestrian from the video frames based on the matching result, wherein the target appearance features of the target pedestrian are generated based on initial appearance features and recent appearance features of the target pedestrian.

[0005] According to another aspect of the present disclosure, there is provided a pedestrian re-identification device comprising a processor and a memory having computer program instructions stored thereon, the computer program instructions, when executed by the processor, causing the processor to perform the steps of: detecting pedestrians in each video frame of a video sequence; extracting appearance features for each detected pedestrian from the video frames; matching the extracted appearance features of each pedestrian with target appearance features of a target pedestrian, and identifying a target pedestrian from the video frames based on the matching result, wherein the target appearance features of the target pedestrian are generated based on initial appearance features and recent appearance features of the target pedestrian.

[0006] According to another aspect of the present disclosure, there is provided a pedestrian re-identification method comprising: detecting pedestrians in each video frame of a video sequence; extracting appearance features for each detected pedestrian from the video frames; matching the extracted appearance features of each pedestrian with target appearance features of a target pedestrian; and identifying the target pedestrian from the video frames based on the matching result, wherein the target appearance features of the target pedestrian are generated based on initial appearance features and recent appearance features of the target pedestrian.

[0007] According to another aspect of the present disclosure, there is provided a computer readable storage medium having computer program instructions stored thereon, the computer program instructions, when executed by a processor, performing the pedestrian re-identification method as described above.

[0008] The above-described device, method, and computer-readable storage medium of the present disclosure improve the accuracy of re-identification for a pedestrian when the pedestrian's pose changes significantly or is identified by a different camera. [Brief description of the drawings]

[0009] These and / or other aspects and advantages of the present disclosure will become more apparent and will be more readily understood from the following detailed description of exemplary embodiments of the present disclosure taken in conjunction with the accompanying drawings. [Figure 1] 1 shows an example of misidentification when re-identifying a pedestrian based on the initial appearance features of a target pedestrian. [Diagram 2] An example of misidentification when re-identifying a pedestrian based on the recent appearance features of a target pedestrian is shown. [Diagram 3] FIG. 1 illustrates a block diagram of a configuration example of a pedestrian re-identification device according to an embodiment of the present disclosure. [Figure 4] FIG. 2 shows a schematic diagram of a pedestrian re-identification device according to an embodiment of the present disclosure for re-identifying a pedestrian based on target appearance features of a target pedestrian; [Diagram 5] FIG. 2 illustrates a hardware block diagram of an example of a computing device for implementing a pedestrian re-identification device according to an embodiment of the present disclosure. [Figure 6] 1 shows a schematic flow chart of a pedestrian re-identification method according to an embodiment of the present disclosure. [Figure 7] 1A and 1B are schematic diagrams showing verification results of pedestrian re-identification methods according to the related art and an embodiment of the present disclosure; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In order to make the present disclosure more fully understood by those skilled in the art, the present disclosure will now be described in more detail with reference to the accompanying drawings and detailed embodiments.

[0011] The technical solutions in the embodiments of the present disclosure are clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is clear that the described embodiments are only some embodiments of the present disclosure, and are not all embodiments. All other embodiments that can be obtained by those skilled in the art based on the embodiments of the present disclosure without creative labor belong to the protection scope of the present disclosure.

[0012] As described above, in the existing pedestrian re-identification method, the features of each pedestrian detected from an image sequence or a video sequence are generally matched with the features of a known target pedestrian in a pre-constructed target pedestrian library to determine whether or not the detected pedestrian corresponds to a known target pedestrian. However, due to various reasons such as changes in the posture of the pedestrian, the conventional pedestrian re-identification method is prone to misidentifying pedestrians. In particular, when the features of the target pedestrian library for feature matching are single, if the posture or appearance of the same pedestrian changes significantly, the pedestrian may be misidentified as another pedestrian, and further, the misidentification may continue. Below, an example of misidentification of a pedestrian that may occur when re-identifying a pedestrian will be described with reference to FIG. 1 and FIG. 2.

[0013] FIG. 1 shows an example of pedestrian misidentification that occurs when feature matching is performed based on the initial appearance features of a target pedestrian. In this case, the features of a target pedestrian in a target pedestrian library at the time of first appearance are stored in the target pedestrian library as the initial appearance features of the target pedestrian ID, and thereafter, matching is performed with the initial appearance features of the target pedestrian. As shown in FIG. 1, among three (contiguous or non-contiguous) images of the same pedestrian taken by cameras with overlapping or non-overlapping fields of view, the target pedestrian is identified as ID1 in the first image, and the features in the first image are used as the initial appearance features of the target pedestrian ID1 for subsequent matching. However, since the posture of the pedestrian and the shooting angle change significantly after that, the target pedestrian is misidentified as ID2 in the second image, and again as ID3 in the third image. In this case, it can be seen that the initial appearance features of the target pedestrian used for matching are constant, and therefore it is not possible to respond well to situations in which the pedestrian posture and the shooting angle change significantly.

[0014] FIG. 2 is a diagram showing an example of pedestrian misidentification that occurs when feature matching is performed based on the most recent appearance feature of the target pedestrian. In this case, the feature of the target pedestrian when it most recently appeared is stored in the target pedestrian library as the most recent appearance feature of the target pedestrian ID, the feature of the target pedestrian when it appeared the previous time is deleted, and thereafter, matching is performed with the most recent appearance feature of the target pedestrian. As shown in FIG. 2, among three images of the same pedestrian taken by cameras with overlapping or non-overlapping fields of view, the target pedestrian may be identified as ID1 in the first image, and the feature of the current appearance may be set as the most recent appearance feature of the target pedestrian ID1. Then, for various reasons such as pedestrians overlapping, the target pedestrian may be misidentified as ID2 in the second image, and the feature of the current appearance may be set as the most recent appearance feature of the target pedestrian ID2 to replace the previously stored appearance feature, and the erroneous result of identifying the target pedestrian ID1 as ID2 in the third image will be maintained. In this case, matching based on recent appearance features can accommodate changes in pedestrian posture and imaging angle to some extent, but if misidentification occurs due to overlapping pedestrians or other reasons, the result of the misidentification tends to be maintained without being corrected.

[0015] In view of such circumstances, in the present disclosure, a feature of a target pedestrian is constructed by comprehensively taking into consideration the initial appearance feature and the most recent appearance feature of the target pedestrian, and this is used to match the feature with the feature of a detected pedestrian, thereby making it possible to provide identification results with sufficient accuracy even in scenes where the pedestrian's posture has changed significantly or where the pedestrian is identified using a different camera.

[0016] Hereinafter, a pedestrian re-identification device according to an embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 shows a block diagram of a configuration example of a pedestrian re-identification device 100 according to an embodiment of the present disclosure. As shown in Fig. 3, the pedestrian re-identification device 100 may include a pedestrian detection unit 110, a feature extraction unit 120, and a pedestrian identification unit 130. Hereinafter, main functions of each unit of the pedestrian re-identification device 100 will be described.

[0017] The pedestrian detection unit 110 can detect a pedestrian from each video frame of a video sequence. As described above, the pedestrian re-identification technique according to the present disclosure is different from normal pedestrian tracking in that it identifies a specific target pedestrian from image sequences or video sequences captured by different cameras, enabling long-term tracking and monitoring of the specific target pedestrian. More specifically, the video sequence is a video sequence to be analyzed that requires the presence or absence of a target pedestrian to be identified, and the video sequence and a video frame in which the target pedestrian was previously captured and from which the initial appearance feature or the recent appearance feature was extracted may be captured by different cameras or may be captured by the same camera at different times.

[0018] Specifically, the video sequence analyzed by the pedestrian detection unit 110 may include one or more video frames, each of which may be consecutive or non-consecutive video frames captured by a single camera or multiple cameras with overlapping or non-overlapping fields of view. The pedestrian detection unit 110 may detect pedestrians from each video frame of the video sequence using any suitable image detection technique in the art, and the present disclosure is not limited thereto. For example, the pedestrian detection unit 110 may perform processes such as foreground segmentation, edge extraction, and motion detection for each video frame to identify each sub-image region corresponding to each pedestrian appearing in the video frame, and may represent the sub-image region of the video frame by a rectangular frame circumscribing the body contour of the detected pedestrian. Also, for example, the pedestrian detection unit 110 may use a machine learning technique such as a neural network or a support vector machine to perform pedestrian detection for each video frame using a pre-trained pedestrian detection classifier to identify the position of each pedestrian appearing in the video frame.

[0019] The feature extraction unit 120 may extract appearance features for each pedestrian detected from the video frames. In the present disclosure, the appearance features may include features reflecting the appearance morphology of the pedestrian, such as color features, texture features, shape features, and face features, and are not limited thereto. The above-mentioned methods for extracting various features are well known in the art, and will not be described here.

[0020] The pedestrian identification unit 130 may match the appearance features of each detected pedestrian with the target appearance features of the target pedestrian, and identify the target pedestrian from the video frame based on the matching result. Specifically, the pedestrian identification unit 130 may match the appearance features of each detected pedestrian with the appearance features of the target pedestrian, and determine whether the pedestrian detected from the video frame corresponds to the target pedestrian based on the similarity between the two, thereby re-identifying the pedestrian. Optionally, the number of target pedestrians may be multiple, and correspondingly, the pedestrian identification unit 130 may match the appearance features of each detected pedestrian with the appearance features of each constructed target pedestrian, and determine whether the pedestrian detected from the video frame corresponds to any one of the target pedestrians based on the similarity between the two, thereby re-identifying the pedestrian. It can be understood that the target appearance features of the target pedestrian are included in a pre-constructed pedestrian appearance feature library, and the pedestrian appearance feature library may be stored locally in the pedestrian re-identification device or in a server accessible by the pedestrian re-identification device. Alternatively, the target appearance features of the target pedestrian may not be stored in advance in the pedestrian appearance feature library, but may be generated from a photograph and input from outside as necessary. In the present disclosure, the method of storing or acquiring the target appearance features of the target pedestrian is not limited.

[0021] As described above, when identification is performed based only on the initial appearance feature of the target pedestrian, it is difficult to cope with the situation where the posture or the imaging angle of the pedestrian changes, and when identification is performed based only on the recent appearance feature of the target pedestrian, it is difficult to correct the misidentification. In the present disclosure, by generating a target appearance feature of the target pedestrian based on both the initial appearance feature and the recent appearance feature of the target pedestrian, the generated feature can fully reflect the unique appearance characteristics of the target pedestrian and the changes in its appearance and posture, and the accuracy of the re-identification of the pedestrian can be ensured. The initial appearance feature may include an appearance feature previously extracted from an initial video frame including the target pedestrian, and the recent appearance feature may include an appearance feature previously extracted from a recent video frame including the target pedestrian. In the present disclosure, the initial appearance feature and the recent appearance feature of the target pedestrian may be fused to generate a target appearance feature of the target pedestrian, and the detected pedestrian feature may be matched with the target appearance feature. Hereinafter, the target appearance feature of the target pedestrian based on which the pedestrian identification unit 130 performs feature matching will be described.

[0022] According to an embodiment of the present disclosure, the initial appearance feature and the recent appearance feature of the target pedestrian can be weighted and fused based on a preset weight to obtain a target appearance feature of the target pedestrian. In the embodiment of the present disclosure, the initial appearance feature and the recent appearance feature of the target pedestrian can be weighted and fused based on a preset weight in various ways, and for the sake of completeness of the description, two exemplary methods are described below.

[0023] In one example, first, the number of initial video frames and the number of recent video frames of the target pedestrian to be used in the weighted fusion may be determined based on a preset weight. Specifically, the relative number of recent video frames to the number of initial video frames of the target pedestrian may be determined based on the preset weight, taking into consideration the weights of both the initial appearance feature and the recent appearance feature of the target pedestrian in the weighted fusion. The preset weight may be obtained in various ways, for example, by using an empirical value or machine learning, and is not limited to this in the present disclosure. Then, a weighted average value of each appearance feature of the target pedestrian in the determined number of initial video frames and the determined number of recent video frames may be calculated. Specifically, the appearance feature may be extracted from a sub-image region including the target pedestrian among the determined number of video frames using the above-mentioned feature extraction method such as color feature, texture feature, shape feature, face feature, etc., and the average value of each extracted appearance feature may be calculated as the target appearance feature of the target pedestrian. As an example, when the weighting of the initial appearance feature and the most recent appearance feature is 0.3:0.7, the number of initial video frames and the number of recent video frames required for weighted fusion are determined to be 3 frames and 7 frames, respectively, and further, the average value of the appearance features of this target pedestrian in a total of 10 video frames may be set as the target appearance feature of the target pedestrian.

[0024] In another example, a predetermined number of initial video frames and recent video frames of the target pedestrian may be obtained first, for example, 5 initial video frames and 5 recent video frames of the target pedestrian may be obtained. Then, during weighted fusion, different weights are assigned to the appearance features extracted from the initial video frames and the recent video frames. For example, when the weights of the initial appearance features and the recent appearance features are 0.8:0.2, the sum of 0.8 times the average value of the initial appearance features of the target pedestrian in the 5 initial video frames and 0.2 times the average value of the recent appearance features of the target pedestrian in the 5 recent video frames may be calculated as the target appearance feature of the target pedestrian.

[0025] It should be noted that the examples of weights and video frame numbers in the above embodiments are all merely illustrative. In the present disclosure, the initial appearance features of the target pedestrian in any number of initial video frames and the most recent appearance features in any number of recent video frames may be weighted and fused based on preset weights according to actual requirements for calculation speed and accuracy.

[0026] According to another embodiment of the present disclosure, the initial appearance feature and the most recent appearance feature of the target pedestrian are cascaded, and the initial appearance feature and the most recent appearance feature are cascaded and fused to become the target appearance feature of the target pedestrian by performing a convolution operation on the cascaded appearance feature. The initial appearance feature and the most recent appearance feature of the target pedestrian can be cascaded and fused in various ways, but for completeness, one exemplary method is described below. First, the initial appearance feature and the most recent appearance feature are cascaded in the channel dimension. Then, a convolution operation, such as a 1×1 convolution layer, is applied to the cascaded features to fuse the cascaded features and compress the dimension of the cascaded features to at least the same as the previous dimension of the cascaded features, thereby reducing the amount of calculation by reducing the dimension and fusing the features of each channel.

[0027] An example of the target appearance feature of a target pedestrian based on which the pedestrian re-identification by the pedestrian identification unit 130 according to the embodiment of the present disclosure is performed has been described above. Hereinafter, the pedestrian identification operation by the pedestrian identification unit 130 will be specifically described. As described above, the pedestrian identification unit 130 may match the appearance feature of each detected pedestrian with the target appearance feature of the target pedestrian and identify the target pedestrian from the video frame based on the similarity between the two. Specifically, the pedestrian identification unit 130 may calculate a feature distance between the appearance feature of each detected pedestrian and the target appearance feature of the target pedestrian, such as a Manhattan distance, a Euclidean distance, or a Bhattacharyya distance, and compare the calculated feature distance with a preset threshold to determine whether the detected pedestrian corresponds to the target pedestrian. Optionally, the number of target pedestrians may be multiple, in which case the pedestrian identification unit 130 may calculate the feature distance of each detected pedestrian with each target pedestrian, compare each feature distance with a preset threshold, identify the closest feature distance among the feature distances lower than the preset threshold, and identify the detected pedestrian as the target pedestrian corresponding to the closest feature distance. The pedestrian identification unit 130 may output a pedestrian re-identification result based on the matching result, for example, if the detected pedestrian corresponds to a target pedestrian, associate the detected pedestrian with the identity of the target pedestrian, and if the detected pedestrian cannot be matched with the target pedestrian, assign a new identity to the detected pedestrian.

[0028] Hereinafter, with reference to FIG. 4, an example in which the pedestrian identification unit 130 according to the embodiment of the present disclosure re-identifies a pedestrian based on the target appearance feature of the target pedestrian will be described. As shown in FIG. 4, the first image on the left side is the initial video frame of the target pedestrian ID1, and the second image in the center is the latest video frame of the target pedestrian ID1. The initial appearance feature and the latest appearance feature of the target pedestrian ID1 are extracted from the initial video frame and the latest video frame, respectively, and the initial appearance feature and the latest appearance feature are fused to become the target appearance feature of the target pedestrian ID1. The target appearance feature may not only be used to track the latest posture and appearance change of the target pedestrian, but also to retain the initial appearance characteristics of the pedestrian. After that, as shown in the third image on the right side of FIG. 4, even if the posture and appearance of the pedestrian change significantly, the pedestrian can be accurately identified as ID1 based on the target appearance feature.

[0029] Optionally, the pedestrian re-identification device 100 may further include a feature updating unit (not shown). The feature updating unit may update the target appearance features of the target pedestrian based on the identification result by the pedestrian identification unit 130. For example, when the pedestrian identification unit 130 determines that a pedestrian detected from a video frame corresponds to a target pedestrian and associates the detected pedestrian with the identity of the target pedestrian, the feature updating unit may take the appearance features of the detected pedestrian in the video frame as the most recent appearance features of the target pedestrian, fuse the initial appearance features of the target pedestrian with the most recent appearance features to generate new target appearance features of the target pedestrian, or send the most recent appearance features to a server that maintains the target appearance features of the target pedestrian, so that the server updates the target appearance features of the target pedestrian based on the most recent appearance features received, so that the updated features can follow the recent posture and appearance changes of the target pedestrian in real time. Also, if the pedestrian identification unit 130 determines that a pedestrian detected from a video frame cannot be matched with a target pedestrian and assigns a new identity to the detected pedestrian, the feature update unit may make this pedestrian a new target pedestrian and build its target appearance features for subsequent feature matching.

[0030] As described above, the pedestrian re-identification device 100 according to an embodiment of the present disclosure identifies a target pedestrian using the initial appearance features and recent appearance features of the target pedestrian, thereby effectively tracking the recent pose and appearance changes of the target pedestrian and effectively performing error correction based on the initial features of the target pedestrian, thereby improving the accuracy of pedestrian re-identification in scenes where the pedestrian's pose has changed significantly or where the pedestrian is identified by a different camera.

[0031] The block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. The means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one device that is physically and / or logically coupled, or may be realized by two or more devices that are physically and / or logically separated and directly and / or indirectly (e.g., wired and / or wirelessly) connected to each other and these multiple devices.

[0032] 5 is a hardware block diagram showing an example of a computer device of a pedestrian re-identification device according to an embodiment of the present disclosure. The pedestrian re-identification device 100 may be physically configured as a computer device including a processor 501, a memory 502, a storage 503, a communication device 504, an input device 505, an output device 506, a bus 507, etc.

[0033] In the following description, the term "apparatus" may be replaced with a circuit, a device, a unit, etc. The hardware configuration of the pedestrian re-identification device 100 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.

[0034] For example, although only one processor 501 is shown, there may be multiple processors. Furthermore, the processes may be performed by one processor, or by one or more processors simultaneously, sequentially, or in other manners. Furthermore, the processor 501 may be implemented by one or more chips.

[0035] Each function of the pedestrian re-identification device 100 is realized by loading a specific software (program) onto hardware such as a processor 501 and a memory 502, so that the processor 501 performs calculations and controls communication via a communication device 504 and the reading and / or writing of data in the memory 502 and the storage 503.

[0036] The processor 501 controls the entire computer by running, for example, an operating system. The processor 501 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the pedestrian detection unit 110, the feature extraction unit 120, the pedestrian identification unit, etc. may be realized by the processor 501.

[0037] Moreover, the processor 501 reads out programs (program codes), software modules, and data from the storage 503 and / or the communication device 504 into the memory 502, and executes various processes according to the programs. As the programs, programs that cause a computer to execute at least a part of the operations described in the above embodiments are used.

[0038] The memory 502 is a computer-readable recording medium, and may be configured with at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), and other suitable storage media. The memory 502 may be called a register, a cache, a main memory (primary storage device), or the like. The memory 502 can store an executable program (program code), a software module, or the like for implementing a pedestrian re-identification method according to an embodiment of the present disclosure.

[0039] Storage 503 is a computer-readable recording medium, and may be, for example, at least one of a flexible disk, a floppy disk, a magneto-optical disk (e.g., a CD-ROM (Compact Disc ROM) or the like), a digital versatile disk, a Blu-ray (registered trademark) disk, a removable disk, a hard disk drive, a smart card, a flash memory device (e.g., a card, a stick, a key driver), a magnetic strip, a database, a server, and other suitable storage media. Storage 503 may be referred to as an auxiliary storage device.

[0040] The communication device 504 is hardware (transmitting / receiving device) for communication between computers via a wired and / or wireless network, and may also be referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 504 may include, for example, a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to realize Frequency Division Duplex (FDD) and / or Time Division Duplex (TDD). For example, a video sequence to be analyzed may be received via the communication device 504.

[0041] The input device 505 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 506 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 505 and the output device 506 may be integrated into one device (e.g., a touch panel).

[0042] Furthermore, each device such as the processor 501 and the memory 502 is connected by a bus 507 for communicating information. The bus 507 may be configured as a single bus, or may be configured as different buses between the devices.

[0043] Furthermore, the pedestrian re-identification device 100 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 501 may be implemented by at least one of these pieces of hardware.

[0044] Hereinafter, a flowchart of a pedestrian re-identification method according to an embodiment of the present disclosure will be described with reference to FIG.

[0045] As shown in FIG. 6, in step S601, a pedestrian is detected from each video frame of a video sequence. As described above, the pedestrian re-identification technique according to the present disclosure is different from normal pedestrian tracking in that it identifies a specific target pedestrian from image sequences or video sequences captured by different cameras, enabling long-term tracking and monitoring of the specific target pedestrian. The video sequence is a target video sequence to be analyzed in which the presence or absence of a target pedestrian needs to be identified, and the video sequence and a video frame in which the target pedestrian was previously captured and from which its initial or recent appearance features were extracted may be captured by different cameras or may be captured by the same camera at different times. The video sequence may include one or more video frames, and each video frame may be consecutive or non-consecutive video frames captured by a single camera or multiple cameras with overlapping or non-overlapping fields of view.

[0046] In this step, pedestrians may be detected from each video frame of the video sequence using any suitable image detection technique in the art, and the present disclosure is not limited thereto. For example, for each video frame, processing such as foreground segmentation, edge extraction, motion detection, etc. may be performed to identify each sub-image region corresponding to each pedestrian appearing in the video frame, and for example, the sub-image region of the video frame may be represented by a rectangular frame circumscribing the body contour of the detected pedestrian. In addition, the position of each pedestrian appearing in the video frame may be identified by detecting pedestrians in each video frame using a pre-trained pedestrian detection classifier using machine learning techniques such as neural networks and support vector machines.

[0047] In step S602, appearance features may be extracted for each pedestrian detected from the video frame. The appearance features may include features reflecting the appearance morphology of the pedestrian, such as color features, texture features, shape features, and face features, and are not limited in the present disclosure. This step may employ a feature extraction method well known in the art to extract the appearance features of each pedestrian detected, and the description thereof will be omitted here.

[0048] In step S603, the appearance features of each detected pedestrian are matched with the target appearance features of the target pedestrian, and the target pedestrian is identified from the video frame based on the matching result. In this step, the appearance features of each detected pedestrian are matched with the appearance features of the target pedestrian, and based on the similarity between the two, it is determined whether the pedestrian detected from the video frame corresponds to the target pedestrian, and the pedestrian is re-identified. Alternatively, the number of target pedestrians may be multiple, and correspondingly, in this step, the appearance features of each detected pedestrian are matched with the appearance features of each constructed target pedestrian, and based on the similarity between the two, it is determined whether the pedestrian detected from the video frame corresponds to any one of the target pedestrians, and the pedestrian is re-identified. It can be understood that the target appearance features of the target pedestrian are included in a pedestrian appearance feature library constructed in advance, and the pedestrian appearance feature library can be stored locally in the pedestrian re-identification device or in a server accessible by the pedestrian re-identification device. Alternatively, the target appearance features of the target pedestrian may not be stored in advance in the pedestrian appearance feature library, but may be generated from a photograph and input from outside as necessary.

[0049] In this step, the target appearance feature of the target pedestrian based on which the feature matching is performed is generated based on both the initial appearance feature and the recent appearance feature of the target pedestrian, so that the generated feature can fully reflect the inherent appearance characteristics of the target pedestrian and the changes in its appearance and posture, and ensure the accuracy of pedestrian re-identification. The initial appearance feature may include an appearance feature pre-extracted from an initial video frame including the target pedestrian, and the recent appearance feature may include an appearance feature pre-extracted from a recent video frame including the target pedestrian. The initial appearance feature and the recent appearance feature of the target pedestrian may be fused to generate a target appearance feature of the target pedestrian, and the detected pedestrian feature may be matched with the target appearance feature. The target appearance feature of the target pedestrian based on which the feature matching is performed in this step is similar to the content described for the pedestrian identification unit 130 of the pedestrian re-identification device 100 above, and will be briefly described below.

[0050] According to an embodiment of the present disclosure, the initial appearance feature and the most recent appearance feature of the target pedestrian can be weighted and fused based on a preset weight to obtain a target appearance feature of the target pedestrian. In the embodiment of the present disclosure, the initial appearance feature and the most recent appearance feature of the target pedestrian can be weighted and fused based on a preset weight in various ways. In one example, the number of initial video frames and the number of recent video frames of the target pedestrian used for the weighted fusion can be determined based on a preset weight. Then, a weighted average value of each appearance feature of the target pedestrian in the determined number of initial video frames and the determined number of recent video frames can be calculated. In another example, a predetermined number of initial video frames and recent video frames of the target pedestrian can be obtained first. Then, during the weighted fusion, different weights can be assigned to the appearance features extracted from the initial video frames and the recent video frames, and a target appearance feature of the target pedestrian can be obtained by a weighted average.

[0051] According to another embodiment of the present disclosure, the initial appearance feature and the most recent appearance feature of the target pedestrian are cascaded, and a convolution operation is performed on the cascaded appearance feature, so that the initial appearance feature and the most recent appearance feature are cascaded and fused to obtain a target appearance feature of the target pedestrian.

[0052] Returning to step S603, the specific operation of re-identifying pedestrians based on the above target appearance features will be described in detail below. For example, the feature distance between each detected pedestrian's appearance feature and the target appearance feature of a target pedestrian, such as Manhattan distance, Euclidean distance, or Bhattacharyya distance, may be calculated and compared with a preset threshold to determine whether the detected pedestrian corresponds to the target pedestrian. Alternatively, the number of target pedestrians may be multiple. In this case, the feature distance between each detected pedestrian and each target pedestrian may be calculated, each feature distance may be compared with a preset threshold, the closest feature distance may be specified from among the feature distances lower than the preset threshold, and the detected pedestrian may be identified as the target pedestrian corresponding to the closest feature distance. Next, the re-identification of the pedestrian may be output based on this matching result. For example, if the detected pedestrian corresponds to a certain target pedestrian, the detected pedestrian may be associated with the identity of the target pedestrian, and if the detected pedestrian cannot be matched with the target pedestrian, a new identity may be assigned to the detected pedestrian.

[0053] Optionally, the pedestrian re-identification method further includes updating the target appearance features of the target pedestrian based on the identification result. For example, when determining that the detected pedestrian from the video frame corresponds to a target pedestrian as described above and associating the detected pedestrian with the identity of the target pedestrian, the appearance features of the detected pedestrian in the video frame may be the most recent appearance features of the target pedestrian, and the initial appearance features of the target pedestrian and the most recent appearance features may be fused to generate a new target appearance feature of the target pedestrian, so that the updated features can follow the recent pose and appearance changes of the target pedestrian in real time. Also, when determining that the detected pedestrian from the video frame cannot be matched with the target pedestrian as described above and assigning a new identity to the detected pedestrian, the target appearance features of the new target pedestrian may be constructed for subsequent feature matching.

[0054] As described above, in the pedestrian re-identification method according to the embodiment of the present disclosure, by identifying a target pedestrian using the initial appearance features and recent appearance features of the target pedestrian, it is possible to effectively track recent changes in the target pedestrian's posture and appearance and effectively perform error correction based on the initial features of the target pedestrian, thereby improving the accuracy of pedestrian re-identification in scenes where the pedestrian's posture has changed significantly or where the pedestrian is identified by a different camera.

[0055] In addition to the methods and devices described above, embodiments of the present disclosure may also include a computer-readable storage medium having stored thereon computer program instructions executable by a processor to cause the processor to perform the pedestrian re-identification methods described above.

[0056] Result verification The following describes the results of verifying three types of pedestrian re-identification methods: (1) pedestrian re-identification based on the initial appearance features of the target pedestrian, (2) pedestrian re-identification based on the recent appearance features of the target pedestrian, and (3) pedestrian re-identification based on the target appearance features of the target pedestrian. Specifically, in this exemplary verification, a target pedestrian library including appearance features of multiple target pedestrians is constructed in advance, pedestrians are detected for one or more video frames in a video sequence to be analyzed, and the target pedestrian is identified from the video frames by calculating the feature distance between the appearance features of each pedestrian detected from the video frames and the constructed appearance features of the target pedestrian. In this verification, the accuracy of pedestrian re-identification by the above three types of pedestrian re-identification methods is evaluated using the following two indicators.

[0057] Rank1: This index indicates the probability that the target pedestrian with the best identification result (i.e., the target pedestrian with the closest feature distance) among the target pedestrians is the correct answer after matching the appearance features of the detected pedestrian with those of each target pedestrian, and is expressed by the following formula:

[0058] (Formula 1) TIFF0007653228000001.tif14134

[0059] Accuracy rate: This metric indicates the average probability that the same pedestrian has the same ID in an image sequence, and is expressed by the following formula:

[0060] (Formula 2) TIFF0007653228000002.tif15146

[0061] Based on the above two indicators, the results of verifying three pedestrian re-identification methods are shown in Table 1. (Table 1) TIFF0007653228000003.tif43146

[0062] As shown in Table 1, the pedestrian re-identification method based on target appearance features according to the present disclosure achieves the best results in two indicators.

[0063] 7(a)-7(c) are schematic diagrams showing the results of pedestrian re-identification using the three types of pedestrian re-identification methods described above. Specifically, FIG. 7(a) shows the results of pedestrian re-identification based on the initial appearance features of the target pedestrian, in which the same pedestrian who circles is taken as an example, and the pedestrian is identified as P12 and P7 in different video frames due to changes in the imaging angle and posture. FIG. 7(b) shows the results of pedestrian re-identification based on the recent appearance features of the target pedestrian, in which the same pedestrian who circles is taken as an example, and the pedestrian is misidentified from P17 to P15, and thereafter, the misidentification result of P15 is retained and cannot be corrected. FIG. 7(c) shows the results of pedestrian re-identification based on the target appearance features of the target pedestrian, and the target appearance features of the target pedestrian are generated based on its initial appearance features and recent appearance features, so that correct identification results are obtained regardless of the different imaging angles and pedestrian postures.

[0064] It can be seen that the pedestrian re-identification method based on target appearance features of the present invention improves the accuracy of pedestrian re-identification in scenes where the pedestrian's pose has changed significantly or where the pedestrian was identified by a different camera, compared to when pedestrian re-identification is performed based only on the initial or recent appearance features of the target pedestrian.

[0065] Although the basic principle of the present disclosure has been described in relation to specific examples, the advantages, advantages, effects, etc. mentioned in the present disclosure are merely illustrative rather than limiting, and should not be considered as being required for each embodiment of the present disclosure. In addition, the specific details disclosed above are merely illustrative and easy to understand, and are not limiting, and the above details are not necessarily intended to limit the present disclosure to be realized using the above specific details.

[0066] As used herein, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0067] As used herein, any reference to an element using a designation such as "first," "second," etc., does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

[0068] As used herein, the term "determining" may encompass a wide variety of actions. "Determining" may be considered to be, for example, calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), ascertaining, etc. "Determining" may also be considered to be, for example, receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), etc. "Determining" may also be considered to be, for example, resolving, selecting, choosing, establishing, comparing, etc. In other words, a "judgment (decision)" can be considered as making a "judgment (decision)" on some kind of action.

[0069] The terms "connected," "coupled," or any variation thereof, refer to any connection or coupling, direct or indirect, between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to one another. The coupling or connection between elements may be physical, logical, or a combination of both. For example, "connected" may be read as "access." As used herein, two elements may be considered to be "connected" or "coupled" to one another by using one or more wires, cables, and / or printed electrical connections, as well as by using electromagnetic energy, such as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and / or light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0070] To the extent that the terms "including," "comprising," and variations thereof are used herein, these terms are intended to be inclusive, similar to the term "comprising." Further, the term "or" as used herein is not intended to be an exclusive or.

[0071] Each aspect / embodiment described herein may be used alone, in combination, or switched during execution. In addition, the processing procedures, sequences, flow charts, etc. of each aspect / embodiment described herein may be reordered unless inconsistent. For example, the methods described herein present various step elements in an exemplary order and are not limited to the particular order presented.

[0072] It should be noted that, in the devices and methods of the present disclosure, software should be construed broadly to mean commands, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0073] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, and Digital Subscriber Line (DSL)) and / or wireless technologies (such as infrared, microwave, etc.), these wired and / or wireless technologies are included within the definition of transmission media.

[0074] Although the present invention has been described in detail above, it is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present invention.

Claims

1. a pedestrian detection unit for detecting a pedestrian in each video frame of a video sequence; a feature extraction unit for extracting appearance features for each pedestrian detected from a video frame; a pedestrian identification unit that calculates a feature distance between the extracted appearance feature of each pedestrian and a target appearance feature of a target pedestrian, compares the feature distance with a preset threshold, and determines, among pedestrians whose feature distance is lower than the preset threshold, a pedestrian corresponding to the closest feature distance as the target pedestrian; The target appearance feature of the target pedestrian is obtained by fusing the initial appearance feature and the recent appearance feature of the target pedestrian; The initial appearance feature includes an appearance feature extracted in advance from an initial video frame, which is at least one video frame when the target pedestrian first appears; The pedestrian re-identification device, wherein the recent appearance features include appearance features previously extracted from a recent video frame, which is at least one video frame of a most recent appearance of the target pedestrian.

2. The pedestrian re-identification device of claim 1 , further comprising: performing a weighted fusion of the initial appearance feature and the recent appearance feature of the target pedestrian based on a preset weight.

3. 3. The pedestrian re-identification device of claim 2, wherein weighted fusion of the initial appearance features and the recent appearance features of the target pedestrian based on preset weights includes determining a number of initial video frames and a number of recent video frames of the target pedestrian for weighted fusion based on preset weights such that a ratio of weights of the initial appearance features and weights of the recent appearance features is the same as a ratio of the number of the initial video frames and the number of the recent video frames, and calculating an average of the appearance features of the target pedestrian in the determined number of initial video frames and the determined number of recent video frames.

4. The pedestrian re-identification device of claim 1 , further comprising: cascading the initial appearance feature and the most recent appearance feature of the target pedestrian and performing a convolution operation on the cascaded appearance feature to cascade and fuse the initial appearance feature and the most recent appearance feature.

5. The pedestrian re-identification device of claim 1 , further comprising a feature updating unit for setting the appearance features of the detected pedestrian in the video frame as the most recent appearance features of the target pedestrian when the pedestrian detected from the video frame corresponds to the target pedestrian.

6. The pedestrian re-identification device of claim 1 , wherein the video sequence and the initial and recent video frames from which the target appearance features of the target pedestrian are extracted are taken by different cameras or taken by the same camera at different times.

7. a processor and a memory having computer program instructions stored thereon; The computer program instructions, when executed by the processor, cause the processor to: detect pedestrians from each video frame of a video sequence; extracting appearance features for each pedestrian detected from the video frames; calculating a feature distance between the extracted appearance feature of each pedestrian and a target appearance feature of a target pedestrian, comparing the feature distance with a preset threshold, and determining, as the target pedestrian, a pedestrian corresponding to the closest feature distance among pedestrians whose feature distance is lower than the preset threshold; The target appearance feature of the target pedestrian is obtained by fusing the initial appearance feature and the recent appearance feature of the target pedestrian; The initial appearance feature includes an appearance feature extracted in advance from an initial video frame, which is at least one video frame in which the target pedestrian first appears; The recent appearance features include appearance features previously extracted from a recent video frame, the recent video frame being at least one video frame in which the target pedestrian most recently appeared.

Images