Pedestrian clustering method, device and storage medium

Abstract

The present application discloses a pedestrian clustering method, device and storage medium, wherein the pedestrian clustering method comprises: obtaining multiple groups of image sequences, each group of image sequences comprises multiple continuous images, each group of image sequences comprises a first human body image and a second human body image of a pedestrian in a predetermined area, and the first human body image and the second human body image are human body images from different perspectives; detecting human body images in each group of image sequences, obtaining matching first human body images and second human body images to form a group of matching human body image pairs; obtaining associated face images corresponding to each group of human body image pairs to form a ternary image group including human body image pairs and associated face images; clustering using multiple groups of ternary image groups to obtain pedestrian clustering results. The present application obtains the first human body image and the second human body image of pedestrians in a predetermined area, and the time domain restriction greatly ensures the matching accuracy, thereby improving the matching efficiency and matching accuracy of the first human body image and the second human body image.

Description

Technical Field

[0001] The present application belongs to the field of image recognition technology, and specifically relates to a pedestrian clustering method, device and storage medium. Background Art

[0002] Pedestrian clustering is a sub-problem in the field of image clustering. It mainly involves automatically filing together human images of the same person in different cameras or videos. It has very important application value in urban security, public security criminal investigation, smart buildings, and smart retail. In actual monitoring scenarios, due to the complex posture changes, rich perspective changes, and occlusion of the human body, the apparent characteristics of pedestrians with the same identity may vary greatly, which brings huge challenges to human clustering. Summary of the invention

[0003] The present application provides a pedestrian clustering method, device and storage medium to solve the technical problem of difficult human clustering.

[0004] To solve the above technical problems, a technical solution adopted in the present application is: a pedestrian clustering method, comprising: obtaining multiple groups of image sequences, each group of the image sequences includes multiple continuous images, each group of the image sequences respectively includes a first human body image and a second human body image of a pedestrian in a predetermined area, and the first human body image and the second human body image are human body images from different perspectives; detecting the human body images in each group of the image sequences, and obtaining the matching first human body image and second human body image to form a group of matching human body image pairs; obtaining the associated facial images corresponding to each group of the human body image pairs to form a ternary image group including the human body image pairs and the associated facial images; and performing clustering using the multiple groups of the ternary image groups to obtain pedestrian clustering results.

[0005] To solve the above technical problems, another technical solution adopted in the present application is: an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the above pedestrian clustering method.

[0006] In order to solve the above technical problems, another technical solution adopted by the present application is: a computer-readable storage medium having program data stored thereon, and the program data implements the above pedestrian clustering method when executed by a processor.

[0007] The pedestrian clustering method of the present application has at least the following advantages: in a predetermined area of ​​the image sequence obtained by the present application, the time of the first human image and the second human image is close, and they are human images of different perspectives, which improves the matching efficiency and matching accuracy of the first human image and the second human image. In addition, the present application only obtains another matching human image for each human image to form a group of human image pairs, while the existing scheme usually uses all human bodies that meet the threshold conditions to match each other, which has large redundancy and misjudgment. For large-scale pedestrian clustering at the city level, it will cause large computing resource pressure and chain clustering error amplification effect, which is fatal to pedestrian clustering. The matching scheme in the present application can effectively reduce information redundancy and weaken the error amplification effect of chain clustering compared with the existing scheme. In addition, by using face and body clustering separately, the present application can effectively avoid the problem of mismatching semantic information of face and body compared to directly clustering face and body information together, and by clustering face and body separately, they can also correct each other, which will greatly improve the accuracy and recall rate of pedestrian clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0009] Figure 1 It is a flowchart of an embodiment of a pedestrian clustering method of the present application;

[0010] Figure 2 It is a schematic diagram of the framework of an embodiment of a pedestrian clustering device of the present application;

[0011] Figure 3 It is a schematic diagram of a framework of an electronic device according to an embodiment of the present application;

[0012] Figure 4 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] See also Figure 1 , Figure 1It is a flowchart of an embodiment of a pedestrian clustering method of the present application.

[0015] An embodiment of the present application provides a pedestrian clustering method, comprising the following steps:

[0016] S11: Acquire multiple groups of image sequences, each group of image sequences includes multiple continuous images, and each group of image sequences includes a first human body image and a second human body image of a pedestrian in a predetermined area.

[0017] A plurality of image sequences are obtained, each of which includes a plurality of continuous images, and the image sequences are obtained by a camera device. Each of the image sequences includes a first human body image and a second human body image of a pedestrian in a predetermined area, and the first human body image and the second human body image are human body images from different perspectives.

[0018] Wherein, each group of image sequences is acquired by a two-way camera group. Each two-way camera group includes a first camera and a second camera, and the predetermined area is a collection of areas that can be photographed by the first camera and the second camera, and the angle between the shooting axes of the first camera and the second camera meets the preset angle requirement. It should be noted that the first camera and the second camera are not suitable for shooting side by side in the same direction, that is, the angle between the shooting axes of the first camera and the second camera cannot be 0°, so as to obtain the first human body image and the second human body image from different perspectives. Preferably, the preset angle requirement is that the angle between the shooting axes of the first camera and the second camera is any angle between 30° and 180°, such as 30°, 45°, 90°, 180°, etc. When a pedestrian passes through the predetermined area, the two cameras obtain the first human body image and the second human body image of the pedestrian respectively.

[0019] Specifically, the angle between the shooting axes of the first camera and the second camera is 180°, that is, the first camera and the second camera are arranged facing each other or facing each other. The first human body image is a front human body image, that is, the human body image taken by the first camera when the pedestrian faces the first camera; the second human body image is a back human body image, that is, the human body image taken by the second camera when the pedestrian faces away from the first camera.

[0020] By setting a plurality of predetermined areas in a larger area, each predetermined area can obtain a set of image sequences through a bidirectional camera group, which are used to obtain a first human body image and a second human body image of a pedestrian, thereby obtaining a plurality of image sequences.

[0021] By introducing a two-way camera group as a basic unit, the first human image and the second human image of the same pedestrian ID can be obtained at the same time, which increases the information capacity of the basic unit and helps solve the problem of large changes in human posture in monitoring scenarios, thereby improving the effect of human clustering. In addition, since the image sequence obtained by a two-way camera group is in a predetermined area, the time of the first human image and the second human image taken by the two cameras is close, which improves the matching efficiency and matching accuracy of the first human image and the second human image.

[0022] S12: Detecting human body images in each group of image sequences, and acquiring matching first human body images and second human body images to form a group of matching human body image pairs.

[0023] Using a human body detection algorithm to detect human body images in each set of image sequences, obtaining matching first human body images and second human body images to form a set of matching human body image pairs, specifically including:

[0024] Detect the human body in each set of image sequences. Select the first human body image P obtained by the first camera 1 , and take the first human image P 1 As a reference, obtain the first human image P obtained by the second camera 1 A set of human images within a preset time period before and after B. The preset time period is adjusted according to the setting of the specific two-way camera group. The preset time period is the time when the target human body enters the shooting area of ​​the second camera, which may be before or after the target human body enters the shooting area of ​​the first camera.

[0025] Calculate the first human image P 1 The similarity with all images in the human image set B can be calculated by using the pedestrian re-identification model to extract the normalized features of all images in the human image set B and calculate the similarity of the first human image P 1 The cosine similarity with all images in the human image set B.

[0026] From the images whose similarity is greater than the preset threshold, select the image with the highest similarity as the second human image P 2 Specifically, in the selection process, all images with similarity greater than a preset threshold can be first taken to form a human body set B S , finally take B S The first human image P 1 The second human image P with the highest similarity 2 As the first human image P 1 Of course, you can also directly select the image with the highest similarity, and then compare the highest similarity with a preset threshold, and use the image with the highest similarity greater than the preset threshold as the second human image P 2Thus, the optimal second human image P that meets the preset threshold can be selected. 2 , avoiding the optimal second human image P 2 If the preset threshold is not met and it is directly used in the next step, the matching accuracy of the overall method will be affected. 1 If there is no image with a similarity greater than a preset threshold, then the second human image P cannot be obtained in the group of image sequences. 2 , that is, it is impossible to form a set of matching human image pairs (P 1 , P 2 ).

[0027] Using the first human body image P 1 and the second human image P 2 Form a set of matching human image pairs (P 1 , P 2 ).

[0028] It should be noted that, after the target human body enters the predetermined area, a tracking is obtained, that is, a plurality of consecutive target human body images are obtained, and the best one can be selected as the first human body image P 1 Specifically, it includes: tracking the target human body photographed by the first camera, obtaining a target human body image sequence, the target human body image sequence includes a plurality of consecutive target human body images; inputting the target human body image into the human body quality model, obtaining the quality score of each target human body image, obtaining the target human body image with the highest score as the first human body image P 1 The human body mass model is used to obtain the optimal human body image in the target human body sequence as the first human body image P 1 , which can ensure that the first human image P 1 and the second human image P 2 The quality of the human body image is then optimized (P 1 , P 2 ).

[0029] Moreover, the first camera and the second camera are a two-way camera group, which capture the same target human body within a certain time range, so that the human body image captured by one of the cameras is used as a benchmark, and the most similar human body image above a preset threshold in the other camera within a certain time range is taken out for matching, and the restrictions of the time domain and similarity greatly ensure the matching accuracy. At the same time, in each two-way camera group, the present application only obtains the best matching human body image with the highest similarity of each human body image under the other camera, forming an optimal group of optimal human body image pairs. The existing scheme usually uses all human bodies that meet the threshold conditions to match each other, which has large redundancy and misjudgment. For large-scale pedestrian clustering at the city level, it will cause high pressure on computing resources and the error amplification effect of chain clustering, which is fatal to pedestrian clustering. The optimal matching scheme in the present application can effectively reduce information redundancy and weaken the error amplification effect of chain clustering compared with the existing scheme.

[0030] S13: Acquire the associated face images corresponding to each group of human image pairs to form a ternary image group including the human image pairs and the associated face images.

[0031] For the human body image pair (P 1 , P 2 ), obtaining an associated face image corresponding to the first human image in each group of human image pairs, specifically comprising:

[0032] Tracking human image pairs (P 1 , P 2 ) to obtain a facial image sequence. The facial image sequence is input into the face quality matching model to obtain the quality score of each facial image, and the facial image with the highest score is obtained as the associated facial image F, forming a ternary image group (P) including a human image pair and an associated facial image. 1 , P 2 , F). Thus, a bidirectional camera group obtains a pedestrian ID consisting of a triple image group (P 1 , P 2 , F).

[0033] S14: Perform clustering using multiple sets of triplet image groups to obtain pedestrian clustering results.

[0034] Clustering multiple triplet image groups obtained from multiple image sequences to obtain the final pedestrian clustering results includes:

[0035] Using the face images and body images in the multiple sets of ternary image groups, face clustering and body clustering are performed in their respective semantic spaces. In this embodiment, the clustering uses the DBSCAN algorithm. When the faces and bodies are clustered separately, the face clustering value and the body clustering value can be obtained. By clustering the faces and bodies separately, compared with directly clustering the face and body information together, the problem of mismatching semantic information of faces and bodies can be effectively avoided, and the subsequent clustering results will also be promoted.

[0036] Next, the face and body clustering results are merged. When merging, the face relaxation threshold and the body relaxation threshold need to be pre-set for subsequent judgment. There are several judgment methods when merging:

[0037] First, in response to the face clustering values ​​of the two ternary image groups being greater than the face clustering threshold, and the body clustering values ​​being greater than the body clustering threshold, the two ternary image groups are directly clustered together as one ID.

[0038] The second method is to respond to the face clustering value of the two ternary image groups being greater than the face clustering threshold, while the body clustering value being less than the body clustering threshold; determine the first similarity of the two first body images in the two ternary image groups; and in response to the first similarity being greater than the body relaxation threshold, cluster the two ternary image groups together as one ID.

[0039] The third type, in response to the human body clustering value of the two sets of ternary image groups being greater than the human body clustering threshold, and the face clustering value being less than the face clustering threshold; determining the second similarity of the two best face images in the two sets of ternary image groups; in response to the second similarity being greater than the face relaxation threshold, clustering the two sets of ternary image groups together as one ID.

[0040] By clustering faces and bodies separately and then correcting each other, the accuracy and recall rate of pedestrian clustering will be greatly improved.

[0041] The pedestrian clustering method of this application has at least the following advantages:

[0042] First, by introducing a two-way camera group as a basic unit, the first human image and the second human image of the same pedestrian ID can be obtained at the same time, which increases the information capacity of the basic unit and helps solve the problem of large changes in human posture in monitoring scenarios, thereby improving the effect of human clustering. In addition, since the image sequence obtained by a two-way camera group is in a predetermined area, the time of the first human image and the second human image taken by the two cameras is close, which improves the matching efficiency and matching accuracy of the first human image and the second human image.

[0043] 2. In each two-way camera group, the present application only obtains the matching human image of each human image under the other camera to form an optimal set of human image pairs. The existing scheme usually matches all human bodies that meet the threshold conditions, which has large redundancy and misjudgment. For large-scale pedestrian clustering at the city level, it will cause high pressure on computing resources and the error amplification effect of chain clustering, which is fatal to pedestrian clustering. The optimal matching scheme in the present application can effectively reduce information redundancy and weaken the error amplification effect of chain clustering compared with the existing scheme.

[0044] Third, by clustering faces and bodies separately, compared with directly clustering face and body information together, the problem of semantic information mismatch between faces and bodies can be effectively avoided. In addition, by clustering faces and bodies separately and then correcting each other, the accuracy and recall rate of pedestrian clustering will be greatly improved.

[0045] See also Figure 2 , Figure 2 It is a schematic diagram of the framework of an embodiment of a pedestrian clustering device of the present application.

[0046] Another embodiment of the present application provides a pedestrian clustering device 20, including an acquisition module 21 and a processing module 22. The acquisition module 21 acquires multiple groups of image sequences, each group of image sequences includes multiple continuous images, each group of image sequences includes a first human body image and a second human body image of a pedestrian in a predetermined area, and the first human body image and the second human body image are human body images from different perspectives. The processing module 22 detects the human body images in each group of image sequences, acquires the matching first human body image and the second human body image, to form a group of matching human body image pairs; the processing module 22 acquires the associated face image corresponding to each group of human body image pairs, to form a ternary image group including human body image pairs and associated face images; the processing module 22 uses multiple groups of ternary image groups for clustering to obtain pedestrian clustering results.

[0047] The pedestrian clustering device 20 of the present application can implement the pedestrian clustering method in any of the above embodiments. The pedestrian clustering device 20 has at least the following advantages:

[0048] First, by introducing a two-way camera group as a basic unit, the first human image and the second human image of the same pedestrian ID can be obtained at the same time, which increases the information capacity of the basic unit and helps solve the problem of large changes in human posture in monitoring scenarios, thereby improving the effect of human clustering. In addition, since the image sequence obtained by a two-way camera group is in a predetermined area, the time of the first human image and the second human image taken by the two cameras is close, which improves the matching efficiency and matching accuracy of the first human image and the second human image.

[0049] 2. In each two-way camera group, the present application only obtains the best matching human image of each human image under the other camera to form an optimal set of human image pairs. The existing scheme usually matches all human bodies that meet the threshold conditions, which has large redundancy and misjudgment. For large-scale pedestrian clustering at the city level, it will cause great pressure on computing resources and the error amplification effect of chain clustering, which is fatal to pedestrian clustering. The optimal matching scheme in the present application can effectively reduce information redundancy and weaken the error amplification effect of chain clustering compared with the existing scheme.

[0050] Third, by clustering faces and bodies separately, compared with directly clustering face and body information together, the problem of semantic information mismatch between faces and bodies can be effectively avoided. In addition, by clustering faces and bodies separately and then correcting each other, the accuracy and recall rate of pedestrian clustering will be greatly improved.

[0051] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of an electronic device of the present application.

[0052] Another embodiment of the present application provides an electronic device 30, including a memory 31 and a processor 32 coupled to each other, the processor 32 is used to execute program instructions stored in the memory 31 to implement the pedestrian clustering method of any of the above embodiments. In a specific implementation scenario, the electronic device 30 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 30 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0053] Specifically, the processor 32 is used to control itself and the memory 31 to implement the steps in the pedestrian clustering method of any of the above embodiments. The processor 32 can also be called a CPU (Central Processing Unit). The processor 32 may be an integrated circuit chip with signal processing capabilities. The processor 32 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 32 can be implemented by an integrated circuit chip.

[0054] See also Figure 4 , Figure 4It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

[0055] Yet another embodiment of the present application provides a computer-readable storage medium 40 on which program data 41 is stored. When the program data 41 is executed by a processor, the pedestrian clustering method of any of the above embodiments is implemented.

[0056] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0057] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0058] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0059] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 40. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a readable storage medium 40, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of each implementation method of the present application. The aforementioned readable storage medium 40 includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0060] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A pedestrian clustering method, characterized in that: include: Acquire multiple groups of image sequences, each group of the image sequences includes multiple continuous images, each group of the image sequences is acquired by a two-way camera group, each of the two-way camera groups includes a first camera and a second camera, each group of the image sequences includes a first human body image and a second human body image of a pedestrian in a predetermined area, the first human body image and the second human body image are human body images from different perspectives; Detecting human body images in each group of the image sequence, and acquiring matching first human body images and second human body images to form a group of matching human body image pairs; Acquire the associated facial images corresponding to each group of the human image pairs to form a ternary image group including the human image pairs and the associated facial images; Using a plurality of the ternary image groups to perform clustering to obtain pedestrian clustering results; The detecting of the human body images in each group of the image sequence and obtaining the matching first human body image and the second human body image to form a group of matching human body image pairs includes: Detecting a human body appearing in each group of the image sequences; Selecting a first human body image obtained by the first camera, and acquiring a set of human body images within a preset time period before and after the first human body image obtained by the second camera; Calculating the similarity between the first human body image and all images in the human body image set; From the images whose similarity is greater than a preset threshold, select the image with the highest similarity to determine as the second human body image; A set of matched human body image pairs is formed using the first human body image and the second human body image.

2. The method according to claim 1, characterized in that The included angle between the shooting axes of the first camera and the second camera meets a preset angle requirement.

3. The method according to claim 1, characterized in that The selecting a first human body image obtained by a first camera comprises: Tracking the target human body photographed by the first camera to obtain a target human body image sequence; A human body image with the highest quality score in the target human body image sequence is acquired by using a human body quality model as the first human body image.

4. The method according to claim 1, characterized in that The calculating the similarity between the first human body image and all images in the human body image set includes: A person re-identification model is used to extract normalized features of all images in the human image set, and cosine similarities between the first human image and all images in the human image set are calculated.

5. The method according to claim 1, characterized in that The obtaining of the associated face image corresponding to each group of the human body image pairs includes: Tracking the facial images corresponding to the human body image pair to obtain a facial image sequence; The face quality matching model is used to obtain the face image with the highest quality score in the face image sequence, and determine it as the associated face image.

6. The method according to claim 1, characterized in that The method of clustering the plurality of ternary image groups to obtain pedestrian clustering results includes: Using the plurality of ternary image groups, respectively, to perform face clustering and body clustering to obtain face clustering values ​​and body clustering values; In response to the face clustering values ​​of the two groups of the ternary image groups being greater than the face clustering threshold, and the body clustering values ​​being greater than the body clustering threshold, the two groups of the ternary image groups are directly clustered together.

7. The method according to claim 6, characterized in that The method of clustering multiple groups of the ternary image groups to obtain pedestrian clustering results further includes: In response to the face clustering values ​​of the two sets of the ternary image groups being greater than the face clustering threshold, and the body clustering values ​​being less than the body clustering threshold; A first similarity between two of the first human images in the two triple image groups is determined, and in response to the first similarity being greater than a human relaxation threshold, the two triple image groups are clustered together.

8. The method according to claim 6, characterized in that The method of clustering multiple groups of the ternary image groups to obtain pedestrian clustering results further includes: In response to the human body clustering values ​​of the two sets of the ternary image groups being greater than the human body clustering threshold, and the face clustering values ​​being less than the face clustering threshold; Determining a second similarity between two optimal face images in the two sets of ternary image groups; In response to the second similarity being greater than a face relaxation threshold, the two groups of triple image groups are clustered together.

9. An electronic device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images