A dense crowd correspondence relationship interpretation system and method based on point cloud matching

Abstract

The application discloses a kind of dense crowd corresponding relation interpretation system and method based on point cloud matching, the method includes: dense crowd corresponding relation interpretation method based on point cloud matching, with the upper left corner of each scene image as origin, respectively with right and downward as positive direction respectively establish x axis and y axis, so that each node in pedestrian point cloud has only pixel coordinate expression;Pedestrian point cloud matching network extracts pedestrian point cloud feature, finds pedestrian point cloud corresponding relation, and the input is the feature tensor of two pedestrian point clouds;Interpretation result correction module modifies result after obtaining pedestrian node corresponding matrix.This method has practical value in analyzing the trend of multiple behavior individuals in dense crowd, preventing major public safety accidents, and is closely related to the construction of a safe city, the realization of intelligent monitoring and processing of dense crowd and public danger.

Description

TECHNICAL FIELD

[0001] The present application relates to a dense crowd interpretation method, in particular to a dense crowd correspondence relationship interpretation system and method based on point cloud matching. BACKGROUND

[0002] At present, pedestrian re-identification has become one of the popular applications in the field of computer vision, and its goal is to match pedestrians with the same identity in different sensor devices. Since different modal sensors include a large amount of information of monitored targets, it helps to improve the supervision of dense crowds and the risk handling capability, so it is more and more common to use multiple sensors of different modalities to monitor dense crowds, such as using infrared and visible light cameras to monitor the same place. In this case, how to quickly determine the monitoring situation of the same target under different modalities requires cross-modal dense crowd correspondence relationship interpretation, that is, quickly determining the correspondence relationship between dense crowds in multiple different modal sensor data. However, this correspondence relationship interpretation has two difficulties, one is the feature loss of the target caused by the mutual occlusion of the dense crowd, and the other is the feature difference of the target caused by the modal difference.

[0003] The current point cloud matching usually includes two steps, initial matching and accurate matching. The initial matching method mainly includes center coincidence method, label method, feature extraction method, etc.; the accurate matching mainly includes ICP algorithm, PLICP algorithm, NICP algorithm, etc.

[0004] There is an end-to-end learnable DeepICP network based on ICP, which is divided into two structure modules of feature extraction and corresponding point generation, which combines the traditional point cloud matching idea and introduces a deep learning network as a tool, which has a milestone significance. However, in the point cloud registration algorithm based on deep learning, the most famous is still the PointNet point cloud matching network, which uses one-dimensional convolution conv1d for operation. In addition, on the basis of the PointNet point cloud matching network, there is a PointNet++ network.

[0005] The purpose of the cross-modal scene correspondence relationship interpretation of dense crowds is to find the one-to-one correspondence relationship of individuals in the dense crowds under two modalities. The existing dense crowd detection technology can only locate the position information of the pedestrian target in different modalities, but cannot obtain the correspondence relationship of pedestrians under different modalities. Image registration cannot locate all pedestrians, which greatly hinders the subsequent correspondence relationship interpretation.

[0006] The problem focused on by the correspondence determination method of dense crowds in a cross-modal scene is how to quickly determine the monitoring situation of the same target in different modalities, that is, how to quickly determine the correspondence between the dense crowds in multiple different modal sensor data. In the process of solving this problem, the difficulties and challenges encountered include the following aspects:

[0007] 1) The universality problem of cross-modal determination. The research method of the present application should minimize the influence of the differences in visual features of targets in different modalities. Although the current cross-modal monitoring system is mostly a multi-modal monitoring system composed of an infrared camera (or thermal imager) and a visible light monitoring camera (including black and white and color cameras), with the development and progress of science and technology, more and more visual modalities will emerge. At the same time, as human needs continue to increase, the application scenarios of the present application will become increasingly rich and complex. Therefore, in order to ensure universality, the method of the present application should minimize the dependence on modalities and minimize the influence of different modalities in the research process.

[0008] 2) The determination problem of the same pedestrian existing binocular disparity and in different positions. It goes without saying that when different modal cameras at the same height shoot the same scene, there will be a binocular perspective difference, which may cause many influences including the following situations:

[0009] (1) The same pedestrian is in different positions in the images presented in different modalities;

[0010] (2) Due to the difference caused by different perspectives, some pedestrians existing in the image presented in one modality are missing in the image of another modality; and the image of another modality usually has some pedestrians that are missing in the image of the other modality. This is particularly evident for pedestrians in or near the edge zone of the scene. SUMMARY

[0011] The main purpose of the present application is to provide a dense crowd correspondence determination system and method based on point cloud matching, to solve the problem of determining the correspondence of dense crowds in a cross-modal scene, that is, to explore how to quickly and accurately determine the correspondence of multiple pedestrian targets in a dense scene.

[0012] According to one aspect of the present application, a dense crowd correspondence determination system based on point cloud matching is provided, comprising:

[0013] A pedestrian point cloud matching network for extracting pedestrian point cloud features and finding pedestrian point cloud correspondence, the input of which is the feature tensor of two pedestrian point clouds;

[0014] A determination result correction module for correcting the result after obtaining the pedestrian node correspondence matrix.

[0015] According to another aspect of the present application, a dense crowd correspondence relationship interpretation method based on point cloud matching is provided, comprising:

[0016] The dense crowd correspondence relationship interpretation method based on point cloud matching establishes x-axis and y-axis respectively with right and down as positive direction, respectively, with the upper left corner of each scene image as the origin, so that each node in the pedestrian point cloud has a unique pixel coordinate representation;

[0017] The pedestrian point cloud matching network extracts pedestrian point cloud features and finds pedestrian point cloud correspondence relationship, and the input is the feature tensor of two pedestrian point clouds;

[0018] The interpretation result correction module corrects the result after obtaining the pedestrian node correspondence matrix.

[0019] Further, the pedestrian point cloud matching network extracting pedestrian point cloud features and finding pedestrian point cloud correspondence relationship comprises:

[0020] The feature extraction module of the point cloud is realized by a convolutional neural network CNN;

[0021] The pedestrian point cloud matching network is based on a Siamese network structure;

[0022] The extracted feature set of the pedestrian point cloud X is represented by , the deep learning feature extraction model initialized by the parameters , and the output feature of the point cloud can be represented as formula (1):

[0023] (1)

[0024] The similarity values of the same node and other nodes are compared, and the node with the largest similarity value is taken as the final correspondence result, and the specific calculation method is shown in formula (2):

[0025] (2)

[0026] In the formula:

[0027] is a 01 matrix, representing the final correspondence result of the two point clouds is a full 1 matrix;

[0028] indicates that the data in the parentheses has been processed by row normalization and column normalization to obtain a double random matrix; , respectively represent the tensor set composed of the node features of the point cloud and the point cloud ; ​M represents that the M matrix is binarized, and the maximum value of each row is set to 1, and the remaining values are set to 0.

[0029] Further, the result correction module corrects the result after obtaining the pedestrian node corresponding matrix, and the correction of the result includes:

[0030] The position angle is used for correction, and the specific calculation method of the position angle between two pedestrian nodes is shown in formula (3)

[0031] (3)

[0032] In the formula:

[0033] Y2 represents the pixel vertical coordinate in the second pedestrian node scene;

[0034] Y1 represents the pixel vertical coordinate in the first pedestrian node scene;

[0035] X2 represents the pixel horizontal coordinate in the second pedestrian node scene;

[0036] X1 represents the pixel horizontal coordinate in the first pedestrian node scene.

[0037] Advantages of the present application:

[0038] The present application fills the blank of the dense crowd cross-modal interpretation method, and for the cross-modal related method, the related method at home and abroad is more than the cross-modal re-identification, cross-modal image fusion, cross-modal image matching and other directions, and the related research on the corresponding relationship of the cross-scene dense crowd is relatively rare,

[0039] Play a major role in the construction and improvement of the social security prevention and control system, and is conducive to maintaining the stability of the society and promoting the harmonious development of the society;

[0040] Compared with the commonly used interpretation method, the present application has obvious effect. In order to test the effect of the present method, a dense target point cloud data set CrowdSet is constructed. The experiment shows that on the CrowdSet data set, the average interpretation coverage of the present application can reach 0.91, and the average interpretation accuracy can reach 0.78. While the commonly used ORB combined with RANSAC algorithm has an interpretation accuracy of 0.82 and an interpretation coverage of 0.21, which is lower than the present application.

[0041] In addition to the purposes, features and advantages described above, the present application has other purposes, features and advantages. The present application will be further described below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application, illustrate the preferred embodiment of the application and assist in

[0043] Figure 1 is a pedestrian point cloud example diagram of the present application;

[0044] Figure 2 is a dense crowd corresponding relationship interpretation method structure diagram based on point cloud matching of the present application;

[0045] Figure 3 is a point cloud matching network structure diagram of the present application;

[0046] Figure 4 is a result correction method flow chart of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0048] The dense crowd corresponding relationship interpretation method based on point cloud matching establishes x-axis and y-axis respectively with right and down as positive direction respectively with the upper left corner of each scene image as the origin, so that each node in the pedestrian point cloud has a unique pixel coordinate representation. Figure 1 represents a pedestrian point cloud example.

[0049] The method content mainly includes the following two parts.

[0050] 1) Pedestrian point cloud matching network

[0051] The network is a lightweight network model designed for extracting pedestrian point cloud features and finding pedestrian point cloud corresponding relationship. The input of the network is the feature tensor of two pedestrian point clouds. Among them, the feature extraction module of the point cloud is designed and realized by convolutional neural network CNN. The network adopts the structure of Siamese neural network, which is to ensure that the pedestrian point cloud data in two scenes can pass through the same structure of neural network for corresponding feature extraction, so that the pedestrian nodes in the two point clouds are more related and easy to match subsequently.

[0052] The pedestrian point cloud matching network is based on the Siamese network structure, and the specific structure is as shown in Figure 3 The input of the network is the feature of two point clouds, which is processed by the feature extraction module. In the point cloud feature and point cloud After feature extraction, the input is input into the ComputeMatchMatrix module, and the output is the pedestrian node correspondence matrix. This matrix is the preliminary interpretation result.

[0053] In point cloud matching, convolutional neural network (CNN) has always been a very popular tool, which has generality for target feature extraction, so it can be used for pedestrian point cloud feature extraction. For a certain pedestrian point cloud , if represents the point enhanced features, represents the corresponding set, then can represent the feature combination of each node in the point cloud, which is also the feature of the point cloud in this method. Use to represent the feature set of the extracted pedestrian point cloud X, to represent the deep learning feature extraction model initialized by the parameter , then the output feature of the point cloud can be represented as formula (1) as follows:

[0054] (1)

[0055] The ComputeMatchMatrix algorithm module is used to find the correspondence of each pedestrian node according to the obtained point cloud features. Therefore, this module needs to calculate the similarity between the features of each two nodes. Here, the module calculates the Euclidean distance between the nodes and performs normalization operation, so as to obtain the probability of the correspondence of the pedestrian node to other pedestrian nodes. This probability is the value describing the similarity between two pedestrian nodes. The greater the value, the more similar the two nodes. Then, the module compares a group of similarity values of the same node to other nodes, and takes the node with the largest similarity value as the final corresponding result. The specific calculation method is shown in formula (2).

[0056] (2)

[0057] In the formula:

[0058] is a 01 matrix, representing the final correspondence result of two point clouds is a full 1 matrix;

[0059] indicates that the data in the parentheses has been processed by row normalization and column normalization to obtain a double random matrix. , respectively represent the tensor set composed of the node features of point cloud and point cloud .

[0060] M represents that the M matrix is binarized, the maximum value of each row is set to 1, and the remaining values are set to 0.

[0061] 2) Interpretation result correction module

[0062] After obtaining the pedestrian node corresponding matrix, the result needs to be corrected. Here, the position angle is used for correction. The specific calculation method of the position angle between two pedestrian nodes is shown in formula (3)

[0063] (3)

[0064] In the formula:

[0065] Y2 - the pixel vertical coordinate in the second pedestrian node scene;

[0066] Y1 - the pixel vertical coordinate in the first pedestrian node scene;

[0067] X2 - the pixel horizontal coordinate in the second pedestrian node scene;

[0068] X1 - the pixel horizontal coordinate in the first pedestrian node scene.

[0069] The specific method of position angle correction is shown in the flow Figure 4 chart.

[0070] The present application fills the blank of the dense crowd cross-modal interpretation method to a certain extent. For the related method of cross-modal, the related method at home and abroad is more than the cross-modal re-identification, cross-modal image fusion, cross-modal image matching and other directions, and the related research on the corresponding relationship of the cross-scene dense crowd is relatively rare.

[0071] It plays an important role in the construction and improvement of the social security and prevention system, and is conducive to maintaining the stability of the society and promoting the harmonious development of the society. Most of the factors leading to the danger of dense crowds come from the inside of the dense crowd, and in most cases, a few individuals panic and spread this psychology, leading to large-scale loss of control and causing a large number of casualties. The method has practical value in analyzing the trend of multiple behavioral individuals in the dense crowd and preventing major public safety accidents, and is closely related to the construction of a safe city and the realization of intelligent monitoring and processing of dense crowds and public dangers.

[0072] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A dense crowd correspondence interpretation method based on point cloud matching, characterized in that, It comprises: A dense crowd correspondence interpretation method based on point cloud matching, taking the upper left corner of each scene image as the origin, respectively establishing the x-axis and y-axis with right and down as the positive direction, so that each node in the pedestrian point cloud has a unique pixel coordinate representation; The pedestrian point cloud matching network extracts pedestrian point cloud features and finds pedestrian point cloud correspondence, and the input is the feature tensor of two pedestrian point clouds; The interpretation result correction module corrects the result after obtaining the pedestrian node correspondence matrix; The pedestrian point cloud matching network extracts pedestrian point cloud features and finds pedestrian point cloud correspondence, which includes: The feature extraction module of the point cloud is realized by a convolutional neural network CNN; The pedestrian point cloud matching network is based on the Siamese network structure; with X represents the feature set of the extracted pedestrian point cloud, X represents the feature set of the extracted pedestrian point cloud, the deep learning feature extraction model initialized with the output feature of the point cloud can be represented as formula (1): （1） Compare a group of similarity values of the same node and other nodes, and take the node with the largest similarity value as the final corresponding result, and the specific calculation method is shown in formula (2): （2） In the formula: is a 01 matrix, representing the final correspondence result of two point clouds represents a full 1 matrix; represents that the data in the parentheses has been processed by row normalization and column normalization, and a double random matrix is obtained; , respectively represent the tensor sets composed of the node features of the point cloud and the point cloud ; represents that the M matrix is processed by binaryzation, the maximum value of each row is set to 1, and the remaining values are set to 0; The interpretation result correction module corrects the result after obtaining the pedestrian node correspondence matrix, which includes: Use the position angle to correct, and the specific calculation method of the position angle between two pedestrian nodes is shown in formula (3) （3） In the formula: - the pixel ordinate in the second pedestrian junction scenario; - the vertical coordinate of the pixel in the first pedestrian junction scenario; - pixel abscissa in the second pedestrian junction scenario; - pixel abscissa in the first pedestrian junction scenario.

2. A system for applying the method for dense crowd correspondence interpretation based on point cloud matching according to claim 1, characterized in that, It comprises: The pedestrian point cloud matching network is used to extract pedestrian point cloud features and find pedestrian point cloud correspondence, and the input is the feature tensor of two pedestrian point clouds; The interpretation result correction module is used to correct the result after obtaining the pedestrian node correspondence matrix.

Images