Pedestrian and vehicle interaction state recognition method and system

Abstract

The present application relates to a kind of pedestrian and vehicle interaction state recognition method and system, belong to interactive state identification technical field.The laser radar point cloud data of each training sample is processed first, the distance feature of pedestrian and vehicle is obtained, image data is processed, the head orientation feature of pedestrian and the motion state feature of pedestrian are obtained.Then each kind of feature of all training samples is formed into a feature sample set, and the feature sample set is processed using spectral clustering algorithm, to obtain label set.Finally, feature sample set and label set are used as input, three classifiers are obtained by using classifier algorithm training, to identify the interaction state of pedestrian and vehicle using three classifiers, to assist the driving behavior of unmanned vehicle or driver.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of interaction state recognition, in particular to a pedestrian and vehicle interaction state recognition method and system. BACKGROUND

[0002] With the development of automobile intelligence, people's requirements for automobile safety are getting higher and higher. In the actual road traffic environment, pedestrians are one of the most complex traffic elements, and the possibility of accidents with vehicles is relatively large. Therefore, accurately recognizing the behavior state of pedestrian and vehicle interaction plays an extremely important role in assisting vehicles or drivers to better respond.

[0003] At present, most researches focus on the recognition method of pedestrian behavior state, which usually needs to artificially define and label the interactive behavior, and artificially delimit the parameter range of judging the interactive behavior, which has certain subjectivity and limitation, is difficult to adapt to the pedestrian samples with different behavior preferences in the real situation, reduces the model generalization, and directly judges through the original image data. The limitation of this recognition method is that the data volume is large, the operation is complex, and there is much interference information, so that the method cannot accurately recognize the behavior state of the pedestrian, and has the problem of not strong practicality. At the same time, the pedestrian behavior state defined by the above method does not consider the interactive behavior of pedestrians and vehicles, and cannot directly assist the driving behavior of vehicles or drivers.

[0004] Therefore, there is an urgent need for a technology capable of recognizing the interaction state of pedestrians and vehicles. SUMMARY

[0005] The purpose of the present application is to provide a pedestrian and vehicle interaction state recognition method and system, which can recognize the interaction state of pedestrians and vehicles and assist the driving behavior of unmanned vehicles or drivers.

[0006] To achieve the above purpose, the present application provides the following scheme:

[0007] A pedestrian and vehicle interaction state recognition method, comprising:

[0008] Obtaining a plurality of training samples; each of the training samples comprises laser radar point cloud data and image data collected during vehicle driving;

[0009] For each of the training samples, the laser radar point cloud data is processed to obtain the distance feature of the pedestrian and the vehicle; the image data is processed to obtain the head orientation feature of the pedestrian and the motion state feature of the pedestrian;

[0010] The distance features of pedestrians and vehicles of all the training samples are combined to form a first feature sample set; the first feature sample set is processed by using a spectral clustering algorithm to obtain a first label set;

[0011] The head orientation features of pedestrians of all the training samples are combined to form a second feature sample set; the second feature sample set is processed by using the spectral clustering algorithm to obtain a second label set;

[0012] The motion state features of pedestrians of all the training samples are combined to form a third feature sample set; the third feature sample set is processed by using the spectral clustering algorithm to obtain a third label set;

[0013] The first feature sample set and the first label set are taken as inputs, and a first classifier is trained by using a classifier algorithm; the second feature sample set and the second label set are taken as inputs, and a second classifier is trained by using the classifier algorithm; the third feature sample set and the third label set are taken as inputs, and a third classifier is trained by using the classifier algorithm;

[0014] The first classifier, the second classifier and the third classifier are used to identify the interaction state of pedestrians and vehicles.

[0015] A pedestrian and vehicle interaction state identification system comprises:

[0016] A sample acquisition module is configured to acquire a plurality of training samples; each of the training samples comprises laser radar point cloud data and image data collected during vehicle driving;

[0017] A feature extraction module is configured to, for each of the training samples, process the laser radar point cloud data to obtain distance features of pedestrians and vehicles, and process the image data to obtain head orientation features of pedestrians and motion state features of pedestrians;

[0018] A label generation module is configured to combine the distance features of pedestrians and vehicles of all the training samples to form a first feature sample set; the first feature sample set is processed by using a spectral clustering algorithm to obtain a first label set; the head orientation features of pedestrians of all the training samples are combined to form a second feature sample set; the second feature sample set is processed by using the spectral clustering algorithm to obtain a second label set; the motion state features of pedestrians of all the training samples are combined to form a third feature sample set; the third feature sample set is processed by using the spectral clustering algorithm to obtain a third label set;

[0019] The training module is used to train a first classifier using the first feature sample set and the first label set as input, and to train a second classifier using the classifier algorithm using the second feature sample set and the second label set as input; and to train a third classifier using the classifier algorithm using the third feature sample set and the third label set as input.

[0020] The recognition module is used to identify the interaction state between pedestrians and vehicles using the first classifier, the second classifier and the third classifier.

[0021] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0022] This invention provides a method and system for recognizing pedestrian-vehicle interaction states. First, the lidar point cloud data of each training sample is processed to obtain the distance features between pedestrians and vehicles. Image data is then processed to obtain pedestrian head orientation features and pedestrian motion state features. Next, each feature from all training samples is grouped into a feature sample set, and this set is processed using a spectral clustering algorithm to obtain a label set. Finally, using the feature sample set and label set as input, three classifiers are trained using a classifier algorithm to recognize the interaction states between pedestrians and vehicles, assisting in the driving behavior of autonomous vehicles or drivers. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the identification method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a schematic diagram of the identification method provided in Embodiment 1 of the present invention;

[0026] Figure 3 This is a schematic diagram of the pedestrian skeleton node location numbering provided in Embodiment 1 of the present invention;

[0027] Figure 4 This is a system block diagram of the identification system provided in Embodiment 2 of the present invention;

[0028] Figure 5 This is a schematic diagram of the identification system provided in Embodiment 2 of the present invention. Detailed Implementation

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0030] The present application aims to provide a pedestrian and vehicle interaction state recognition method and system, which can recognize the interaction state of pedestrians and vehicles, and assist the driving behavior of unmanned vehicles or drivers.

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0032] Embodiment 1:

[0033] The present embodiment is used to provide a pedestrian and vehicle interaction state recognition method, as shown in Figure 1 and Figure 2 , comprising:

[0034] S1: obtaining a plurality of training samples; each of the training samples comprises laser radar point cloud data and image data collected during vehicle driving;

[0035] Specifically, the present embodiment can use the vehicle-mounted data acquisition platform installed on the vehicle to collect pedestrian data, that is, use a plurality of sensors to collect perception data (including laser radar point cloud data and image data) during vehicle driving, to obtain a plurality of training samples. The vehicle-mounted data acquisition platform includes a laser radar and a camera, the laser radar point cloud data is collected by the laser radar installed on the vehicle, and the image data is collected by the camera installed on the vehicle. The laser radar and the camera will collect the data of the left, middle and right in front of the driver's perspective, so that the obtained data can as much as possible comprehensively include all pedestrians around the vehicle.

[0036] S2: for each of the training samples, processing the laser radar point cloud data to obtain the distance feature of the pedestrian and the vehicle; processing the image data to obtain the head orientation feature of the pedestrian and the motion state feature of the pedestrian;

[0037] The present embodiment extracts pedestrian feature information through S2, processes the training samples to obtain a plurality of feature information of pedestrians, and the feature information includes the distance feature of the pedestrian and the vehicle D , the head orientation feature of the pedestrian H and the motion state feature of the pedestrian M .

[0038] It should be noted that the laser radar point cloud data and image data described in this embodiment are collected in a very short time interval, so the laser radar point cloud data is a continuous sequence including multiple frames of point cloud data, and the image data is a continuous sequence including multiple frames of images. For example, when the time interval is 0.3s, the total number of frames of point cloud data and image data can be about 10 frames.

[0039] This embodiment can extract distance features of three types of pedestrians and vehicles from the laser radar point cloud data, denoted as D ={ D 1, D 2, D 3}, the distance of pedestrians and vehicles is extracted from each frame of point cloud data of the laser radar point cloud data d , and the distance features of three types of pedestrians and vehicles are extracted according to the distances of all point cloud data. Specifically, the distance features of pedestrians and vehicles can include:

[0040] (1) extracting the distance corresponding to each frame of point cloud data included in the laser radar point cloud data, the distance being the distance of pedestrians and vehicles;

[0041] (2) calculating the average value of the distances corresponding to all point cloud data to obtain the relative distance of pedestrians and vehicles;

[0042] The relative distance of pedestrians and vehicles D 1 is obtained by calculating the average value of the distances of pedestrians and vehicles in consecutive k frames, k is the total number of frames of point cloud data included in the laser radar point cloud data, D The calculation formula of

[0043] ;

[0044] wherein, d i is the distance corresponding to the i frame of point cloud data.

[0045] (3) for each frame of point cloud data, calculate the difference between the distance corresponding to the point cloud data and the distance corresponding to the previous frame of point cloud data of the point cloud data to obtain the distance difference; calculate the increase of the distance of pedestrians and vehicles and the decrease of the distance of pedestrians and vehicles according to the distance differences corresponding to all point cloud data; the relative distance of pedestrians and vehicles, the increase of the distance of pedestrians and vehicles, and the decrease of the distance of pedestrians and vehicles form the distance features of pedestrians and vehicles.

[0046] The increase of the distance of pedestrians and vehicles in consecutive k frames D 2 is calculated by the formula:

[0047] ;

[0048] in, d i+1 For the first i +1 frame of point cloud data corresponds to the distance.

[0049] continuous k The reduction in the distance between pedestrians and vehicles within a frame D The formula for calculating 3 is:

[0050] ;

[0051] In S2, image data processing to obtain pedestrian head orientation features and pedestrian motion state features may include:

[0052] (1) The pedestrian skeleton recognition method is used to identify each frame of the image data to obtain the pedestrian skeleton node coordinate data of each frame. The pedestrian skeleton node coordinate data includes the nose tip node coordinate data, the upper spinal insertion point coordinate data, the lower spinal insertion point coordinate data, the left hip joint node coordinate data, the right hip joint node coordinate data, the left knee joint node coordinate data, the right knee joint node coordinate data, the left ankle joint node coordinate data, and the right ankle joint node coordinate data.

[0053] The pedestrian skeleton recognition method in this embodiment can employ existing algorithms such as OpenPose. Figure 3 As shown, the pedestrian skeleton nodes involved in this embodiment include the nose tip node (node ​​0), the upper spinal stop point (node ​​1), the lower spinal stop point (node ​​8), the left hip joint node (node ​​9), the right hip joint node (node ​​12), the left knee joint node (node ​​10), the right knee joint node (node ​​13), the left ankle joint node (node ​​11), and the right ankle joint node (node ​​14). The coordinate data of the above nodes in each frame of the image can be obtained by the pedestrian skeleton recognition method.

[0054] (2) For each frame of image, calculate the first angle between the line connecting the tip of the nose and the upper end of the spine and the horizontal line based on the coordinate data of the tip of the nose node and the upper end of the spine; calculate the height of the spine based on the coordinate data of the upper end of the spine and the lower end of the spine; calculate the distance between the knees based on the coordinate data of the left knee joint node and the right knee joint node; calculate the distance between the ankles based on the coordinate data of the left ankle joint node and the right ankle joint node; calculate the bending angle of the left leg based on the coordinate data of the left hip joint node, the left knee joint node, and the left ankle joint node; calculate the bending angle of the right leg based on the coordinate data of the right hip joint node, the right knee joint node, and the right ankle joint node.

[0055] The first angle formed by the line connecting the tip of the pedestrian's nose and the upper end of the spine with the horizontal line. The calculation formula is:

[0056] ;

[0057] in, y 0 is the node 0 y coordinate; y 1 is node 1 y coordinate; x 0 is the node 0 x coordinate; x 1 is node 1 x coordinate.

[0058] The angle of the pedestrian's left leg bend The calculation formula is:

[0059] ;

[0060] in, x 9 is node 9 x coordinate; x 10 For node 10 x coordinate; x 11 For node 11 x coordinate; y 9 is node 9 y coordinate; y 10 For node 10 y coordinate; y 11 For node 11 y coordinate; d （9,10） This is the distance between node 9 and node 10; d （10,11） This represents the distance between node 10 and node 11.

[0061] The angle of the pedestrian's right leg bend The calculation formula is:

[0062] ;

[0063] in, x 12 For node 12 x coordinate; x 13 For node 13 x coordinate; x 14 For node 14 x coordinate;y 12 the coordinates of node 12; y 13 the coordinates of node 13; y 14 the coordinates of node 14. y （12,13） the distance between node 12 and node 13; y （13,14） the distance between node 13 and node 14. y d d (3) Calculate the average of the first angle of all images to obtain the pedestrian head orientation feature;

[0064] Extract a class of pedestrian head orientation features from image data, denoted as

[0065] = H}. By solving the mean of the first angle between the line connecting the nose tip and the upper end of the spine of the pedestrian in the continuous H frame and the horizontal line, the head facing direction of the pedestrian in the continuous k frame is obtained k , and the calculation formula is: H

[0066] ;

[0067] wherein is the first angle of the i-th frame image. i

[0068] (4) According to the spine height, the distance between the two knees, and the distance between the two ankles of all images, calculate the double-leg motion amplitude change feature; according to the left leg bending angle and the right leg bending angle of all images, calculate the double-leg bending degree feature; the double-leg motion amplitude change feature and the double-leg bending degree feature constitute the pedestrian motion state feature.

[0069] Extract six classes of pedestrian motion state features from image data, denoted as M = M 1, M 2, M 3, M 4, M 5, M 6}.

[0070] Among the six classes of pedestrian motion state features, M 1, M 2 describes the double-leg motion amplitude change feature of the pedestrian, which can better distinguish between stationary and moving pedestrians. M 1 is obtained by the variance of the distance between the two knees M ​​​​2. This is obtained through the variance of the distance between the two ankles, calculated using the following formula:

[0071] ;

[0072] ;

[0073] in, dis 1 is the first ratio of the distance between the left and right knee nodes (i.e., the distance between the two knees) to the height of the spine; dis 2 is the second ratio of the distance between the left and right ankle nodes (i.e., the bi-ankle distance) divided by the height of the spine. dis 1,i For the first i The first ratio of the frame image; The average of the first ratios of all images; dis 2,i For the first i The second ratio of the frame image; This is the average of the second ratio of all images.

[0074] The formulas for calculating the first ratio and the second ratio are:

[0075] ;

[0076] In the formula for calculating the first ratio, the numerator is the distance between the two knees, and the denominator is the height of the spine. x 10 For node 10 x coordinate; x 13 For node 13 x coordinate; y 10 For node 10 y coordinate; y 13 For node 13 y coordinate; x 1 is node 1 x coordinate; x 8 is node 8 x coordinate; y 1 is node 1 y coordinate; y 8 is node 8 y Coordinates; the numerator of the formula for calculating the second ratio is the bi-ankle distance, and the denominator is the spine height; x 11 For node 11 x coordinate; x 14 For node 14 x coordinate; y 11 For node 11y coordinates; y 14 for the node 14 y coordinates.

[0077] M 3, M 4, M 5, M 6 is a double-leg bending degree feature, which describes the bending degree of the double legs. Since the human body will bend more when moving, the double-leg bending degree feature can better evaluate the intensity of the pedestrian movement, indicating the state of the pedestrian running or walking. M 3, M 4 describes the bending degree of the pedestrian double legs, which is obtained by calculating the average value of the double-leg bending angle of the pedestrian in consecutive k frames, M 5, M 6 describes the proportion of the frame number when the pedestrian double legs are bent too much or too little in consecutive k frames. Specifically, the calculation of the double-leg bending degree feature according to the left leg bending angle and the right leg bending angle of all images can include:

[0078] (4.1) calculating the average value of the left leg bending angle of all images to obtain the left leg bending degree;

[0079] The calculation formula of the left leg bending degree M 3 is:

[0080] ;

[0081] wherein, is the left leg bending angle of the i-th image. i

[0082] (4.2) calculating the average value of the right leg bending angle of all images to obtain the right leg bending degree;

[0083] The calculation formula of the right leg bending degree M 4 is:

[0084] ;

[0085] wherein, is the right leg bending angle of the i-th image. i

[0086] (4.3) counting the first number of the left leg bending angle greater than the first preset angle in all images, counting the second number of the right leg bending angle greater than the first preset angle in all images, and calculating the ratio of the sum of the first number and the second number to the total frame number to obtain the first proportion;

[0087] ​​When the first preset angle is 175 degrees, the first proportion M 5 is calculated by the following formula:

[0088] ;

[0089] Wherein, n represents the number; k represents the total number of frames of images.

[0090] (4.4) A third number of all images in which the left leg bending angle is less than a second preset angle is counted, a fourth number of all images in which the right leg bending angle is less than the second preset angle is counted, and a ratio of a sum of the third number and the fourth number to the total number of frames of images is calculated to obtain a second proportion;

[0091] When the second preset angle is 160 degrees, the second proportion M 6 is calculated by the following formula:

[0092] ;

[0093] The left leg bending degree, the right leg bending degree, the first proportion and the second proportion constitute the double leg bending degree feature.

[0094] S3: The distance features of pedestrians and vehicles of all the training samples are combined to form a first feature sample set; the first feature sample set is processed by using a spectral clustering algorithm to obtain a first label set; the head direction features of pedestrians of all the training samples are combined to form a second feature sample set; the second feature sample set is processed by using the spectral clustering algorithm to obtain a second label set; the motion state features of pedestrians of all the training samples are combined to form a third feature sample set; the third feature sample set is processed by using the spectral clustering algorithm to obtain a third label set;

[0095] Through pedestrian feature extraction, three kinds of feature sample sets can be obtained in the embodiment, including a first feature sample set D ={ D 1 , D 2 , ...... , D N} a second feature sample set H ={ H 1 , H 2 , ...... , H N} and a third feature sample set M ={ M 1 , M 2, ...... , M N}, N is the total number of samples. For the i-th data in the sample set of each feature, there are i D i = { , , } ; H i = { H i} ; M i = { , , , , , }.

[0096] After obtaining the three feature sample sets, the classification of the pedestrian-vehicle interaction behavior state can be performed. For the obtained pedestrian feature sample set, the different pedestrian-vehicle interaction states are identified through a spectral clustering algorithm.

[0097] The spectral clustering algorithm mainly includes two steps.

[0098] First, spectral decomposition of the feature sample set: through dimension transformation and inner product calculation on the original data in the feature sample set, the original data in the feature sample set is decomposed into a group of linearly independent eigenvectors, and a group of eigenvectors with a specific number are selected to form a feature spectrum. The feature spectrum extracts the key information of the original data, reduces the influence of noise in the data on the clustering effect, and also reduces the operation amount of the clustering operation. Among them, the number of selected eigenvectors is determined according to the size of the eigenvalue corresponding to the eigenvector. When the size of the current K non-zero eigenvalue is significantly lower than that of the K+1 non-zero eigenvalue, let K be the number of selected eigenvectors. The number of eigenvectors K reflects that the original data can be mainly divided into K categories, and thus can be clustered into K categories.

[0099] Second, project the original data into the feature spectrum composed of the first K non-zero eigenvectors, and perform K-means clustering on the projected data with the specified number of clustering categories being K. The K-means clustering calculates the similarity between data, so that samples with similar structural characteristics are clustered into the same category, and samples with different structural characteristics are as far away as possible, and K categories are clustered. By analyzing the feature data distribution of each category obtained, the actual pedestrian-vehicle interaction state meaning represented by different categories is defined.

[0100] ​Specifically, processing the first feature sample set using the spectral clustering algorithm to obtain the first label set may include: performing spectral decomposition on the first feature sample set to obtain a feature spectrum; projecting the data within the first feature sample set onto the feature spectrum to obtain projected data; performing K-means clustering on the projected data to obtain K groups; and determining the label for each group to obtain the first label set. The process of processing the second and third feature sample sets using the spectral clustering algorithm is the same as described above and will not be repeated here.

[0101] More specifically, for the first feature sample set D ={ D 1 , D 2 , ...... , D N To perform spectral clustering, the original data in the first feature sample set is first analyzed to obtain the dimension. K D The feature spectrum, and make the number of clusters equal to K D After projecting the sample data onto the feature spectrum, the projected data Ф is obtained. D ). For Ф ( D The number of clusters is K D K-means clustering can divide the first feature sample set into... K D Groups with different distance distributions and distance variation trends S D ={ , ,..., }, that is, for the first feature sample set D ={ D 1 , D 2 , ...... , D N}, which has a corresponding first tag set L D ={ , ,..., }, where for the first i There are 10 sample points, Different numbers represent different category labels; that is, the first label set includes the label of each sample point in the first feature sample set. In this embodiment, the category labels can be customized as needed. Specifically, based on the feature data distribution corresponding to different categories, the pedestrian and vehicle interaction states of each clustered category can be defined. Among the category features, relative distance... DThe smaller the mean of 1, the closer the pedestrian is to the vehicle; the increase in distance D The smaller the mean of 2, the greater the reduction in distance. D The larger the mean of 3, the more likely the pedestrians are to approach the vehicle. Groups exhibiting this distribution characteristic are closer to the vehicle and show a tendency to approach; therefore, the vehicle should exercise higher safety precautions against them. Thus, groups with smaller distances and a tendency to approach can be defined as the dangerous interaction distance group. Similarly, groups where these three characteristics increase in opposite directions represent a relatively lower level of interaction danger and are defined as the safe interaction distance group.

[0102] For the second feature sample set H ={ H 1 , H 2 , ...... , H N After performing spectral clustering, groups with different head orientation features can be obtained. S H ={ , ,..., }, and has a corresponding second tag set L H ={ , ,..., }, where for the first i There are 10 sample points, , K H The total number of clusters corresponding to the second feature sample set. For vehicles, pedestrians with their heads turned to the sides show less attention to vehicles and are less likely to actively avoid them. Within these clusters... H The smaller the mean, the larger the angle between the pedestrian's head and the vehicle's direction of travel, indicating a lower level of attention the pedestrian pays to the vehicle. Therefore, the vehicle should be more vigilant. Hence, the definition... H Groups with larger mean values ​​are considered dangerous interaction groups. Similarly, H A lower mean value indicates a lower level of interaction risk, and these are defined as a safe interaction group.

[0103] For the third feature sample set M ={ M 1 , M 2 , ...... , M N After performing spectral clustering, groups with different ranges of motion and degrees of flexion of the legs can be obtained. S M ={ , ,..., }, and has a corresponding third tag set L M ={ , ,..., }, where for the first i There are 10 sample points, , K M The total number of clusters corresponding to the third feature sample set. M 1. M 2. Groups with larger mean values ​​indicate that the distance between their legs varies more and the range of motion of their legs is greater. M 3. M 4. Groups with smaller means indicate that their knee joint angles are smaller and their legs are more bent. M The mean of 5 is relatively small. M Groups with larger mean values ​​indicate a higher frequency of large knee flexion and a lower frequency of small knee flexion, and a greater degree of leg flexion. This distribution suggests that pedestrians exhibiting this characteristic, with larger leg movement amplitudes and greater leg flexion, are moving more vigorously, at higher speeds, and with greater uncertainty, posing a greater danger to vehicles. Therefore, groups with larger leg movement amplitudes and greater leg flexion can be defined as dangerous movement groups, while groups with smaller leg movement amplitudes and a tendency to straighten their legs can be defined as safe movement groups. Similarly, groups with these six characteristics increasing in opposite directions represent a relatively lower level of interaction danger and are defined as safe interaction distance groups.

[0104] S4: Using the first feature sample set and the first label set as input, a first classifier is trained using a classifier algorithm to obtain a first classifier; using the second feature sample set and the second label set as input, a second classifier is trained using the classifier algorithm to obtain a second classifier; using the third feature sample set and the third label set as input, a third classifier is trained using the classifier algorithm to obtain a third classifier.

[0105] Specifically, based on the first feature sample set D ={ D 1 , D 2 , ...... , D N}, second feature sample set H ={ H 1 , H 2 , ...... , H N}, third feature sample set M ={M 1 , M 2 , ...... , M N} and the label set L corresponding to the different feature sample sets D , H , M Through a classifier algorithm, a pedestrian-vehicle interaction behavior classifier is trained. With the obtained feature sample set D , H , M as input, and the corresponding label set L D , H , M as output, the classifier algorithm trains different pedestrian-vehicle interaction behavior state classifiers, respectively denoted as a first classifier , a second classifier , and a third classifier . The classifier algorithm includes but is not limited to a support vector machine algorithm, a multi-layer neural network, etc. By using the above process, the pedestrian-vehicle interaction behavior state classifier training can be completed.

[0106] S5: identifying the interaction state of the pedestrian and the vehicle by using the first classifier, the second classifier, and the third classifier.

[0107] By inputting the pedestrian features into the classifier, the classifier can output the interaction state between the pedestrian and the vehicle. The input pedestrian features include D , H , M The interaction state between the pedestrian and the vehicle includes the degree of approaching or moving away from the vehicle, the degree of sensing the vehicle, and the motion amplitude size.

[0108] In order to adapt to the variable pedestrian-vehicle interaction state and behavior mode in the real situation and simplify the identification process, the embodiment proposes a pedestrian-vehicle interaction state identification method. The distance of the pedestrian relative to the vehicle and the pedestrian skeleton information are extracted from the laser radar point cloud data and image data, and multiple features representing the pedestrian-vehicle interaction state are extracted. The features are clustered and learned, and the classifier is trained. In a data-driven manner, multiple pedestrian-vehicle interaction state classifiers are trained, which can identify the pedestrian-vehicle interaction state.

[0109] Compared with the prior art, the advantages of the identification method of the embodiment include:

[0110] (1) short training time and small data transmission amount

[0111] Unlike traditional methods that directly use pedestrian image data as the data source for pedestrian behavior recognition, this recognition method extracts distance information and pedestrian skeleton nodes from LiDAR point cloud data and image data to further extract relevant features of pedestrian-vehicle interaction states. These features are then used as input to a clustering algorithm, simplifying the pedestrian-vehicle interaction state recognition process and reducing training time. Furthermore, the amount of extracted feature data is significantly less than the amount of original LiDAR point cloud data and image data, reducing the data transmission pressure in the system.

[0112] (2) It is highly intuitive and easy to adjust the system according to specific situations.

[0113] This recognition method uses key nodes of the pedestrian skeleton as the basis for behavior recognition. It is highly intuitive, can easily assign practical meaning to relevant feature parameters, has strong interpretability, reduces the difficulty of feature selection, and facilitates the adjustment of relevant features or parameters for specific situations.

[0114] (3) There is no need to manually define the mathematical model of the interaction state between pedestrians and vehicles, thus avoiding the need to manually define the parameter range for interaction state recognition.

[0115] This identification method uses a spectral clustering algorithm to classify pedestrians with different feature parameters into multiple categories. Different categories of pedestrians have different human-vehicle interaction states. Based on the clustering results, a pedestrian-vehicle interaction state classifier can be trained. The classifier can identify different human-vehicle interaction states based on feature parameters, avoiding the subjectivity and weak generalization of traditional pedestrian behavior identification methods that rely on subjectively dividing parameter ranges to determine pedestrian behavior states.

[0116] Example 2:

[0117] This embodiment provides a pedestrian and vehicle interaction state recognition system, such as... Figure 4 and Figure 5 As shown, it includes:

[0118] The sample acquisition module M1 is used to acquire multiple training samples; each training sample includes lidar point cloud data and image data collected during vehicle operation.

[0119] The feature extraction module M2 is used to process the lidar point cloud data for each training sample to obtain the distance features between pedestrians and vehicles; and to process the image data to obtain the pedestrian head orientation features and pedestrian motion state features.

[0120] The label generation module M3 is configured to group the pedestrian-vehicle distance features of all the training samples into a first feature sample set, process the first feature sample set by using a spectral clustering algorithm to obtain a first label set, group the pedestrian head orientation features of all the training samples into a second feature sample set, process the second feature sample set by using the spectral clustering algorithm to obtain a second label set, group the pedestrian motion state features of all the training samples into a third feature sample set, and process the third feature sample set by using the spectral clustering algorithm to obtain a third label set.

[0121] The training module M4 is configured to take the first feature sample set and the first label set as input, train a first classifier by using a classifier algorithm, take the second feature sample set and the second label set as input, train a second classifier by using the classifier algorithm, and take the third feature sample set and the third label set as input, train a third classifier by using the classifier algorithm.

[0122] The recognition module M5 is configured to recognize the interaction state between the pedestrian and the vehicle by using the first classifier, the second classifier, and the third classifier.

[0123] In the present specification, each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0124] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above embodiment description is only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In conclusion, the content of the present specification should not be understood as the limitation of the present application.

Claims

1. A method for recognizing the interaction state between pedestrians and vehicles, characterized in that, include: Obtain multiple training samples; Each training sample includes lidar point cloud data and image data collected during vehicle operation; For each training sample, the LiDAR point cloud data is processed to obtain distance features between pedestrians and vehicles. Specifically, this includes: extracting the distance corresponding to each frame of point cloud data included in the LiDAR point cloud data, where the distance is the distance between pedestrians and vehicles; calculating the average of the distances corresponding to all the point cloud data to obtain the relative distance between pedestrians and vehicles; for each frame of point cloud data, calculating the difference between the distance corresponding to the point cloud data and the distance corresponding to the previous frame of point cloud data to obtain the distance difference; calculating the increase and decrease in the distance between pedestrians and vehicles based on the distance differences corresponding to all the point cloud data; combining the relative distance between pedestrians and vehicles, the increase and decrease in the distance between pedestrians and vehicles to form the distance features between pedestrians and vehicles; and processing the image data to obtain pedestrian head orientation features and pedestrian motion state features. The formula for calculating the increase in distance D2 between pedestrians and vehicles is: Where k is the total number of frames of point cloud data included in the lidar point cloud data; d i+1 d represents the distance corresponding to the point cloud data in the (i+1)th frame; i The distance corresponding to the point cloud data in the i-th frame; The formula for calculating the reduction in distance between pedestrians and vehicles, D3, is as follows: The pedestrian head orientation feature includes the head facing direction, and the formula for calculating the head facing direction H is as follows: Where, α 1,i The first included angle of the i-th frame of the image data; The formula for calculating the first included angle α1 is: Where y0 is the y-coordinate of the nasal tip node coordinate data; y1 is the y-coordinate of the upper spinal insertion point coordinate data; x0 is the x-coordinate of the nasal tip node coordinate data; and x1 is the x-coordinate of the upper spinal insertion point coordinate data. The pedestrian motion state characteristics include the characteristics of changes in the amplitude of leg movements and the characteristics of the degree of leg bending. The distance features between pedestrians and vehicles in all the training samples are combined into a first feature sample set; the first feature sample set is processed using a spectral clustering algorithm to obtain a first label set; The pedestrian head orientation features of all the training samples are combined into a second feature sample set; the second feature sample set is processed using the spectral clustering algorithm to obtain a second label set; The pedestrian motion state features of all the training samples are combined into a third feature sample set; the spectral clustering algorithm is used to process the third feature sample set to obtain a third label set; Using the first feature sample set and the first label set as input, a first classifier is trained using a classifier algorithm; using the second feature sample set and the second label set as input, a second classifier is trained using the same classifier algorithm; using the third feature sample set and the third label set as input, a third classifier is trained using the same classifier algorithm. The interaction state between pedestrians and vehicles is identified using the first classifier, the second classifier, and the third classifier. The interaction state between pedestrians and vehicles includes: the degree of approaching or moving away from the vehicle, the degree of perception of the vehicle, and the magnitude of movement.

2. The method for recognizing pedestrian and vehicle interaction states according to claim 1, characterized in that, The lidar point cloud data is obtained by lidar installed on the vehicle; the image data is obtained by camera installed on the vehicle.

3. The method for recognizing pedestrian and vehicle interaction states according to claim 1, characterized in that, The processing of the image data to obtain pedestrian head orientation features and pedestrian movement state features specifically includes: A pedestrian skeleton recognition method is used to identify each frame of the image data to obtain the pedestrian skeleton node coordinate data of each frame of the image; the pedestrian skeleton node coordinate data includes the nose tip node coordinate data, the upper spinal insertion point coordinate data, the lower spinal insertion point coordinate data, the left hip joint node coordinate data, the right hip joint node coordinate data, the left knee joint node coordinate data, the right knee joint node coordinate data, the left ankle joint node coordinate data, and the right ankle joint node coordinate data. For each frame of the image, the following calculations are performed: First, the angle between the line connecting the tip of the nose and the upper spinal insertion point and the horizontal line is calculated based on the nose tip coordinate data and the upper spinal insertion point coordinate data; Second, the pedestrian's spinal height is calculated based on the upper spinal insertion point coordinate data and the lower spinal insertion point coordinate data; Third, the distance between the pedestrian's knees is calculated based on the left knee joint coordinate data and the right knee joint coordinate data; Fourth, the distance between the pedestrian's ankles is calculated based on the left ankle joint coordinate data and the right ankle joint coordinate data; Fifth, the bending angle of the pedestrian's left leg is calculated based on the left hip joint coordinate data, the left knee joint coordinate data, and the left ankle joint coordinate data; Sixth, the bending angle of the pedestrian's right leg is calculated based on the right hip joint coordinate data, the right knee joint coordinate data, and the right ankle joint coordinate data. Calculate the average of the first included angle for all the images to obtain the pedestrian head orientation feature; Based on the spine height, the distance between the knees, and the distance between the ankles in all the images, calculate the variation characteristics of the range of motion of the legs; based on the bending angle of the left leg and the bending angle of the right leg in all the images, calculate the bending degree characteristics of the legs; combine the variation characteristics of the range of motion of the legs and the bending degree characteristics of the legs to form the pedestrian movement state characteristics.

4. The method for recognizing pedestrian and vehicle interaction states according to claim 3, characterized in that, The step of calculating the degree of bending of both legs based on the bending angle of the left leg and the bending angle of the right leg in all the images specifically includes: Calculate the average of the left leg bending angles for all the images to obtain the degree of left leg bending; Calculate the average of the right leg bending angles for all the images to obtain the degree of right leg bending; The first number of images in which the left leg bending angle is greater than the first preset angle is counted, and the second number of images in which the right leg bending angle is greater than the first preset angle is counted. The sum of the first number and the second number is then calculated as the ratio of the total number of frames in the images to obtain the first proportion. The third number of images in which the left leg bending angle is less than the second preset angle is counted, and the fourth number of images in which the right leg bending angle is less than the second preset angle is counted. The sum of the third and fourth numbers is then calculated as the ratio of the total number of frames in the images to obtain the second proportion. The degree of bending of the left leg, the degree of bending of the right leg, the first proportion, and the second proportion constitute the characteristic of the degree of bending of both legs.

5. The method for recognizing pedestrian and vehicle interaction states according to claim 1, characterized in that, The process of using spectral clustering algorithm to process the first feature sample set to obtain the first label set specifically includes: Perform spectral decomposition on the first feature sample set to obtain the feature spectrum; Projecting the data in the first feature sample set onto the feature spectrum yields the projected data; K-means clustering is performed on the projected data to obtain K types of clusters; The label for each of the aforementioned groups is determined to obtain the first label set.

6. A pedestrian and vehicle interaction status recognition system, characterized in that, include: The sample acquisition module is used to acquire multiple training samples; Each training sample includes lidar point cloud data and image data collected during vehicle operation; The feature extraction module is used to process the LiDAR point cloud data for each training sample to obtain the distance features between pedestrians and vehicles. Specifically, this includes: extracting the distance corresponding to each frame of point cloud data included in the LiDAR point cloud data, where the distance is the distance between pedestrians and vehicles; calculating the average of the distances corresponding to all the point cloud data to obtain the relative distance between pedestrians and vehicles; for each frame of point cloud data, calculating the difference between the distance corresponding to the point cloud data and the distance corresponding to the previous frame of point cloud data to obtain the distance difference; calculating the increase and decrease in the distance between pedestrians and vehicles based on the distance differences corresponding to all the point cloud data; combining the relative distance between pedestrians and vehicles, the increase and decrease in the distance between pedestrians and vehicles to form the distance features between pedestrians and vehicles; and processing the image data to obtain pedestrian head orientation features and pedestrian motion state features. The formula for calculating the increase in distance D2 between pedestrians and vehicles is: Where k is the total number of frames of point cloud data included in the lidar point cloud data; d i+1 d represents the distance corresponding to the point cloud data in the (i+1)th frame; i The distance corresponding to the point cloud data in the i-th frame; The formula for calculating the reduction in distance between pedestrians and vehicles, D3, is as follows: The pedestrian head orientation feature includes the head facing direction, and the formula for calculating the head facing direction H is as follows: Where, α 1,i The first included angle of the i-th frame of the image data; The formula for calculating the first included angle α1 is: Where y0 is the y-coordinate of the nasal tip node coordinate data; y1 is the y-coordinate of the upper spinal insertion point coordinate data; x0 is the x-coordinate of the nasal tip node coordinate data; and x1 is the x-coordinate of the upper spinal insertion point coordinate data. The pedestrian motion state characteristics include the characteristics of changes in the amplitude of leg movements and the characteristics of the degree of leg bending. The label generation module is used to: ... The training module is used to train a first classifier using the first feature sample set and the first label set as input, and to train a second classifier using the classifier algorithm using the second feature sample set and the second label set as input; and to train a third classifier using the classifier algorithm using the third feature sample set and the third label set as input. The recognition module is used to identify the interaction state between pedestrians and vehicles using the first classifier, the second classifier and the third classifier. The interaction state between pedestrians and vehicles includes: the degree of approaching or moving away from the vehicle, the degree of noticing the vehicle, and the magnitude of movement.

Images