A pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model

Abstract

The present invention discloses a method for predicting pedestrian crossing trajectories that integrates pedestrian intention and social force models. The method comprises the following steps: obtaining vehicle driving state information, an image in front of the vehicle, and pedestrian motion state information; detecting key points of the pedestrian's posture in the image and fitting a human posture skeleton; selecting key points that can express the pedestrian's intention based on the human posture skeleton, and calculating the distance and angle parameters between the key points; determining the pedestrian's intention; establishing the self-driving force, destination attraction, and mutual repulsion between the pedestrian and the vehicle when crossing the street, and obtaining the resultant force; and predicting the pedestrian's crossing trajectory based on a set time step. The pedestrian crossing trajectory prediction method of the present invention integrates the pedestrian intention recognition process and improves the traditional social force model. It can reflect the actual pedestrian crossing behavior and improve the accuracy of pedestrian crossing trajectory prediction.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving safety technology, and specifically relates to a pedestrian crossing trajectory prediction method that integrates pedestrian intention and social force models. Background Art

[0002] In recent years, the safety of autonomous vehicles has become a hot topic. Pedestrians, a crucial component of road traffic, often collide with autonomous vehicles when crossing on roads without traffic signals. Therefore, predicting pedestrian crossing trajectories plays a crucial role in autonomous vehicle safety decisions.

[0003] Currently, there are two main methods for predicting pedestrian crossing trajectories: one is based on data-driven network models, which utilize deep learning networks to build pedestrian trajectory prediction algorithms to obtain predicted values. The other is based on mechanical models, primarily pedestrian trajectory prediction algorithms based on social force models.

[0004] Pedestrians, as active decision-makers in road use, possess a certain degree of subjective perception when crossing the street, and their movements are characterized by high mobility. However, the aforementioned social force model-based approach simplifies pedestrians into particles, failing to account for the microscopic psychological changes experienced by pedestrians as active decision-makers when facing oncoming vehicles. This can lead to prediction bias in pedestrian crossing trajectories, causing intelligent vehicles to make inaccurate decisions, and seriously compromising the safety of both vehicles and pedestrians crossing the street. Summary of the Invention

[0005] To address the shortcomings of the aforementioned prior art, the present invention aims to provide a pedestrian crossing trajectory prediction method that integrates pedestrian intention and social force models. This method addresses the problem that existing pedestrian crossing trajectory prediction methods based on traditional social force models struggle to describe the micro-psychological changes of pedestrians when facing oncoming vehicles, resulting in inaccurate pedestrian crossing trajectory prediction. This method considers pedestrians as decision-makers regarding road use, considers their micro-psychological changes when facing oncoming vehicles, and integrates pedestrian intention recognition with social force models to predict pedestrian crossing trajectories. The proposed pedestrian intention recognition method, based on human posture skeleton estimation, can identify pedestrian crossing intention based on the current pedestrian posture skeleton using the LVQ neural network model. The social force model is used to describe the pedestrian's self-driving force, destination attraction, and pedestrian-vehicle repulsion when crossing the street. The crossing intention is quantified in the model. Finally, based on a set time step, a predicted pedestrian trajectory is obtained for a preset duration, thereby improving the accuracy of pedestrian crossing trajectory prediction.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The present invention provides a pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model, comprising the following steps:

[0008] (1) Obtaining the vehicle's driving status information, the vehicle's front image, and the pedestrian's motion status information;

[0009] (2) Based on the image in front of the vehicle collected in step (1), the Openpose algorithm is used to detect the key points of the pedestrian's posture in the image and fit the human posture skeleton; based on the human posture skeleton, the key points that can express the pedestrian's intention are selected, and the distance and angle parameters between the key points are calculated;

[0010] (3) Inputting the distance and angle parameters in step (2) as feature vectors into the trained LVQ neural network model to determine the pedestrian's intention;

[0011] (4) Establish the self-driving force, destination attraction, and mutual repulsion between pedestrians and vehicles when pedestrians cross the street, and calculate the resultant force, quantify the pedestrian intention in the form of weights in the social force model; predict the pedestrian crossing trajectory based on the set time step.

[0012] Furthermore, the specific steps of step (1) are as follows:

[0013] (11) Using the vehicle CAN bus to obtain the vehicle's current driving state information, the driving state information including the vehicle's driving speed information and position information;

[0014] (12) using a camera mounted on the front windshield of the vehicle to obtain an image in front of the vehicle;

[0015] (13) using a millimeter-wave radar installed on the front bumper of the vehicle to obtain relative motion state information of the pedestrian, wherein the relative motion state information of the pedestrian includes position information and speed information of the pedestrian relative to the vehicle;

[0016] (14) Based on the vehicle driving state information and the pedestrian's relative motion state information, the pedestrian motion state information is obtained by using the parameter conversion from the millimeter wave radar coordinate system to the world coordinate system.

[0017] Furthermore, the specific steps of step (2) are as follows:

[0018] (21) Based on the Openpose algorithm, the image in front of the vehicle obtained in step (1) is initialized through the first ten layers of the VGG-19 convolutional network to obtain a feature map F, and a dual-branch multi-stage convolutional network is used to detect the key points of the pedestrian crossing the street and fit the human posture skeleton;

[0019] (22) (The human body posture skeleton of pedestrians presents different characteristics under different intentions of crossing the street) According to the characteristics of pedestrians facing the left side of the oncoming vehicle when crossing the street, key points that can express the pedestrian's intention are selected. Specifically, the neck, left shoulder, left elbow, left wrist, left hip, left knee, left ankle, right hip, right knee, and right ankle in the human body posture skeleton are selected and represented by the numbers 1, 5, 6, 7, 11, 12, 13, 8, 9, and 10 respectively, and there are connection relationships 1-5-6-7, 11-12-13, and 8-9-10;

[0020] (23) Calculate the distance and angle between key points as feature parameters: calculate the distance d1 from the left shoulder to the left wrist (i.e., between key points 5 and 7), the distance d2 between the left knee and right knee joints (i.e., between key points 9 and 12), and the distance d3 between the left ankle and right ankle joints (i.e., between key points 10 and 13); calculate the angle α1 between the left forearm and the left hind arm (i.e., between key points 5-6 and 6-7), the left leg joint angle α2 (i.e., between key points 11-12 and 12-13), and the right leg joint angle α3 (i.e., between key points 8-9 and 9-10);

[0021] The distance between two key points mentioned above is expressed as Euclidean distance:

[0022]

[0023] Where ||v|| is the Euclidean distance between two key points, ΔX and ΔY are the distance differences between the two key points in the X-axis and Y-axis directions respectively;

[0024] The angle between the line segment formed by the two key points and the X-axis is recorded as α. The three key points form a triangle, and the angle is calculated as follows:

[0025]

[0026]

[0027] Furthermore, the step (21) specifically includes:

[0028] (211) The dual-branch multi-stage convolutional network contains two branches. The first branch is used to predict the confidence map S of the key points of the detected body parts. t , using non-maximum suppression to obtain the candidate key point set and obtain the position of the human posture key points in the image; the second branch is used to predict the part affinity field L connecting the key points of the body parts and limbs t , calculate the global optimal solution for connecting the key points of the image and obtain the human body posture skeleton;

[0029] (212) The L2 norm is used to represent the loss function of the two branches, namely, the loss function f used to detect the confidence map of the key points of the body parts. st and the loss function f used to predict the part affinity field between the key points connecting the body parts L t, so that the final total loss function f is minimized;

[0030]

[0031]

[0032]

[0033] Where, Represents the true value of the confidence map of the key points of the body parts; represents the predicted value of the confidence map of the key points of the body part at stage t; represents the true value of the site affinity field; represents the predicted value of the part affinity field at the tth stage; W(p) represents the weight of the loss function; J represents the number of samples in the confidence map branch; C represents the number of samples in the part affinity field branch; f represents the final total loss function; t, T represent the iteration stage.

[0034] Furthermore, the specific steps of step (3) are as follows:

[0035] (31) The six characteristic parameters calculated in step (23) are input as characteristic vectors into the trained LVQ neural network model to determine the pedestrian's intention; the pedestrian's intention includes three types of crossing intentions: stopping to avoid, passing normally, and accelerating to pass.

[0036] Furthermore, the step (31) specifically includes:

[0037] (311) Construct an LVQ neural network model that can identify pedestrians’ current intention to cross the street based on human body posture skeleton feature vectors;

[0038] (312) The LVQ neural network model includes: an input layer, a competitive layer, and a linear output layer; the number of neurons in the input layer is selected to be m, the number of neurons in the competitive layer is selected to be n, and the number of neurons in the output layer is selected to be k. The number of neurons in the input layer is determined according to the number of feature parameters used to identify pedestrian intentions, the number of neurons in the output layer is determined according to the number of pedestrian crossing intentions set, and the number of neurons in the competitive layer is determined by empirical estimation methods and then by experiments. The maximum number of iterations is p.

[0039] in,

[0040]

[0041] (313) Training process of LVQ neural network model: Initialize the weight ω between the input layer and the competition layer ijAnd the learning rate η, the input layer vector is selected as follows:

[0042]

[0043] The input vector Normalize to get the normalized input vector x, and send it to the input layer to calculate the distance l between the competition layer neuron and the input vector i as follows:

[0044]

[0045] Where, l i is the distance between the competition layer neuron and the input vector; ω ij is the weight between input layer neuron j and competition layer neuron i; x j is the normalized input layer neuron;

[0046] The competitive layer neuron with the smallest distance to the weight vector is selected and recorded as the winning neuron, so as to find the linear output layer neuron connected to it; if the category corresponding to the linear output layer neuron is consistent with the category of the input vector, it is called correct classification, and the weight is adjusted according to formula (10); otherwise, it is called incorrect classification, and the weight is adjusted according to formula (11), and the next round of training process is entered until the pre-set maximum number of iterations p is met;

[0047] ω ij_new =ω ij +η(x-ω ij ) (10)

[0048] ω ij_new =ω ij -η(x-ω ij ) (11)

[0049] Where, ω ij_new is the adjusted weight; η is the learning rate; x is the normalized input vector;

[0050] (314) Before training the LVQ neural network model, a feature vector data set that can represent the pedestrian's intention is prepared as the LVQ neural network input data, and the corresponding pedestrian intention results are marked as the LVQ neural network output results; 75% of the data volume is taken as the training data set, and the remaining 25% is taken as the test data set, and the LVQ neural network is trained and tested according to the method of step (313).

[0051] Furthermore, the specific steps of step (4) are as follows:

[0052] (41) When pedestrians cross the street on a road without traffic signals, the social force model is used to describe the pedestrian's movement. Based on the vehicle driving state information and pedestrian movement state information in step (1), a self-driving force is established. Destination's appeal to pedestrians and the mutual repulsion between people and vehicles

[0053] (42) Quantify pedestrian intentions in the form of weights in the social force model to obtain self-driving force Destination's appeal to pedestrians and the mutual repulsion between people and vehicles The combined force among the three Predict pedestrian crossing trajectory based on the set time step;

[0054]

[0055]

[0056]

[0057] in, is the net force acting on the pedestrian at time t, without considering the pedestrian's mass. The above forces can all be regarded as accelerations; is the vector representation of the pedestrian's velocity at time t; β1, β2 and β3 are weight parameters; and are the vector representations of the pedestrian’s position at time t and time t+1 respectively; and are the vector representations of the pedestrian’s velocity at time t and time t+1 respectively; Δt is the time step.

[0058] Furthermore, the (41) specifically includes:

[0059] (411) Establishing self-driving force for pedestrians when crossing the street If there is no interference, the pedestrian will move at a speed v d In the desired direction Travel; Due to the necessary deceleration or evasive action when facing oncoming vehicles, the actual speed of the pedestrian and expected speed The difference is determined by a certain relaxation time τ d To correct to the desired speed

[0060]

[0061]

[0062]

[0063] in, is the vector representation of the destination location; is the vector representation of the pedestrian’s position at time t;

[0064] (412) Establish a destination attraction for pedestrians when crossing the street Expressed as:

[0065]

[0066]

[0067] Among them, A ap and B ap is the social force model parameter; R p is the radius of force exerted by pedestrians; d ap (t) is the distance between the pedestrian and the destination at time t; is the force unit vector, directed from the pedestrian p to the destination a;

[0068] (413) Establishing mutual repulsion between people and vehicles By psychological force and physical forces composition;

[0069]

[0070]

[0071]

[0072]

[0073]

[0074] Among them, A vp 、B vp , k and K are parameters of the social force model; R v is the force radius of the vehicle; d vp (t) is the distance between the pedestrian and the vehicle at time t; is the vector representation of the vehicle's position at time t; Δv vp (t) is the instantaneous speed difference between the pedestrian and the vehicle at time t; is the normal force unit vector; is the tangential force unit vector.

[0075] Furthermore, the step (42) specifically includes:

[0076] (421) When it is judged that the pedestrian intends to stop and avoid, the pedestrian's speed at this time is 0; the pedestrian stops and waits for the vehicle to pass, and β1 = β2 = β3 = 0 is selected at this time, that is, the resultant force on the pedestrian at this time is 0; at the same time, the vehicle can continue to pass safely in the current driving state;

[0077] (422) When the pedestrian's intention is to pass normally, select β1 = β2 = β3 = 1, calculate the pedestrian's self-driving force, the destination attraction, and the mutual repulsion between the pedestrian and the vehicle, and obtain the resultant force. According to the set time step, the pedestrian's position vector and velocity vector at the next moment are obtained; at the same time, the vehicle performs obstacle avoidance path planning;

[0078] (423) When it is judged that the pedestrian intends to accelerate through, ignore the partial repulsive force caused by the vehicle, select β1=1, β2>1, β3<1, calculate the self-driving force, destination attraction and mutual repulsion of the pedestrian and the vehicle at this time, and obtain the resultant force. According to the set time step, the position vector and velocity vector of the pedestrian at the next moment are obtained; at the same time, the vehicle performs obstacle avoidance path planning.

[0079] Beneficial effects of the present invention:

[0080] 1. The pedestrian crossing trajectory prediction method of the present invention takes into account the subjective initiative of pedestrians when facing oncoming vehicles when crossing the street. It uses the pedestrian's posture skeleton to estimate the pedestrian's crossing intention, providing a basis for crossing trajectory prediction;

[0081] 2. The pedestrian crossing trajectory prediction method of the present invention integrates the pedestrian intention recognition process and improves the traditional social force model. It can reflect the actual pedestrian crossing behavior and improve the accuracy of pedestrian crossing trajectory prediction;

[0082] 3. The method of the present invention can provide a basis for subsequent decision-making and planning of smart cars, which is beneficial to improving the safety of smart cars driving and pedestrians crossing the street. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a principle block diagram of the method of the present invention;

[0084] Figure 2 Schematic diagram of human body posture skeleton;

[0085] Figure 3 Schematic diagram of pedestrian intention recognition based on human posture skeleton estimation;

[0086] Figure 4 Schematic diagram of the LVQ neural network model structure;

[0087] Figure 5 Schematic diagram of pedestrian forces based on the social force model. DETAILED DESCRIPTION

[0088] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0089] Reference Figure 1 As shown, a pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model of the present invention includes the following steps:

[0090] (1) Obtaining the vehicle's driving status information, the image in front of the vehicle, and the pedestrian's motion status information; specifically including:

[0091] (11) Using the vehicle CAN bus to obtain the vehicle's current driving state information, the driving state information including the vehicle's driving speed information and position information;

[0092] (12) using a camera mounted on the front windshield of the vehicle to obtain an image in front of the vehicle;

[0093] (13) using a millimeter-wave radar installed on the front bumper of the vehicle to obtain relative motion state information of the pedestrian, wherein the relative motion state information of the pedestrian includes position information and speed information of the pedestrian relative to the vehicle;

[0094] (14) Based on the vehicle driving state information and the pedestrian's relative motion state information, the pedestrian motion state information is obtained by using the parameter conversion from the millimeter wave radar coordinate system to the world coordinate system.

[0095] (2) Based on the image in front of the vehicle collected in step (1), the Openpose algorithm is used to detect the key points of the pedestrian crossing the street in the image and fit the human posture skeleton; based on the human posture skeleton, the key points that can express the pedestrian's intention are selected, and the distance and angle parameters between the key points are calculated; Figure 2 As shown; specifically including:

[0096] (21) Based on the Openpose algorithm, the image in front of the vehicle obtained in step (1) is initialized through the first ten layers of the VGG-19 convolutional network to obtain a feature map F, and a dual-branch multi-stage convolutional network is used to detect the key points of the pedestrian crossing the street and fit the human posture skeleton;

[0097] (22) (The human body posture skeleton of pedestrians presents different characteristics under different intentions of crossing the street) According to the characteristics of pedestrians facing the left side of the oncoming vehicle when crossing the street, key points that can express the pedestrian's intention are selected. Specifically, the neck, left shoulder, left elbow, left wrist, left hip, left knee, left ankle, right hip, right knee, and right ankle in the human body posture skeleton are selected and represented by the numbers 1, 5, 6, 7, 11, 12, 13, 8, 9, and 10 respectively, and there are connection relationships 1-5-6-7, 11-12-13, and 8-9-10;

[0098] (23) Calculate the distances and angles between key points as feature parameters: Calculate the distance d1 from the left shoulder to the left wrist (i.e., between key points 5 and 7), the distance d2 between the left and right knee joints (i.e., between key points 9 and 12), and the distance d3 between the left and right ankle joints (i.e., between key points 10 and 13); Calculate the angle α1 between the left forearm and the left hind arm (i.e., between key points 5-6 and 6-7), the left leg joint angle α2 (i.e., between key points 11-12 and 12-13), and the right leg joint angle α3 (i.e., between key points 8-9 and 9-10);

[0099] The distance between two key points mentioned above is expressed as Euclidean distance:

[0100]

[0101] Where ||v|| is the Euclidean distance between two key points, ΔX and ΔY are the distance differences between the two key points in the X-axis and Y-axis directions respectively;

[0102] The angle between the line segment formed by the two key points and the X-axis is recorded as α. The three key points form a triangle, and the angle is calculated as follows:

[0103]

[0104]

[0105] Wherein, the step (21) specifically includes:

[0106] (211) The dual-branch multi-stage convolutional network contains two branches. The first branch is used to predict the confidence map S of the key points of the detected body parts. t , using non-maximum suppression to obtain the candidate key point set and obtain the position of the human posture key points in the image; the second branch is used to predict the part affinity field L connecting the key points of the body parts and limbs t , calculate the global optimal solution for connecting the key points of the image and obtain the human body posture skeleton;

[0107] (212) The L2 norm is used to represent the loss function of the two branches, namely, the loss function f used to detect the confidence map of the key points of the body parts. s t and a loss function for predicting the part affinity field connecting the limb keypoints of the body parts Minimize the final total loss function f;

[0108]

[0109]

[0110]

[0111] Where, Represents the true value of the confidence map of the key points of the body parts; represents the predicted value of the confidence map of the key points of the body part at stage t; represents the true value of the site affinity field; represents the predicted value of the part affinity field at the tth stage; W(p) represents the weight of the loss function; J represents the number of samples in the confidence map branch; C represents the number of samples in the part affinity field branch; f represents the final total loss function; t, T represent the iteration stage.

[0112] (3) Inputting the distance and angle parameters in step (2) as feature vectors into the trained LVQ neural network model to determine the pedestrian's intention; Figure 3 and Figure 4 As shown; specifically including:

[0113] (31) The six characteristic parameters calculated in step (23) are input as characteristic vectors into the trained LVQ neural network model to determine the pedestrian's intention; the pedestrian's intention includes three types of crossing intentions: stopping to avoid, passing normally, and accelerating to pass.

[0114] Specifically, the step (31) includes:

[0115] (311) Construct an LVQ neural network model that can identify pedestrians’ current intention to cross the street based on human body posture skeleton feature vectors;

[0116] (312) The LVQ neural network model includes: an input layer, a competitive layer, and a linear output layer; the number of neurons in the input layer is selected to be m, the number of neurons in the competitive layer is selected to be n, and the number of neurons in the output layer is selected to be k. The number of neurons in the input layer is determined according to the number of feature parameters used to identify pedestrian intentions, the number of neurons in the output layer is determined according to the number of pedestrian crossing intentions set, and the number of neurons in the competitive layer is determined by empirical estimation methods and then by experiments. The maximum number of iterations is p.

[0117] in,

[0118]

[0119] (313) Training process of LVQ neural network model: Initialize the weight ω between the input layer and the competition layer ij And the learning rate η, the input layer vector is selected as follows:

[0120]

[0121] The input vector Normalize to get the normalized input vector x, and send it to the input layer to calculate the distance l between the competition layer neuron and the input vector i as follows:

[0122]

[0123] Where, l i is the distance between the competition layer neuron and the input vector; ω ij is the weight between input layer neuron j and competition layer neuron i; x j is the normalized input layer neuron;

[0124] The competitive layer neuron with the smallest distance to the weight vector is selected and recorded as the winning neuron, so as to find the linear output layer neuron connected to it; if the category corresponding to the linear output layer neuron is consistent with the category of the input vector, it is called correct classification, and the weight is adjusted according to formula (10); otherwise, it is called incorrect classification, and the weight is adjusted according to formula (11), and the next round of training process is entered until the pre-set maximum number of iterations p is met;

[0125] ω ij_new =ω ij +η(x-ω ij ) (10)

[0126] ω ij_new =ω ij -η(x-ω ij ) (11)

[0127] Where, ω ij_new is the adjusted weight; η is the learning rate; x is the normalized input vector;

[0128] (314) Before training the LVQ neural network model, a feature vector data set that can represent the pedestrian's intention is prepared as the LVQ neural network input data, and the corresponding pedestrian intention results are marked as the LVQ neural network output results; 75% of the data volume is taken as the training data set, and the remaining 25% is taken as the test data set, and the LVQ neural network is trained and tested according to the method of step (313).

[0129] (4) Establish the self-driving force, destination attraction, and repulsive force between pedestrians and vehicles when pedestrians cross the street, and calculate the resultant force, quantify the pedestrian intention in the form of weight in the social force model; predict the pedestrian crossing trajectory according to the set time step; refer to Figure 5 As shown; specifically including:

[0130] (41) When pedestrians cross the street on a road without traffic signals, the social force model is used to describe the pedestrian's movement. Based on the vehicle driving state information and pedestrian movement state information in step (1), a self-driving force is established. Destination's appeal to pedestrians and the mutual repulsion between people and vehicles

[0131] (42) Quantify pedestrian intentions in the form of weights in the social force model to obtain self-driving force Destination's appeal to pedestrians and the mutual repulsion between people and vehicles The combined force among the three Predict pedestrian crossing trajectory based on the set time step;

[0132]

[0133]

[0134]

[0135] in, is the net force acting on the pedestrian at time t, without considering the pedestrian's mass. The above forces can all be regarded as accelerations; is the vector representation of the pedestrian's velocity at time t; β1, β2 and β3 are weight parameters; and are the vector representations of the pedestrian’s position at time t and time t+1 respectively; and are the vector representations of the pedestrian’s velocity at time t and time t+1 respectively; Δt is the time step.

[0136] Wherein, said (41) specifically includes:

[0137] (411) Establishing self-driving force for pedestrians when crossing the street If there is no interference, the pedestrian will move at a speed v d In the desired direction Travel; Due to the necessary deceleration or evasive action when facing oncoming vehicles, the actual speed of the pedestrian and expected speed The difference is determined by a certain relaxation time τ d To correct to the desired speed

[0138]

[0139]

[0140]

[0141] in, is the vector representation of the destination location; is the vector representation of the pedestrian’s position at time t;

[0142] (412) Establish a destination attraction for pedestrians when crossing the street Expressed as:

[0143]

[0144]

[0145] Among them, A ap and B ap is the social force model parameter; R p is the radius of force exerted by pedestrians; d ap (t) is the distance between the pedestrian and the destination at time t; is the force unit vector, directed from the pedestrian p to the destination a;

[0146] (413) Establishing mutual repulsion between people and vehicles By psychological force and physical forces composition;

[0147]

[0148]

[0149]

[0150]

[0151]

[0152] Among them, A vp 、B vp , k and K are parameters of the social force model; R v is the force radius of the vehicle; d vp (t) is the distance between the pedestrian and the vehicle at time t; is the vector representation of the vehicle's position at time t; Δv vp (t) is the instantaneous speed difference between the pedestrian and the vehicle at time t; is the normal force unit vector; is the tangential force unit vector.

[0153] Wherein, the step (42) specifically includes:

[0154] (421) When it is determined that the pedestrian intends to stop and avoid, the pedestrian's speed at this time is 0; the pedestrian stops and waits for the vehicle to pass, and β1 = β2 = β3 = 0 is selected at this time, that is, the resultant force on the pedestrian is 0; at the same time, the intelligent vehicle can continue to drive safely in the current state;

[0155] (422) When the pedestrian's intention is to pass normally, β1 = β2 = β3 = 1 is selected, and the self-driving force, the attraction of the destination, and the mutual repulsion of the pedestrian and the vehicle are calculated, and the resultant force is obtained. According to the set time step, the position vector and velocity vector of the pedestrian at the next moment are obtained; at the same time, the intelligent vehicle performs obstacle avoidance path planning;

[0156] (423) When it is judged that the pedestrian intends to accelerate through, ignore the partial repulsive force caused by the vehicle, select β1=1, β2>1, β3<1, calculate the self-driving force, destination attraction and mutual repulsion of the pedestrian and the vehicle at this time, and obtain the resultant force. According to the set time step, the position vector and velocity vector of the pedestrian at the next moment are obtained; at the same time, the intelligent vehicle performs obstacle avoidance path planning.

[0157] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be considered as the scope of protection of the present invention.

Claims

1. A pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model, characterized by: The following steps are involved: (1) Obtaining the vehicle's driving status information, the vehicle's front image, and the pedestrian's motion status information; (2) Based on the image in front of the vehicle collected in step (1), the Openpose algorithm is used to detect the key points of the pedestrian's posture in the image and fit the human posture skeleton; based on the human posture skeleton, the key points that can express the pedestrian's intention are selected, and the distance and angle parameters between the key points are calculated; (3) Inputting the distance and angle parameters in step (2) as feature vectors into the trained LVQ neural network model to determine the pedestrian's intention; (4) Establish the self-driving force, destination attraction, and pedestrian-vehicle repulsion that pedestrians experience when crossing the street, and calculate the resultant force. The pedestrian's intention is quantified in the form of weights in the social force model; the pedestrian's crossing trajectory is predicted based on the set time step. The specific steps of step (4) are as follows: (41) When pedestrians cross the street on a road without traffic signals, the social force model is used to describe the pedestrian's movement. Based on the vehicle driving state information and pedestrian movement state information in step (1), a self-driving force is established. Destination's appeal to pedestrians and the mutual repulsion between people and vehicles (42) Quantify pedestrian intentions in the form of weights in the social force model to obtain self-driving force Destination's appeal to pedestrians and the mutual repulsion between people and vehicles The combined force among the three Predict pedestrian crossing trajectory based on the set time step; in, is the net force acting on the pedestrian at time t; is the vector representation of the pedestrian's velocity at time t; β1, β2 and β3 are weight parameters; and are the vector representations of the pedestrian’s position at time t and time t+1 respectively; and are the vector representations of the pedestrian’s velocity at time t and time t+1 respectively; Δt is the time step.

2. The pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model according to claim 1 is characterized in that: The specific steps of step (1) are as follows: (11) Using the vehicle CAN bus to obtain the vehicle's current driving state information, the driving state information including the vehicle's driving speed information and position information; (12) using a camera mounted on the front windshield of the vehicle to obtain an image in front of the vehicle; (13) using a millimeter-wave radar installed on the front bumper of the vehicle to obtain relative motion state information of the pedestrian, wherein the relative motion state information of the pedestrian includes position information and speed information of the pedestrian relative to the vehicle; (14) Based on the vehicle driving state information and the pedestrian's relative motion state information, the pedestrian motion state information is obtained by using the parameter conversion from the millimeter wave radar coordinate system to the world coordinate system.

3. The pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model according to claim 1 is characterized in that: The specific steps of step (2) are as follows: (21) Based on the Openpose algorithm, the image in front of the vehicle obtained in step (1) is initialized through the first ten layers of the VGG-19 convolutional network to obtain a feature map F, and a dual-branch multi-stage convolutional network is used to detect the key points of the pedestrian crossing the street and fit the human posture skeleton; (22) According to the characteristics of pedestrians facing the left side of the road when crossing the street, key points that can express the pedestrian's intention are selected, specifically the neck, left shoulder, left elbow, left wrist, left hip, left knee, left ankle, right hip, right knee, and right ankle in the human posture skeleton; (23) Calculate the distance and angle between key points as feature parameters: calculate the distance d1 from the left shoulder to the left wrist, the distance d2 between the left knee and the right knee joints, and the distance d3 between the left ankle and the right ankle joints; calculate the angle α1 between the left forearm and the left hind arm, the left leg joint angle α2, and the right leg joint angle α3; The distance between the above key points is expressed as Euclidean distance: Where ||v|| is the Euclidean distance between two key points, ΔX and ΔY are the distance differences between the two key points in the X-axis and Y-axis directions respectively; The angle between the line segment formed by the two key points and the X-axis is recorded as α. The three key points form a triangle, and the angle is calculated as follows:

4. The pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model according to claim 3 is characterized in that: The step (21) specifically includes: (211) The dual-branch multi-stage convolutional network contains two branches. The first branch is used to predict the confidence map S of the key points of the detected body parts. t , using non-maximum suppression to obtain the candidate key point set and obtain the position of the human posture key points in the image; the second branch is used to predict the part affinity field L connecting the key points of the body parts and limbs t , calculate the global optimal solution for connecting the key points of the image and obtain the human body posture skeleton; (212) The L2 norm is used to represent the loss function of the two branches, that is, the loss function used to detect the confidence map of the key points of the body parts. and a loss function for predicting the part affinity field connecting the limb keypoints of the body parts Minimize the final total loss function f; Where, Represents the true value of the confidence map of the key points of the body parts; represents the predicted value of the confidence map of the key points of the body part at stage t; represents the true value of the site affinity field; represents the predicted value of the part affinity field at the tth stage; W(p) represents the weight of the loss function; J represents the number of samples in the confidence map branch, and C represents the number of samples in the part affinity field branch; f represents the final total loss function; t, T represent the iteration stage.

5. The pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model according to claim 3 is characterized in that: The specific steps of step (3) are as follows: (31) The six characteristic parameters calculated in step (23) are input as characteristic vectors into the trained LVQ neural network model to determine the pedestrian's intention; the pedestrian's intention includes three types of crossing intentions: stopping to avoid, passing normally, and accelerating to pass.

6. The pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model according to claim 5 is characterized in that: The step (31) specifically includes: (311) Construct an LVQ neural network model that can identify pedestrians’ current intention to cross the street based on human body posture skeleton feature vectors; (312) The LVQ neural network model includes: an input layer, a competitive layer, and a linear output layer; the number of neurons in the input layer is selected to be m, the number of neurons in the competitive layer is selected to be n, and the number of neurons in the output layer is selected to be k. The number of neurons in the input layer is determined according to the number of feature parameters used to identify pedestrian intentions, the number of neurons in the output layer is determined according to the number of pedestrian crossing intentions set, and the number of neurons in the competitive layer is determined by empirical estimation methods and then by experiments. The maximum number of iterations is p. in, (313) Training process of LVQ neural network model: Initialize the weight ω between the input layer and the competition layer ij And the learning rate η, the input layer vector is selected as follows: The input vector Normalize to get the normalized input vector x, and send it to the input layer to calculate the distance l between the competition layer neuron and the input vector i as follows: Where, l i is the distance between the competition layer neuron and the input vector; ω ij is the weight between input layer neuron j and competition layer neuron i; x j is the normalized input layer neuron; The competitive layer neuron with the smallest distance to the weight vector is selected and recorded as the winning neuron, so as to find the linear output layer neuron connected to it; if the category corresponding to the linear output layer neuron is consistent with the category of the input vector, it is called correct classification, and the weight is adjusted according to formula (10); otherwise, it is called incorrect classification, and the weight is adjusted according to formula (11), and the next round of training process is entered until the pre-set maximum number of iterations p is met; oh ij_new =ω ij +η(x-ω ij ) (10) oh ij_new =ω ij -η(x-ω ij ) (11) Where, ω ij_new is the adjusted weight; η is the learning rate; x is the normalized input vector; (314) Before training the LVQ neural network model, a feature vector data set that can represent the pedestrian's intention is prepared as the LVQ neural network input data, and the corresponding pedestrian intention results are marked as the LVQ neural network output results; 75% of the data volume is taken as the training data set, and the remaining 25% is taken as the test data set, and the LVQ neural network is trained and tested according to the method of step (313).

7. The pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model according to claim 1 is characterized in that: Said (41) specifically includes: (411) Establishing self-driving force for pedestrians when crossing the street If there is no interference, the pedestrian will move at a speed v d In the desired direction Travel; Due to the necessary deceleration or evasive action when facing oncoming vehicles, the actual speed of the pedestrian and expected speed The difference is measured by the relaxation time τ d To correct to the desired speed in, is the vector representation of the destination location; is the vector representation of the pedestrian’s position at time t; (412) Establish a destination attraction for pedestrians when crossing the street Expressed as: Among them, A ap and B ap is the social force model parameter; R p is the radius of force exerted by pedestrians; d ap (t) is the distance between the pedestrian and the destination at time t; is the force unit vector, directed from the pedestrian p to the destination a; (413) Establishing mutual repulsion between people and vehicles By psychological force and physical forces composition; Among them, A vp 、B vp , k and K are parameters of the social force model; R v is the force radius of the vehicle; d vp (t) is the distance between the pedestrian and the vehicle at time t; is the vector representation of the vehicle's position at time t; Δv vp (t) is the instantaneous speed difference between the pedestrian and the vehicle at time t; is the normal force unit vector; is the tangential force unit vector.

8. The pedestrian crossing trajectory prediction method integrating pedestrian intention and social force model according to claim 1 is characterized in that: The step (42) specifically includes: (421) When it is judged that the pedestrian intends to stop and avoid, the pedestrian's speed at this time is 0; the pedestrian stops and waits for the vehicle to pass, and β1 = β2 = β3 = 0 is selected at this time, that is, the resultant force on the pedestrian at this time is 0; at the same time, the vehicle can continue to pass safely in the current driving state; (422) When the pedestrian's intention is to pass normally, select β1 = β2 = β3 = 1, calculate the pedestrian's self-driving force, the destination attraction, and the mutual repulsion between the pedestrian and the vehicle, and obtain the resultant force. According to the set time step, the pedestrian's position vector and velocity vector at the next moment are obtained; at the same time, the vehicle performs obstacle avoidance path planning; (423) When it is judged that the pedestrian intends to accelerate through, ignore the partial repulsive force caused by the vehicle, select β1=1, β2>1, β3<1, calculate the self-driving force, destination attraction and mutual repulsion of the pedestrian and the vehicle at this time, and obtain the resultant force. According to the set time step, the position vector and velocity vector of the pedestrian at the next moment are obtained; at the same time, the vehicle performs obstacle avoidance path planning.

Images