Pedestrian trajectory prediction method and system based on Deep-SORT and least square method

Abstract

The invention discloses a pedestrian trajectory prediction method and system based on Deep-SORT and a least square method, and solves the problems of low pedestrian trajectory prediction precision and poor real-time performance at present. Firstly, pedestrian detection is realized by using a Yolov7 algorithm. And then, tracking of a pedestrian trajectory is carried out based on a Deep-SORT model of a Kalman filtering algorithm. And finally, based on a depth camera coordinate transformation principle, calculating a three-dimensional trajectory coordinate of the pedestrian at any point in a camera coordinate system. The three-dimensional trajectory of the pedestrian is decoupled into a transverse two-dimensional trajectory and a longitudinal two-dimensional trajectory through a vertical projection method, the transverse two-dimensional trajectory and the longitudinal two-dimensional trajectory of the pedestrian are predicted through a least square method, the transverse predicted trajectory and the longitudinal predicted trajectory are synthesized into the three-dimensional trajectory of the pedestrian, and real-time single-target pedestrian trajectory prediction and multi-target pedestrian trajectory prediction are achieved. The algorithm provided by the invention has the characteristics of high prediction precision and high real-time performance, and can meet the pedestrian anti-collision requirement of the automatic driving automobile.

Description

Technical Field

[0001] The present invention relates to a pedestrian trajectory prediction method and system based on Deep-SORT and least squares method, belonging to the field of computer vision and intelligent driving. Background Art

[0002] There are many risk issues on the roads where self-driving cars are driving, mainly involving potential conflicts between cars and pedestrians at intersections. In order to reduce the probability of traffic accidents between cars and pedestrians, it is necessary to accurately detect pedestrians and accurately predict their trajectories.

[0003] Pedestrian detection is the basis of pedestrian trajectory prediction. Currently, autonomous driving systems often use laser radar, millimeter-wave radar and cameras for detection. Laser radar is expensive and requires a lot of calculations, making it difficult to apply in batches; millimeter-wave radar has difficulty accurately distinguishing pedestrians from other targets; and ordinary cameras have shortcomings such as poor ranging accuracy.

[0004] In terms of pedestrian trajectory prediction, Japanese scholars Takuma et al. used a vehicle-mounted forward-looking camera to create an FPL pedestrian dataset and predicted the trajectory of pedestrians in the video. Park et al. proposed a method for first-person trajectory prediction based on data retrieval, using AlexNet to extract scene features to establish an EgoRetinal map, mapping the training sample path to the video and predicting the pedestrian path in the first person perspective by estimating the blank area behind the occluded object. Su et al. applied the data retrieval method to the scene of a virtual game and predicted the trajectory of basketball players in the first person perspective. Based on the method of predicting pedestrian trajectories in steps studied by previous researchers, Wang et al. proposed a model for SGNet-driven trajectory prediction, which uses an autoencoder-decoder structure with conditional variation to predict pedestrian trajectories in multiple time periods. This method can predict the motion trajectory of pedestrians under the vehicle-mounted forward-looking camera, but does not integrate the pedestrian's walking posture information, and does not pay attention to the impact of camera motion errors. Therefore, the accuracy of pedestrian trajectory prediction in the first person perspective is poor.

[0005] In view of the above problems, based on the Astra Pro 3D depth camera, this paper proposes a pedestrian trajectory prediction method based on the fusion of Deep-SORT and the least squares method to improve the accuracy and real-time performance of pedestrian trajectory prediction. By consulting the literature, there is currently no relevant report on the use of an algorithm that combines Deep-SORT with the least squares method in pedestrian trajectory prediction technology. Summary of the invention

[0006] The purpose of the present invention is to propose a pedestrian trajectory prediction method and system based on the fusion of Deep-SORT and least squares method, so as to solve the problems of low accuracy and poor real-time performance of pedestrian trajectory prediction. First, pedestrian detection is realized by using Yolov7 algorithm. Then, pedestrian trajectory is tracked by using Deep-SORT based on Kalman filter algorithm. Finally, based on the principle of deep camera coordinate transformation, the three-dimensional trajectory coordinates of the pedestrian at any point in the camera coordinate system are calculated, and the three-dimensional trajectory of the pedestrian is decoupled into horizontal and vertical two-dimensional trajectories by vertical projection method, and the horizontal and vertical two-dimensional trajectories of the pedestrian are predicted by least squares method, and the horizontal and vertical predicted trajectories are synthesized into the three-dimensional trajectory of the pedestrian, realizing real-time single-target pedestrian trajectory prediction and multi-target pedestrian trajectory prediction. The algorithm proposed by the present invention has the characteristics of high prediction accuracy and high real-time performance, and can meet the pedestrian anti-collision requirements of autonomous driving vehicles.

[0007] The technical solution of the present invention:

[0008] The present invention provides a pedestrian trajectory prediction system based on Deep-SORT and least squares method (such as Figure 1 As shown in the figure, a schematic diagram of the pedestrian trajectory prediction algorithm fused with Yolov7 and Deep-SORT) includes Astra Pro 3D depth camera, Yolov7 algorithm module, Kalman filter algorithm module, coordinate conversion module, least squares module, and Open NI library; the connection relationship between them is: first, based on the Astra Pro 3D depth camera to collect video images, Yolov7 algorithm is used to realize pedestrian detection and recognition, and Deep-SORT based on Kalman filter is used to realize pedestrian trajectory tracking. Then, the imaging principle of AstraPro 3D depth camera is studied, the three-dimensional coordinates of pedestrians in the camera coordinate system are calculated, and according to the calculated three-dimensional coordinates, the three-dimensional trajectory of pedestrians is decoupled into horizontal and vertical two-dimensional trajectories by vertical projection method, and the horizontal and vertical two-dimensional trajectories of pedestrians are predicted by least squares method, and the horizontal and vertical predicted trajectories are synthesized into the three-dimensional trajectory of pedestrians, realizing real-time single-target pedestrian trajectory prediction and multi-target pedestrian trajectory prediction.

[0009] The Astro Pro 3D depth camera adopts the 3D depth camera of Orbbec (such as Figure 2 As shown in the figure, the appearance of AstroPro 3D depth camera is obtained based on 3D structured light imaging technology to obtain the depth image of the object, and can obtain color RGB, depth Depth, infrared IR image information and depth point cloud information. Astro Pro 3D depth camera can be used to record the video of pedestrians walking, providing data support for subsequent pedestrian detection and pedestrian trajectory prediction.

[0010] The Open NI library, nested in Python, is responsible for recording and playing back pedestrian walking videos collected by the Astro Pro 3D depth camera, providing a data source for subsequent pedestrian detection.

[0011] The improved Yolov7 algorithm module uses the GPU architecture model and takes SiLU as the activation function. The Yolov7 framework can be divided into three parts: input layer, backbone network layer, and output layer. Figure 3 As shown in the figure, the structure of Yolov7, the input layer preprocesses the input image, scales the image into a 640×640 RGB image, adds the adaptive anchor frame calculation function of the image, and finds the optimal anchor frame value in different training. The backbone network layer is mainly used to extract features and consists of CBS module, ELAN module and MP module. First, the CBS module is used to change the number of channels and extract features through 4 layers of convolution, and the feature map size is changed to 160×160×128. The head part is replaced with the Bot3 structure, which is a vision transformer structure. Through the global attention mechanism, the characteristics of pedestrian context are fully considered, which effectively improves the accuracy. Then, the ELAN module controls the shortest path with the longest gradient, so that the network model learns more picture features, enhances its robustness, and keeps the length and width of the image input and output unchanged. Then, the MP module downsamples. When the input and output channels are the same, the upper branch first halves the length and width through the MP algorithm, and then halves the channels through the CBS algorithm; while the lower branch halves the channels through a CBS algorithm, and then halves the length and width through a CBS algorithm. Finally, the upper and lower branches are merged to halve the length and width, double the channels, and extract features. The output layer is used to predict image features, image target categories and credibility. The RepVGG-Style algorithm is introduced to upgrade the output layer network structure. Auxiliary head training and corresponding positive and negative sample matching strategies are used, and modules such as BConv, MP, CAT, and RAP are integrated to improve the overall performance of the Yolov7 network output layer and the accuracy of prediction. The position of pedestrians in the recorded video at each moment can be calibrated to obtain the three-dimensional trajectory coordinates of pedestrians in the camera coordinate system, thus realizing the process of pedestrian detection. This algorithm replaces the Shufflenetv2 structure with all convolutional Conv layers, through shuffling and sparse connections between channels. The traditional complex convolution process is eliminated to achieve effective compression of the model, which is conducive to the deployment of the Yolov7 algorithm on resource-constrained embedded devices.

[0012] The Deep-SORT model is implemented based on an improved Kalman filter algorithm, which uses the target information of the previous frame to predict the tracking information of the corresponding target in the next frame. In the process of pedestrian trajectory tracking, if the object is occluded, the apparent characteristics of the target cannot be obtained, resulting in the phenomenon of target loss. The Kalman filter algorithm constructs a motion model of the target object through motion information, which can effectively alleviate the problems of missed detection and false detection caused by occlusion, and improve the accuracy of pedestrian trajectory tracking.

[0013] The least squares method is used in the present invention to predict pedestrian trajectories in real time. In order to achieve real-time prediction of pedestrian trajectories, it is necessary to construct Y c and X c The corresponding relationship between X c is the horizontal coordinate of the pedestrian in the camera coordinate system, Y c is the vertical coordinate of the pedestrian in the camera coordinate system. In order to further reduce the algorithm's requirements on computing power, the present invention adopts the motion decomposition method to study Y c and t and X c and t, and in Y c and X c The direction is fitted and predicted. The details are introduced in the method section below.

[0014] The pedestrian trajectory prediction method based on the fusion of Deep-SORT model and least squares method has the following specific steps:

[0015] Step 1: Pedestrian Recognition

[0016] Step S101: Using the Astro Pro 3D depth camera to record a video of a pedestrian walking;

[0017] Step S102: Play back the video recorded by the Astro Pro 3D depth camera using the Open NI library embedded in Python;

[0018] Step S103: designing the framework structure of the Yolov7 algorithm model, including its input layer, backbone network layer and output layer;

[0019] Step S104: Design the BoT3 network structure, replace 3×3 convolution with Multi-Head Self-Attention (MHSA), and treat the ResNet bottleneck block with self-attention as a Transformer module. Design a replacement solution for the head input layer of Yolov7 based on BoT3;

[0020] Step S105: In the source code of Yolov7, put the BoT3 module into the first layer of Yolov7. Perform a preliminary test and the new header can work normally;

[0021] Step S106: Use the Yolov7 algorithm program written in Python to complete pedestrian recognition on the video content and determine the target position of each frame in the video. The output of the detector includes the bounding box and class of the target;

[0022] Step S107: The new head is trained and tested on the MC-coco dataset. When the distance between the pedestrian and the camera is less than 10 meters, a recall rate greater than 95% can be achieved, and the efficiency is about 6 times higher than that of the baseline model.

[0023] Step S108: Based on the ShuffleNet algorithm, the amount of calculation is reduced through group convolution, and the channel rearrangement technology is used to solve the information isolation problem caused by group convolution, thereby greatly reducing the amount of calculation and parameter of the model while ensuring the accuracy of the model, making it more suitable for running on mobile terminals and embedded devices;

[0024] Step 2: Pedestrian trajectory tracking using the improved Kalman filter algorithm

[0025] Step S201: Target initialization. The first frame of the video or a newly appeared target has its detection box initialized as a new trajectory;

[0026] Step S202: Initialization of Kalman filter prediction

[0027] Step S202.1: Define the state vector

[0028] Define the state vector of the target, including the target's location information (such as the center coordinate c x ,c y , aspect ratio r, height h);

[0029] Step S202.2: Initialize the covariance matrix

[0030] Initialize the covariance matrix. The larger the number in the covariance matrix, the greater the uncertainty.

[0031] Step S203: Predicting the position of the current frame

[0032] Based on the state vector and covariance matrix of the previous frame, the Kalman filter is used to predict the position of the target in the current frame; Step S204: Feature extraction

[0033] Use a convolutional neural network (CNN) to extract the appearance features of the target in each detection box. These features are crucial for target re-identification (Re-ID) and help re-identify and associate the target after it is occluded or lost;

[0034] Step S205: Calculate the cost matrix

[0035] Based on the appearance information and Mahalanobis distance (or IOU), the cost matrix between the detection box in the current frame and the predicted trajectory is calculated. The elements in the cost matrix represent the similarity or distance between the detection box and the trajectory;

[0036] Step S206: Preparation for cascade matching

[0037] Divide the trajectories into confirmed and unconfirmed trajectories;

[0038] Step S207: Priority sorting

[0039] Prioritize the trajectories based on their confidence;

[0040] Step S208: Matching high confidence trajectories

[0041] Use the Hungarian algorithm to match high-confidence trajectories with the detection boxes in the current frame;

[0042] Step S209: Update and mismatch processing

[0043] For successfully matched trajectories, Kalman filtering is used to update them. For unmatched trajectories, their status is retained for subsequent matching;

[0044] Step S210: Filter unmatched trajectories

[0045] Filter out trajectories with fewer time updates from unmatched trajectories as candidates for IOU matching;

[0046] Step S211: Calculate IOU

[0047] Calculate the intersection-over-union (IOU) value between the candidate trajectory and the detection frame in the current frame. IOU is an indicator to measure the degree of overlap between two rectangular frames. The specific calculation method is the ratio of the intersection area to the union area of ​​the two rectangular frames. In this step, the system compares the predicted position of each candidate trajectory in the current frame with all detection frames and calculates the IOU value between them. The present invention takes into account the characteristics of faster pedestrian movement and reduces the IOU from 70% to 40%. By reducing the intersection-over-union area between the two frames, the matching difficulty is reduced;

[0048] Step S212: Matching decision

[0049] According to the preset IOU threshold, determine whether the track matches the detection box. The IOU threshold is set to a number between 0 and 1, such as 0.5. If the IOU value of a candidate track and a detection box is greater than or equal to this threshold, they are considered to be matched successfully; otherwise, they are considered to be unmatched;

[0050] Step S213: Update and delete

[0051] For successfully matched trajectories, Kalman filtering is used to update the state. For unmatched trajectories, they are checked to see if they meet the deletion conditions, such as if they have not been matched with the detection box for a long time. If they meet the deletion conditions, these trajectories are deleted from the trajectory set to avoid useless trajectories occupying resources.

[0052] Step S214: Initialize new trajectory

[0053] Initialize a new track for the unmatched detection box in the current frame. This means that for the new target object that appears in the current frame, create a new track for it for subsequent tracking;

[0054] Step S215: Update track status

[0055] Update the status of all tracks, including confirmed and unconfirmed tracks. Confirmed tracks are those that have been successfully matched with the detection box for multiple consecutive frames, while unconfirmed tracks are those that have just been initialized or have not been successfully matched for multiple consecutive frames. Update the status of the track accordingly according to its matching status;

[0056] Step S216: Integrate matching pairs

[0057] Step 1: Deep-SORT first uses IOU matching to preliminarily associate the detection box of the current frame with the prediction box of the previous frame. It calculates the IOU value between the detection box and the prediction box to construct a cost matrix, and then uses the Hungarian algorithm to find the matching solution with the lowest cost;

[0058] Step 2: For high-confidence trajectories, Deep-SORT performs cascade matching. Cascade matching combines appearance features and motion information to further improve matching accuracy;

[0059] Step S217: Update trajectory

[0060] The present invention uses Kalman filtering to update the state of the successfully matched trajectory again. This step is similar to the update operation in step S213. Through Kalman filtering, the state information such as the position and speed of the target object can be estimated more accurately, thereby improving the accuracy and stability of tracking;

[0061] Step S218: Processing unmatched trajectories

[0062] The present invention performs subsequent processing on unmatched trajectories according to their status (such as time update, confidence, etc.), such as retaining, deleting or lowering the priority;

[0063] Step S219: Initialize new trajectory

[0064] The present invention initializes a new trajectory for the unmatched detection frame in the current frame and sets its initial state;

[0065] Step S220: Track management

[0066] The present invention maintains the trajectory information of each target, including the trajectory ID, status, historical location, etc., and stores the trajectory information of each target in the head of its own queue through a Python list. When performing trajectory prediction, the first 10 elements of the queue are read to continuously track and manage the trajectory so as to match and update it in subsequent frames.

[0067] Step 3: Depth camera coordinate calculation

[0068] Step S301: Using the imaging principle of the Astro Pro 3D depth camera, the working principle of the Astro Pro 3D depth camera is analyzed (such as Figure 4 Astro Pro 3D depth camera working principle diagram).

[0069] The pixel coordinate system of the present invention takes O as the origin, and the x and y axes are as follows: Figure 4 As shown in the figure, o is the center of the photo, the horizontal axis is half of the maximum horizontal resolution, and the vertical axis is half of the maximum vertical resolution; oc is the horizontal offset of the coordinates of point P in the pixel coordinate system, cp is the vertical offset of the coordinates of point P in the pixel coordinate system, and p(x, y) is a point in the pixel coordinate system, reflecting the relative position of the pedestrian in the RGB image; in the world coordinate system, O c As the origin, the spatial rectangular coordinate system is used, which is divided into X c , Y c , Z c Three dimensions, P(X C ,Y C ,Z C ) is the corresponding point in the camera coordinate system, AB is the horizontal coordinate of the pedestrian center in the world coordinate system, AO is c is the vertical coordinate of the pedestrian in the world coordinate system; the pedestrian center is the instantaneous center O c The position of the pedestrian is used to evaluate the pedestrian's motion state; the method for calculating the three-dimensional coordinates of the pedestrian's trajectory in the depth camera coordinate system is as follows:

[0070] Step 1: △O C oC is similar to △O c AB, so we have Established;

[0071] Step 2: △O c Cp is similar to △O c BP, so there is Established;

[0072] Step 3: Combining the above two equations, we can get:

[0073]

[0074] Step 4: According to the horizontal maximum field of view of the depth camera, the field of view from the center point of the pixel coordinate system to the leftmost endpoint can be obtained. Let the horizontal maximum field of view of the depth camera be α, the field of view from the center point of the pixel to the leftmost endpoint be B, and solve the field of view from the center point of the pixel to the leftmost endpoint β, so:

[0075]

[0076] Step 5: According to the maximum horizontal pixel value, the pixel value from the center point of the pixel coordinate system to the leftmost end point can be obtained. The maximum horizontal pixel value is X. Let the pixel distance from the center point of the pixel coordinate system to the leftmost end point be X1. Solve the pixel distance X1 from the center point of the pixel coordinate system to the leftmost end, so we have:

[0077]

[0078] Step 6: Based on the pixel distance X1 from the center point of the pixel coordinate system to the leftmost endpoint and the field of view B from the center point o of the pixel coordinate system to the leftmost endpoint, solve the field of view angle ∠OO from any point p(x,y) to the center point o of the pixel coordinate system c C:

[0079]

[0080] Step 7: Calculate the field of view angle ∠oO from any point p(x,y) to the center point o of the pixel coordinate system c C and O c The length of A, using the tangent relationship, is:

[0081] AB=O c A·tan(∠OO c C) (5)

[0082] Step 8: According to the pedestrian coordinates (x, y) in the camera coordinate system, the camera's maximum field of view angle α, and the depth camera measurement value Z c , the maximum horizontal and vertical resolution, from steps 1-7, the pedestrian's horizontal coordinate X can be solved c The value of

[0083] Z Cis the depth value measured by the depth camera. According to the above steps (1)-(5), the X coordinate of any pedestrian in the camera coordinate system can be calculated. c ,Y c ,Z c Solve for the value of .

[0084] Step 4: Pedestrian trajectory prediction

[0085] Step S401: Find the main parameter values ​​of the Astro Pro depth camera (maximum horizontal field of view of the depth camera, maximum vertical field of view of the depth camera, maximum horizontal resolution of the depth camera, maximum horizontal resolution of the depth camera, etc.), and record them;

[0086] Step S402: Play a video of a pedestrian walking, cut it every 0.5 seconds, start from the 11th point, include this point, push forward 10 points, and use Yolov7 to mark the pedestrian's position respectively;

[0087] Step S403: using the pedestrian position marked by Yolov7, convert the pedestrian's two-dimensional coordinates in the pixel coordinate system into three-dimensional coordinates in the world coordinate system;

[0088] Step S404: Select the first five points of the 11 points in step S402 as the historical data of the pedestrian trajectory for fitting the polynomial. These five points represent the pedestrian's past walking trajectory.

[0089] Step S405: Use a first-order or second-order polynomial to perform fitting. These two polynomials represent a straight line and a parabola respectively, and can preliminarily determine the walking trajectory trend of the pedestrian.

[0090] Step S406: Linear polynomial fitting

[0091] The least squares method is used to fit the horizontal and vertical coordinates of the first five pedestrian trajectories to the system running time t. The slope and intercept of the fitted line are calculated to obtain the equation of the fitted line. Based on the existing data and relationship between the independent variable (X) and the dependent variable (Y), the univariate linear regression equation is preliminarily set as:

[0092] Y=a+bX (6)

[0093] Where a is the intercept and b is the slope.

[0094] Use the least squares method to solve the regression coefficients a and b. The goal of the least squares method is to find a straight line that minimizes the vertical distance from all sample points to the straight line (i.e., the sum of squared errors). By derivation, we can get the solution formula for a and b:

[0095]

[0096]

[0097] Where X i and Y i are the observed values ​​of the horizontal and vertical coordinates of the world coordinate system, and are the means of the independent and dependent variables, respectively.

[0098] Step S407: quadratic polynomial fitting

[0099] Similarly, the least squares method is used to fit the quadratic polynomial (i.e., parabola) to the horizontal and vertical coordinates of the first five pedestrian trajectories and the system running time t, and the coefficients of the fitting parabola are calculated to obtain the equation of the fitting parabola.

[0100] Specify a quadratic equation as the fitted model:

[0101] Y = dx 2 +ex+f (9)

[0102] Among them, d, e and f are the coefficients that need to be solved, Y is the dependent variable, which is the horizontal or vertical coordinate of the pedestrian trajectory, and x is the independent variable, that is, time t.

[0103] The loss function s (also called the residual sum of squares) is defined as the sum of the squares of the vertical distances of all data points to the fitted parabola, and the formula is as follows:

[0104]

[0105] in is the model prediction value of the i-th fitted point, Y i is the true value of the i-th fitted point, and n is the total number of samples.

[0106] In order to find the coefficients d, e, and f that minimize the loss function S, we take the partial derivatives of S with respect to d, e, and f, and set them equal to zero:

[0107]

[0108]

[0109]

[0110] Then use the numpy.solve function to solve this equation and get the values ​​of coefficients d, e, and f.

[0111] Step S408: Calculate prediction error

[0112] Using the 6th to 10th points (i.e. the pedestrian's next walking trajectory) as the prediction basis, the prediction errors of the two fitting polynomials at these points are calculated respectively. The prediction error can be measured by calculating the difference between the predicted value and the true value;

[0113] Step S409: The present invention compares the prediction error values ​​of the two fitting polynomials at points 6-10, and selects the polynomial with the smaller prediction error as the optimal fitting polynomial;

[0114] Step S410: Apply the best fitting polynomial

[0115] Apply the selected best fit polynomial to the 11th point to perform trajectory prediction. Calculate the predicted value of the 11th point;

[0116] Step S411: superimpose multiple single-target real-time pedestrian trajectory predictions to form a real-time multi-target pedestrian trajectory prediction.

[0117] Beneficial effects of the present invention:

[0118] 1. The present invention proposes a pedestrian trajectory prediction method based on Deep-SORT and least squares method, which improves the efficiency of pedestrian detection by fusing ShuffleNetV2 with Vit (Vision Transformer). Finally, by selecting a polynomial with a smaller prediction error, the problem of overfitting or underfitting is avoided, making the fitting model more generalizable on unknown data and realizing adaptive processing of data. This is particularly important for application scenarios such as self-driving cars that require real-time processing of large amounts of data and accurate predictions, realizing the detection of pedestrian turning intentions and fully considering the characteristics of pedestrian movement.

[0119] 2. Compared with models based on LSTM, BPNN, etc., the present invention is based on an adaptive least squares method, which eliminates the tedious processing of trajectories and can reduce unnecessary calculations, thereby reducing the computing resource consumption of autonomous vehicles. This is of great significance for achieving lightweight autonomous vehicles, improving energy efficiency and extending driving range. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] Figure 1 : A schematic diagram of the relationship between the pedestrian trajectory prediction algorithm of Yolov7 and Deep-SORT;

[0121] Figure 2 : Appearance of Astro Pro 3D depth camera;

[0122] Figure 3 : Structure diagram of Yolov7;

[0123] Figure 4: Astro Pro 3D depth camera working principle diagram; DETAILED DESCRIPTION

[0124] The implementation of the present invention mainly includes the following four aspects:

[0125] 1. Pedestrian detection and model compression. Pedestrian detection is achieved using the improved Yolov7 algorithm;

[0126] 2. Pedestrian trajectory tracking. Pedestrian trajectory tracking is achieved using the Deep-SORT model based on the Kalman filter algorithm;

[0127] 3. Depth camera coordinate calculation. Study the imaging principle of the Astra Pro 3D depth camera and convert the three-dimensional coordinates of pedestrians in the pixel coordinate system into the three-dimensional coordinates of pedestrians in the camera coordinate system;

[0128] 4. Pedestrian trajectory prediction. The three-dimensional trajectory of pedestrians is decoupled into horizontal and vertical two-dimensional trajectories by vertical projection, and the pedestrian's horizontal and vertical two-dimensional trajectories are predicted using the least squares method. The horizontal and vertical predicted trajectories are synthesized into the pedestrian's three-dimensional trajectory, realizing real-time single-target pedestrian trajectory prediction and multi-target pedestrian trajectory prediction;

[0129] The present invention will be further described below in conjunction with the accompanying drawings.

[0130] like Figure 1 As shown, the purpose of the present invention is to design a pedestrian trajectory prediction algorithm based on the fusion of Yolov7 and Deep-SORT, so that pedestrian detection is more accurate, pedestrian trajectory prediction is more precise, and the probability of traffic accidents between cars and pedestrians is reduced to a greater extent.

[0131] The technical solution of the present invention:

[0132] The present invention provides a pedestrian trajectory prediction system based on Deep-SORT and least squares method (such as Figure 1As shown in the figure, a schematic diagram of the pedestrian trajectory prediction algorithm fused with Yolov7 and Deep-SORT) includes Astra Pro 3D depth camera, Yolov7 algorithm module, Kalman filter algorithm module, coordinate conversion module, least squares module, and Open NI library; the connection relationship between them is: first, based on the Astra Pro 3D depth camera to collect video images, Yolov7 algorithm is used to realize pedestrian detection and recognition, and Deep-SORT based on Kalman filter is used to realize pedestrian trajectory tracking. Then, the imaging principle of AstraPro 3D depth camera is studied, the three-dimensional coordinates of pedestrians in the camera coordinate system are calculated, and according to the calculated three-dimensional coordinates, the three-dimensional trajectory of pedestrians is decoupled into horizontal and vertical two-dimensional trajectories by vertical projection method, and the horizontal and vertical two-dimensional trajectories of pedestrians are predicted by least squares method, and the horizontal and vertical predicted trajectories are synthesized into the three-dimensional trajectory of pedestrians, realizing real-time single-target pedestrian trajectory prediction and multi-target pedestrian trajectory prediction.

[0133] The Astro Pro 3D depth camera adopts the 3D depth camera of Orbbec (such as Figure 2 As shown in the figure, the appearance of AstroPro 3D depth camera is obtained based on 3D structured light imaging technology to obtain the depth image of the object, and can obtain color RGB, depth Depth, infrared IR image information and depth point cloud information. Astro Pro 3D depth camera can be used to record the video of pedestrians walking, providing data support for subsequent pedestrian detection and pedestrian trajectory prediction.

[0134] The Open NI library, nested in Python, is responsible for recording and playing back pedestrian walking videos collected by the Astro Pro 3D depth camera, providing a data source for subsequent pedestrian detection.

[0135] The improved Yolov7 algorithm module uses the GPU architecture model and takes SiLU as the activation function. The Yolov7 framework can be divided into three parts: input layer, backbone network layer, and output layer. Figure 3As shown in the figure, the structure of Yolov7, the input layer preprocesses the input image, scales the image into a 640×640 RGB image, adds the adaptive anchor frame calculation function of the image, and finds the optimal anchor frame value in different training. The backbone network layer is mainly used to extract features and consists of CBS module, ELAN module and MP module. First, the CBS module is used to change the number of channels and extract features through 4 layers of convolution, and the feature map size is changed to 160×160×128. The head part is replaced with the Bot3 structure, which is a vision transformer structure. Through the global attention mechanism, the characteristics of pedestrian context are fully considered, which effectively improves the accuracy. Then, the ELAN module controls the shortest path with the longest gradient, so that the network model learns more picture features, enhances its robustness, and keeps the length and width of the image input and output unchanged. Then, the MP module downsamples. When the input and output channels are the same, the upper branch first halves the length and width through the MP algorithm, and then halves the channels through the CBS algorithm; while the lower branch halves the channels through a CBS algorithm, and then halves the length and width through a CBS algorithm. Finally, the upper and lower branches are merged to halve the length and width, double the channels, and extract features. The output layer is used to predict image features, image target categories and credibility. The RepVGG-Style algorithm is introduced to upgrade the output layer network structure. Auxiliary head training and corresponding positive and negative sample matching strategies are used, and modules such as BConv, MP, CAT, and RAP are integrated to improve the overall performance of the Yolov7 network output layer and the accuracy of prediction. The position of pedestrians in the recorded video at each moment can be calibrated to obtain the three-dimensional trajectory coordinates of pedestrians in the camera coordinate system, thus realizing the process of pedestrian detection. This algorithm replaces the Shufflenetv2 structure with all convolutional Conv layers, through shuffling and sparse connections between channels. The traditional complex convolution process is eliminated to achieve effective compression of the model, which is conducive to the deployment of the Yolov7 algorithm on resource-constrained embedded devices.

[0136] The Deep-SORT model is implemented based on an improved Kalman filter algorithm, which uses the target information of the previous frame to predict the tracking information of the corresponding target in the next frame. In the process of pedestrian trajectory tracking, if the object is occluded, the apparent characteristics of the target cannot be obtained, resulting in the phenomenon of target loss. The Kalman filter algorithm constructs a motion model of the target object through motion information, which can effectively alleviate the problems of missed detection and false detection caused by occlusion, and improve the accuracy of pedestrian trajectory tracking.

[0137] The least squares method is used in the present invention to predict pedestrian trajectories in real time. In order to achieve real-time prediction of pedestrian trajectories, it is necessary to construct Y c and X cThe corresponding relationship between X c is the horizontal coordinate of the pedestrian in the camera coordinate system, Y c is the vertical coordinate of the pedestrian in the camera coordinate system. In order to further reduce the algorithm's requirements on computing power, the present invention adopts the motion decomposition method to study Y c and t and X c and t, and in Y c and X c The direction is fitted and predicted. The details are introduced in the method section below.

[0138] The pedestrian trajectory prediction method based on the fusion of Deep-SORT model and least squares method has the following specific steps:

[0139] Step 1: Pedestrian Recognition

[0140] Step S101: Using the Astro Pro 3D depth camera to record a video of a pedestrian walking;

[0141] Step S102: Play back the video recorded by the Astro Pro 3D depth camera using the Open NI library embedded in Python;

[0142] Step S103: designing the framework structure of the Yolov7 algorithm model, including its input layer, backbone network layer and output layer;

[0143] Step S104: Design the BoT3 network structure, replace 3×3 convolution with Multi-Head Self-Attention (MHSA), and treat the ResNet bottleneck block with self-attention as a Transformer module. Design a replacement solution for the head input layer of Yolov7 based on BoT3;

[0144] Step S105: In the source code of Yolov7, put the BoT3 module into the first layer of Yolov7. Perform a preliminary test and the new header can work normally;

[0145] Step S106: Use the Yolov7 algorithm program written in Python to complete pedestrian recognition on the video content and determine the target position of each frame in the video. The output of the detector includes the bounding box and class of the target;

[0146] Step S107: The new head is trained and tested on the MC-coco dataset. When the distance between the pedestrian and the camera is less than 10 meters, a recall rate greater than 95% can be achieved, and the efficiency is about 6 times higher than that of the baseline model.

[0147] Step S108: Based on the ShuffleNet algorithm, the amount of calculation is reduced through group convolution, and the channel rearrangement technology is used to solve the information isolation problem caused by group convolution, thereby greatly reducing the amount of calculation and parameter of the model while ensuring the accuracy of the model, making it more suitable for running on mobile terminals and embedded devices;

[0148] Step 2: Pedestrian trajectory tracking using the improved Kalman filter algorithm

[0149] Step S201: Target initialization. The first frame of the video or a newly appeared target has its detection box initialized as a new trajectory;

[0150] Step S202: Initialization of Kalman filter prediction

[0151] Step S202.1: Define the state vector

[0152] Define the state vector of the target, including the target's location information (such as the center coordinate c x ,c y , aspect ratio r, height h);

[0153] Step S202.2: Initialize the covariance matrix

[0154] Initialize the covariance matrix. The larger the number in the covariance matrix, the greater the uncertainty.

[0155] Step S203: Predicting the position of the current frame

[0156] Based on the state vector and covariance matrix of the previous frame, the Kalman filter is used to predict the position of the target in the current frame; Step S204: Feature extraction

[0157] Use a convolutional neural network (CNN) to extract the appearance features of the target in each detection box. These features are crucial for target re-identification (Re-ID) and help re-identify and associate the target after it is occluded or lost;

[0158] Step S205: Calculate the cost matrix

[0159] Based on the appearance information and Mahalanobis distance (or IOU), the cost matrix between the detection box in the current frame and the predicted trajectory is calculated. The elements in the cost matrix represent the similarity or distance between the detection box and the trajectory;

[0160] Step S206: Preparation for cascade matching

[0161] Divide the trajectories into confirmed and unconfirmed trajectories;

[0162] Step S207: Priority sorting

[0163] Prioritize the trajectories based on their confidence;

[0164] Step S208: Matching high confidence trajectories

[0165] Use the Hungarian algorithm to match high-confidence trajectories with the detection boxes in the current frame;

[0166] Step S209: Update and mismatch processing

[0167] For successfully matched trajectories, Kalman filtering is used to update them. For unmatched trajectories, their status is retained for subsequent matching;

[0168] Step S210: Filter unmatched trajectories

[0169] Filter out trajectories with fewer time updates from unmatched trajectories as candidates for IOU matching;

[0170] Step S211: Calculate IOU

[0171] Calculate the intersection-over-union (IOU) value between the candidate trajectory and the detection frame in the current frame. IOU is an indicator to measure the degree of overlap between two rectangular frames. The specific calculation method is the ratio of the intersection area to the union area of ​​the two rectangular frames. In this step, the system compares the predicted position of each candidate trajectory in the current frame with all detection frames and calculates the IOU value between them. The present invention takes into account the characteristics of faster pedestrian movement and reduces the IOU from 70% to 40%. By reducing the intersection-over-union area between the two frames, the matching difficulty is reduced;

[0172] Step S212: Matching decision

[0173] According to the preset IOU threshold, determine whether the track matches the detection box. The IOU threshold is set to a number between 0 and 1, such as 0.5. If the IOU value of a candidate track and a detection box is greater than or equal to this threshold, they are considered to be matched successfully; otherwise, they are considered to be unmatched;

[0174] Step S213: Update and delete

[0175] For successfully matched trajectories, Kalman filtering is used to update the state. For unmatched trajectories, they are checked to see if they meet the deletion conditions, such as if they have not been matched with the detection box for a long time. If they meet the deletion conditions, these trajectories are deleted from the trajectory set to avoid useless trajectories occupying resources.

[0176] Step S214: Initialize new trajectory

[0177] Initialize a new track for the unmatched detection box in the current frame. This means that for the new target object that appears in the current frame, create a new track for it for subsequent tracking;

[0178] Step S215: Update track status

[0179] Update the status of all tracks, including confirmed and unconfirmed tracks. Confirmed tracks are those that have been successfully matched with the detection box for multiple consecutive frames, while unconfirmed tracks are those that have just been initialized or have not been successfully matched for multiple consecutive frames. Update the status of the track accordingly according to its matching status;

[0180] Step S216: Integrate matching pairs

[0181] Step 1: Deep-SORT first uses IOU matching to preliminarily associate the detection box of the current frame with the prediction box of the previous frame. It calculates the IOU value between the detection box and the prediction box to construct a cost matrix, and then uses the Hungarian algorithm to find the matching solution with the lowest cost;

[0182] Step 2: For high-confidence trajectories, Deep-SORT performs cascade matching. Cascade matching combines appearance features and motion information to further improve matching accuracy;

[0183] Step S217: Update trajectory

[0184] The present invention uses Kalman filtering to update the state of the successfully matched trajectory again. This step is similar to the update operation in step S213. Through Kalman filtering, the state information such as the position and speed of the target object can be estimated more accurately, thereby improving the accuracy and stability of tracking;

[0185] Step S218: Processing unmatched trajectories

[0186] The present invention performs subsequent processing on unmatched trajectories according to their status (such as time update, confidence, etc.), such as retaining, deleting or lowering the priority;

[0187] Step S219: Initialize new trajectory

[0188] The present invention initializes a new trajectory for the unmatched detection frame in the current frame and sets its initial state;

[0189] Step S220: Track management

[0190] The present invention maintains the trajectory information of each target, including the trajectory ID, status, historical location, etc., and stores the trajectory information of each target in the head of its own queue through a Python list. When performing trajectory prediction, the first 10 elements of the queue are read to continuously track and manage the trajectory so as to match and update it in subsequent frames.

[0191] Step 3: Depth camera coordinate calculation

[0192] Step S301: Using the imaging principle of the Astro Pro 3D depth camera, the working principle of the Astro Pro 3D depth camera is analyzed (such as Figure 4 Astro Pro 3D depth camera working principle diagram).

[0193] The pixel coordinate system of the present invention takes O as the origin, and the x and y axes are as follows: Figure 4 As shown in the figure, o is the center of the photo, the horizontal axis is half of the maximum horizontal resolution, and the vertical axis is half of the maximum vertical resolution; oc is the horizontal offset of the coordinates of point P in the pixel coordinate system, cp is the vertical offset of the coordinates of point P in the pixel coordinate system, and p(x, y) is a point in the pixel coordinate system, reflecting the relative position of the pedestrian in the RGB image; in the world coordinate system, O c As the origin, the spatial rectangular coordinate system is used, which is divided into X c , Y c , Z c Three dimensions, P(X C ,Y C ,Z C ) is the corresponding point in the camera coordinate system, AB is the horizontal coordinate of the pedestrian center in the world coordinate system, AO is c is the vertical coordinate of the pedestrian in the world coordinate system; the pedestrian center is the instantaneous center O c The position of the pedestrian is used to evaluate the pedestrian's motion state; the method for calculating the three-dimensional coordinates of the pedestrian's trajectory in the depth camera coordinate system is as follows:

[0194] Step 1: △O C oC is similar to △O c AB, so we have Established;

[0195] Step 2: △O c Cp is similar to △O c BP, so there is Established;

[0196] Step 3: Combining the above two equations, we can get:

[0197]

[0198] Step 4: According to the horizontal maximum field of view of the depth camera, the field of view from the center point of the pixel coordinate system to the leftmost endpoint can be obtained. Let the horizontal maximum field of view of the depth camera be α, the field of view from the center point of the pixel to the leftmost endpoint be B, and solve the field of view from the center point of the pixel to the leftmost endpoint β, so:

[0199]

[0200] Step 5: According to the maximum horizontal pixel value, the pixel value from the center point of the pixel coordinate system to the leftmost end point can be obtained. The maximum horizontal pixel value is X. Let the pixel distance from the center point of the pixel coordinate system to the leftmost end point be X1. Solve the pixel distance X1 from the center point of the pixel coordinate system to the leftmost end, so we have:

[0201]

[0202] Step 6: Based on the pixel distance X1 from the center point of the pixel coordinate system to the leftmost endpoint and the field of view B from the center point o of the pixel coordinate system to the leftmost endpoint, solve the field of view angle ∠OO from any point p(x,y) to the center point o of the pixel coordinate system c C:

[0203]

[0204] Step 7: Calculate the field of view angle ∠oO from any point p(x,y) to the center point o of the pixel coordinate system c C and O c The length of A, using the tangent relationship, is:

[0205] AB=O c A·tan(∠OO c C) (5)

[0206] Step 8: According to the pedestrian coordinates (x, y) in the camera coordinate system, the camera's maximum field of view angle α, and the depth camera measurement value Z c , the maximum horizontal and vertical resolution, from steps 1-7, the pedestrian's horizontal coordinate X can be solved c The value of

[0207] Z C is the depth value measured by the depth camera. According to the above steps (1)-(5), the X coordinate of any pedestrian in the camera coordinate system can be calculated. c ,Y c ,Z c Solve for the value of .

[0208] Step 4: Pedestrian trajectory prediction

[0209] Step S401: Find the main parameter values ​​of the Astro Pro depth camera (maximum horizontal field of view of the depth camera, maximum vertical field of view of the depth camera, maximum horizontal resolution of the depth camera, maximum horizontal resolution of the depth camera, etc.), and record them;

[0210] Step S402: Play a video of a pedestrian walking, cut it every 0.5 seconds, start from the 11th point, include this point, push forward 10 points, and use Yolov7 to mark the pedestrian's position respectively;

[0211] Step S403: using the pedestrian position marked by Yolov7, convert the pedestrian's two-dimensional coordinates in the pixel coordinate system into three-dimensional coordinates in the world coordinate system;

[0212] Step S404: Select the first five points of the 11 points in step S402 as the historical data of the pedestrian trajectory for fitting the polynomial. These five points represent the pedestrian's past walking trajectory.

[0213] Step S405: Use a first-order or second-order polynomial to perform fitting. These two polynomials represent a straight line and a parabola respectively, and can preliminarily determine the walking trajectory trend of the pedestrian.

[0214] Step S406: Linear polynomial fitting

[0215] The least squares method is used to fit the horizontal and vertical coordinates of the first five pedestrian trajectories to the system running time t. The slope and intercept of the fitted line are calculated to obtain the equation of the fitted line. Based on the existing data and relationship between the independent variable (X) and the dependent variable (Y), the univariate linear regression equation is preliminarily set as:

[0216] Y=a+bX (6)

[0217] Where a is the intercept and b is the slope.

[0218] Use the least squares method to solve the regression coefficients a and b. The goal of the least squares method is to find a straight line that minimizes the vertical distance from all sample points to the straight line (i.e., the sum of squared errors). By derivation, we can get the solution formula for a and b:

[0219]

[0220]

[0221] Where X i and Y i are the observed values ​​of the horizontal and vertical coordinates of the world coordinate system, and are the means of the independent and dependent variables, respectively.

[0222] Step S407: quadratic polynomial fitting

[0223] Similarly, the least squares method is used to fit the quadratic polynomial (i.e., parabola) to the horizontal and vertical coordinates of the first five pedestrian trajectories and the system running time t, and the coefficients of the fitting parabola are calculated to obtain the equation of the fitting parabola.

[0224] Specify a quadratic equation as the fitted model:

[0225] Y = dx 2 +ex+f (9)

[0226] Among them, d, e and f are the coefficients that need to be solved, Y is the dependent variable, which is the horizontal or vertical coordinate of the pedestrian trajectory, and x is the independent variable, that is, time t.

[0227] The loss function s (also called the residual sum of squares) is defined as the sum of the squares of the vertical distances of all data points to the fitted parabola, and the formula is as follows:

[0228]

[0229] in is the model prediction value of the i-th fitted point, which is Y i The true value of the i-th fitted point.

[0230]

[0231]

[0232]

[0233] Then use the numpy.solve function to solve this equation and get the values ​​of coefficients d, e, and f.

[0234] Step S408: Calculate prediction error

[0235] Using the 6th to 10th points (i.e. the pedestrian's next walking trajectory) as the prediction basis, the prediction errors of the two fitting polynomials at these points are calculated respectively. The prediction error can be measured by calculating the difference between the predicted value and the true value;

[0236] Step S409: The present invention compares the prediction error values ​​of the two fitting polynomials at points 6-10, and selects the polynomial with the smaller prediction error as the optimal fitting polynomial;

[0237] Step S410: Apply the best fitting polynomial

[0238] Apply the selected best fit polynomial to the 11th point to perform trajectory prediction. Calculate the predicted value of the 11th point;

[0239] Step S411: superimpose multiple single-target real-time pedestrian trajectory predictions to form a real-time multi-target pedestrian trajectory prediction.

[0240] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A pedestrian trajectory prediction method based on Deep-SORT and least squares method, characterized in that: The steps include: Step 1: Pedestrian recognition; include: Step S101: Recording a video of a pedestrian walking using a 3D depth camera; Step S102: using the Open NI library embedded in Python to play back the video recorded by the 3D depth camera; Step S103: designing the framework structure of the improved Yolov7 algorithm model, including its input layer, backbone network layer and output layer; Step S104: Design the BoT3 network structure, replace 3×3 convolution with Multi-Head Self-Attention (MHSA), and regard the ResNet bottleneck block with self-attention as a Transformer module; Step S105: Place the BoT3 module into the first layer of Yolov7 and conduct preliminary tests to ensure that the new head can work properly; Step S106: Use the Yolov7 algorithm model to complete pedestrian recognition on the video content and determine the target position of each frame in the video. The output of the detector includes the bounding box and class of the target. Step S107: training and testing the new head on the MC-coco dataset; Step S108: Based on the ShuffleNet algorithm, group convolution is used to reduce the amount of calculation, and channel rearrangement is used to solve the information isolation caused by group convolution. Under the premise of ensuring the accuracy of the model, the amount of calculation and the number of parameters of the model are reduced, making it more suitable for running on mobile terminals and embedded devices; Step 2: Pedestrian trajectory tracking using the improved Kalman filter algorithm Step S201: Target initialization: For the first frame of the video or a newly appearing target, its detection frame is initialized as a new trajectory; Step S202: Kalman filter algorithm initialization Step S203: Position prediction of the current frame Based on the state vector and covariance matrix of the previous frame, the Kalman filter is used to predict the position of the target in the current frame; Step S204: Feature extraction Use convolutional neural network to extract the appearance features of the target in each detection box; Step S205: Calculate the cost matrix Based on the appearance information and Mahalanobis distance (or IOU), the cost matrix between the detection box in the current frame and the predicted trajectory is calculated. The elements in the cost matrix represent the similarity or distance between the detection box and the trajectory. Step S206: Preparation for cascade matching Divide the trajectories into confirmed and unconfirmed trajectories; Step S207: Priority sorting Prioritize the trajectories based on their confidence; Step S208: Matching high confidence trajectories Use the Hungarian algorithm to match high-confidence trajectories with the detection boxes in the current frame; Step S209: Update and mismatch processing For successfully matched trajectories, Kalman filtering is used to update them; for unmatched trajectories, their status is retained for subsequent matching; Step S210: Filter unmatched trajectories Filter out trajectories with fewer time updates from unmatched trajectories as candidates for IOU matching; Step S211: Calculate IOU Calculate the intersection-over-union (IOU) value between the candidate trajectory and the detection box in the current frame. In this step, the predicted position of each candidate trajectory in the current frame is compared with all the detection boxes to calculate the IOU value between them. Step S212: Matching decision According to the preset IOU threshold, determine whether the trajectory matches the detection box. If the IOU value of a candidate trajectory and a detection box is greater than or equal to this threshold, they are considered to be matched successfully; otherwise, they are considered to be unmatched; Step S213: Update and delete For successfully matched trajectories, Kalman filtering is used to update the status; for unmatched trajectories, they are checked to see if they meet the deletion conditions. For example, if they do not match the detection box for a long time, if they meet the deletion conditions, these trajectories are deleted from the trajectory set. Step S214: Initialize new trajectory Initialize a new trajectory for the unmatched detection box in the current frame, and create a new trajectory for the new target object that appears in the current frame for subsequent tracking; Step S215: Update track status Update the status of all trajectories, including confirmed and unconfirmed trajectories; confirmed trajectories are those that have been successfully matched with the detection box for multiple consecutive frames, while unconfirmed trajectories are those that have just been initialized or have not been successfully matched for multiple consecutive frames. According to the matching status of the trajectory, its status is updated accordingly; Step S216: Integrate matching pairs Integrate the matching results to obtain all matching pairs; Step S217: Update trajectory For the successfully matched trajectory, the Kalman filter is used again to update the state; this step is the same as the update operation in step S213. Through the Kalman filter, the state information such as the position and speed of the target object can be estimated more accurately, thereby improving the accuracy and stability of tracking; Step S218: Processing unmatched trajectories The present invention performs subsequent processing on the unmatched tracks according to their status, including retaining, deleting or lowering the priority; Step S219: Initialize new trajectory For the unmatched detection boxes in the current frame, initialize a new trajectory and set its initial state; Step S220: Track management By designing a Python list, the trajectory information of each target is stored in the head of its own queue. When performing trajectory prediction, the first 10 elements of the queue are read to continuously track and manage the trajectory so that it can be matched and updated in subsequent frames. Step 3: Depth camera coordinate calculation; including: Step S301: Let p(x,y) be a point on the pixel coordinate system, P(x C ,Y C ,Z C ) is the corresponding point in the camera coordinate system, and the three-dimensional coordinates X of the pedestrian trajectory in the depth camera coordinate system are calculated. c ,Y c ,Z c ; Step 4: Pedestrian trajectory prediction; including: Step S401: Find the main parameter values ​​of the depth camera, including the maximum horizontal field of view of the depth camera, the maximum vertical field of view of the depth camera, the maximum horizontal resolution of the depth camera, the maximum horizontal resolution of the depth camera, etc., and record them; Step S402: Play the video of pedestrians walking, cut it every 0.5 seconds, start from the 11th point, include this point, push forward 10 points, and use Yolov7 algorithm to mark the positions of pedestrians respectively; Step S403: using the Yolov7 algorithm to mark the pedestrian position, convert the pedestrian's two-dimensional coordinates in the pixel coordinate system into three-dimensional coordinates in the world coordinate system; Step S404: selecting the first five points of the 11 points as the historical data of the pedestrian trajectory for fitting the polynomial, and these five points represent the pedestrian's past walking trajectory; Step S405: fitting using polynomials, where the two polynomials represent a straight line and a parabola respectively, and can preliminarily determine the walking trajectory trend of the pedestrian; Step S408: Calculate prediction error Using the 6th to 10th points as prediction basis, calculate the prediction errors of the fitting polynomial at these points respectively; Step S409: Selecting a polynomial with a smaller prediction error as the optimal fitting polynomial; Step S410: applying the selected optimal fitting polynomial to the 11th point, performing trajectory prediction, and calculating the predicted value of the 11th point; Step S411: superimpose multiple single-target real-time pedestrian trajectory predictions to form a real-time multi-target pedestrian trajectory prediction.

2. The pedestrian trajectory prediction method based on Deep-SORT and least squares method according to claim 1, characterized in that: The step S301 calculates the three-dimensional coordinates X of the pedestrian trajectory in the depth camera coordinate system. c ,Y c ,Z c The specific method is as follows: Establish the coordinate system: pixel coordinate system, establish the x and y axes with O as the origin, where oc is the horizontal offset of the coordinate of point P in the pixel coordinate system, cp is the vertical offset of the coordinate of point P in the pixel coordinate system, and p(x,y) is a point in the pixel coordinate system, reflecting the relative position of the pedestrian in the RGB image; world coordinate system, with O as the origin c As the origin, the spatial rectangular coordinate system is used, which is divided into X c , Y c , Z c Three dimensions, P(X C ,Y C ,Z C ) is the corresponding point in the camera coordinate system, AB is the horizontal coordinate of the pedestrian center in the world coordinate system, AO is c O is the vertical coordinate of the pedestrian in the world coordinate system; c C is the horizontal distance between the camera focus and the pedestrian in the image coordinate system, O c B is the horizontal distance between the camera focus and the pedestrian, O c P is the distance from the focus to the pedestrian in the direct spatial coordinate system, and the pedestrian center is the instantaneous center O c The position of is used to evaluate the pedestrian's motion state, and then the following steps are calculated: Step 1: △O C oC is similar to △O c AB, so we have Established; Step 2: △O c CP is similar to △O c BP, so there is Established; Step 3: Combining the above two equations, we can get: Step 4: According to the maximum horizontal field of view of the depth camera, the field of view angle from the center point of the pixel coordinate system to the leftmost endpoint is obtained. Let the maximum horizontal field of view of the depth camera be α, the field of view angle from the center point of the pixel to the leftmost endpoint be B, and solve the field of view angle β from the center point of the pixel to the leftmost endpoint, so: Step 5: According to the maximum horizontal pixel value, the pixel value from the center point of the pixel coordinate system to the leftmost end point can be obtained. The maximum horizontal pixel value is X. Let the pixel distance from the center point of the pixel coordinate system to the leftmost end point be X1. Solve the pixel distance X1 from the center point of the pixel coordinate system to the leftmost end, so we have: Step 6: Based on the pixel distance X1 from the center point of the pixel coordinate system to the leftmost endpoint and the field of view B from the center point o of the pixel coordinate system to the leftmost endpoint, solve the field of view angle ∠OO from any point p(x,y) to the center point o of the pixel coordinate system c C: Step 7: Calculate the field of view angle ∠oO from any point p(x,y) to the center point o of the pixel coordinate system c C and O c The length of A, using the tangent relationship, is: AB=O c A·tan(∠OO c C) (5) Step 8: According to the pedestrian coordinates (x, y) in the camera coordinate system, the camera's maximum field of view angle α, and the depth camera measurement value Z c , the maximum horizontal and vertical resolution, from steps 1-7, the pedestrian's horizontal coordinate X can be solved c The value of Z C The depth value measured by the depth camera; According to the above steps, the X coordinate of any pedestrian in the camera coordinate system is c ,Y c ,Z c Solve for the value of .

3. The pedestrian trajectory prediction method based on Deep-SORT and least squares method according to claim 1, characterized in that: The polynomial fitting of step S405 includes: Step S406: Linear polynomial fitting The least squares method is used to fit the horizontal and vertical coordinates of the first five pedestrian trajectories to the system running time t. The slope and intercept of the fitting line are calculated to obtain the equation of the fitting line. According to the data relationship between the independent variable X and the dependent variable Y, the univariate linear regression equation is preliminarily set as: Y=a+bX (6) Where a is the intercept and b is the slope; Use the least squares method to solve the regression coefficients a and b. The solution formulas for a and b are as follows: Where X i and Y i are the observed values ​​of the independent and dependent variables, respectively. and are the means of the independent and dependent variables, respectively; Step S407: quadratic polynomial fitting The least squares method is used to fit the quadratic polynomial to the system running time t for the horizontal and vertical coordinates of the first five pedestrian trajectories, and the coefficients of the fitting parabola are calculated to obtain the equation of the fitting parabola; specifically: Specify a quadratic equation as the fitted model: Y=dx 2 +ex+f (9) Among them, d, e and f are the coefficients that need to be solved. The loss function s (also called the residual sum of squares) is defined as the sum of the squares of the vertical distances of all data points to the fitted parabola, and the formula is as follows: in is the model prediction value of the i-th fitted point, Yi is the true value of the i-th fitted point, and n is the total sample size. In order to find the coefficients d, e, and f that minimize the loss function S, we take the partial derivatives of S with respect to d, e, and f, and set them equal to zero. Then use the numpy.solve function to solve this equation and get the values ​​of coefficients d, e, and f.

4. The pedestrian trajectory prediction method based on Deep-SORT and least squares method according to claim 1, characterized in that: The initialization of step S202 includes: Define the state vector: define the location information of the target, including the center coordinates (c x ,c y ), aspect ratio r, height h.

5. The pedestrian trajectory prediction method based on Deep-SORT and least squares method according to claim 1, characterized in that: The IOU calculation method in step S211 is the ratio of the intersection area of ​​two rectangular frames to the union area.

6. The pedestrian trajectory prediction method based on Deep-SORT and least squares method according to claim 1, characterized in that: The IOU threshold in step S212 is set to a number between 0 and 1, such as 0.

5.

7. A system for implementing the pedestrian trajectory prediction method based on Deep-SORT and least squares method as claimed in claim 1, characterized in that: Including: 3D depth camera, improved Yolov7 algorithm module, Deep-SORT module, coordinate conversion module, least squares method module, Open NI library; using the 3D depth camera to collect video images, the Open NI library records and plays back the pedestrian walking video collected by the 3D depth camera, using the improved Yolov7 algorithm module to realize pedestrian detection and recognition, and using the Deep-SORT module based on the Kalman filter algorithm to realize pedestrian trajectory tracking; using the coordinate conversion module to calculate the three-dimensional coordinates of the pedestrian in the camera coordinate system, according to the calculated three-dimensional coordinates, the three-dimensional trajectory of the pedestrian is decoupled into horizontal and vertical two-dimensional trajectories through the vertical projection method, using the least squares method module to predict the horizontal and vertical two-dimensional trajectories of the pedestrian, and synthesizing the horizontal and vertical predicted trajectories into the three-dimensional trajectory of the pedestrian, realizing real-time single-target pedestrian trajectory prediction and multi-target pedestrian trajectory prediction.

8. The pedestrian trajectory prediction system based on Deep-SORT and least squares method according to claim 7, characterized in that: The improved Yolov7 algorithm module includes three parts: input layer, backbone network layer, and output layer; The input layer preprocesses the input video image, scales the image into a 640×640 RGB image, adds the adaptive anchor frame calculation function of the image, and finds the optimal anchor frame value in different trainings; The backbone network layer is mainly used to realize feature extraction, including CBS module, ELAN module and MP module; first, the CBS module changes the number of channels and performs feature extraction through 4 layers of convolution, changes the feature map size to 160×160×128, and replaces the head part with Bot3 structure, that is, a vision transformer structure, which fully considers the characteristics of pedestrian context through the global attention mechanism; then, the ELAN module controls the shortest path with the longest gradient, so that the network model can learn more picture characteristics, enhance its robustness, and keep the length and width of the image input and output unchanged; then, the MP module downsamples, and when the input and output channels are the same, the upper branch first halves the length and width through the MP algorithm, and then halves the channels through the CBS algorithm; while the lower branch halves the channels through a CBS algorithm, and then halves the length and width through a CBS algorithm; finally, the upper and lower branches are merged to halve the length and width, double the channels, and extract features; The output layer is used to predict image features, image target categories and credibility. The RepVGG-Style algorithm is introduced, the output layer network structure is upgraded, auxiliary head training and corresponding positive and negative sample matching strategies are used, and BConv, MP, CAT, and RAP modules are integrated to improve the overall performance and prediction accuracy of the Yolov7 network output layer. It can calibrate the positions of pedestrians in the recorded video at various times, obtain the three-dimensional trajectory coordinates of pedestrians in the camera coordinate system, and realize pedestrian detection.

9. The pedestrian trajectory prediction system based on Deep-SORT and least squares method according to claim 7, characterized in that: The improved Yolov7 algorithm module uses SiLU as the activation function.

10. The pedestrian trajectory prediction system based on Deep-SORT and least squares method according to claim 7, characterized in that: The Open NI library, nested in Python, is responsible for recording and playing back pedestrian walking videos collected by the Astro Pro 3D depth camera, providing a data source for subsequent pedestrian detection.

Images