Pedestrian trajectory prediction method and system based on multi-camera BEV view feature fusion based on attention mechanism

Abstract

This invention discloses a pedestrian trajectory prediction method based on multi-camera BEV perspective feature fusion based on an attention mechanism. The method comprises the following steps: S1, arranging M cameras for bird's-eye global perception to obtain multi-perspective images; S2, extracting high-dimensional semantic features from the multi-perspective images and outputting multi-scale feature maps using a feature pyramid network; S3, performing multi-target motion tracking on the multi-scale feature maps; S4, aggregating 2D sampling point features using an attention mechanism to update the tracking sequence and 3D reference points; S5, decoding the updated tracking sequence; S6, filtering out pedestrian target categories based on a preset confidence threshold, retaining tracking sequences with a confidence threshold above the preset confidence threshold and adding them to the next moment tracking sequence set; otherwise, removing them; S7, performing trajectory prediction based on the tracking sequence set to generate a target prediction trajectory for a certain future time domain. This invention can achieve pedestrian trajectory prediction in specific scenarios, thereby improving on-site monitoring.

Description

Technical Field

[0001] The present invention relates to the field of trajectory prediction, and in particular to a pedestrian trajectory prediction method and system based on multi-camera BEV perspective feature fusion based on an attention mechanism. Background Art

[0002] Risk monitoring is a crucial task for ensuring the safety of workers and the normal operation of production on construction sites. Existing risk monitoring methods can be broadly categorized into two types: non-manual methods based on machine learning and manual monitoring. Existing non-manual methods primarily utilize deep learning-based algorithms to monitor workers' immediate status, such as issuing warnings when a person enters a restricted area. However, due to the lack of predictions about pedestrians' future status, more proactive risk monitoring is difficult to achieve. Manual monitoring primarily involves on-site patrols and video surveillance. Both methods rely heavily on human experience and suffer from issues such as poor real-time performance, low supervisory efficiency, and high warning error rates.

[0003] Pedestrian trajectory prediction is an important research direction in computer vision. By analyzing the environment and pedestrians' historical motion information, real-time predictions of pedestrians' future trajectories can be achieved, making it suitable for proactive risk monitoring. Currently, pedestrian trajectory prediction methods are mainly divided into two categories: model-driven and data-driven. Model-driven pedestrian trajectory prediction methods primarily utilize manually designed prediction models to simulate pedestrian motion behavior and predict their movement patterns. Common examples include pedestrian trajectory prediction models based on Markov decision processes and trajectory prediction models based on kinematic models. Model-driven pedestrian trajectory prediction methods have weak generalization capabilities and are prone to increased computational complexity and decreased computational efficiency as the number of targets increases. With the development of deep learning, data-driven pedestrian trajectory prediction methods have developed rapidly. These methods utilize large amounts of training data to learn the motion characteristics of pedestrians. Their prediction performance is more suitable for handling pedestrian prediction tasks in complex environments than rule-based methods.

[0004] There are a large number of high-speed mechanical equipment and no-entry areas in construction sites. These dangerous areas pose a large number of potential risks. Therefore, accurate pedestrian trajectory prediction to prevent them from entering dangerous areas is of great significance to ensuring the safety of construction workers and normal production operations. Summary of the Invention

[0005] The main purpose of the present invention is to provide a pedestrian trajectory prediction method and system based on multi-camera BEV perspective feature fusion based on attention mechanism, which can effectively monitor potential risks in specific scenes.

[0006] The technical solution adopted in the present invention is:

[0007] A pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism is provided, which includes the following steps:

[0008] S1. Arrange M cameras at different positions in a specified scene to perform bird's-eye global perception and obtain multi-view images. Simultaneously, perform image calibration and timestamp alignment on the M cameras to obtain a homogeneous projection matrix from spatial points to image pixels in a unified reference frame.

[0009] S2. Input the multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features from low-dimensional image features, and use the feature pyramid network to output multi-scale feature maps;

[0010] S3, performing multi-target motion tracking on the multi-scale feature map to generate a target tracking sequence and a 3D reference point representing the spatial position of the corresponding target;

[0011] S4. According to the homogeneous projection matrix, the 3D reference point is projected onto the multi-view image of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points.

[0012] S5. Decode the updated tracking sequence to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence, which represent target motion characteristics;

[0013] S6. Filter pedestrian target categories based on a preset confidence threshold, and retain tracking sequences with confidence levels above the preset confidence threshold and add them to the next tracking sequence set; otherwise, remove them;

[0014] S7. Perform trajectory prediction based on the tracking sequence set to generate a target prediction trajectory in a certain future time domain.

[0015] Following the above technical solution, the pre-trained image feature extraction network is the ResNet-50 network.

[0016] Following the above technical solution, in step S3, a certain number of tracking sequences are randomly initialized to represent all observed moving targets in the scene, and then a multi-layer perceptron is used to decode the tracking sequences to generate corresponding 3D reference points representing the spatial position of the target.

[0017] Following the above technical solution, in step S5, two multi-layer perceptron networks are specifically used to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

[0018] Following the above technical solution, the tracking sequence eliminated in step S6 no longer participates in the tracking and trajectory prediction at the next moment.

[0019] Following the above technical solution, step S7 is specifically as follows:

[0020] After accumulating the multi-target tracking sequence of a certain observation time, the tracking sequence of a single target of a certain observation time is encoded using the long short-term memory network to obtain the encoding vector, and further obtain the hidden state vector;

[0021] The hidden state vectors of multiple moving targets at the previous moment are accumulated and combined with the hidden state vectors of other targets to generate the influence tensor at time t. This influence tensor implicitly contains the spatial interaction features between targets and is input into the next layer of LSTM module together with the encoding vector at time t.

[0022] Obtain the hidden state output by the last LSTM module, map it into a one-dimensional ternary array, and use the two-dimensional Gaussian distribution to predict the coordinates at time t+1, and further iterate to generate the target prediction trajectory in a certain future time domain.

[0023] Following the above technical solution, the ternary array includes the Gaussian distribution mean, standard deviation and correlation coefficient.

[0024] The present invention also provides a pedestrian trajectory prediction system based on multi-camera BEV view feature fusion based on attention mechanism, comprising:

[0025] The multi-view image acquisition module is used to arrange M cameras at different positions in a specified scene to perform bird's-eye view global perception and obtain multi-view images;

[0026] The calibration module is used to perform image calibration and timestamp alignment on M cameras to obtain the homogeneous projection matrix from spatial points to image pixels in a unified reference system;

[0027] The multi-scale feature map acquisition module is used to input multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features from low-dimensional image features, and output multi-scale feature maps using the feature pyramid network;

[0028] The multi-target tracking module is used to track multi-target motion using multi-scale feature maps, generating a target tracking sequence and 3D reference points representing the spatial positions of the corresponding targets. Based on the homogeneous projection matrix, the 3D reference points are projected onto the multi-view images of the corresponding camera to obtain a set of 2D sampling points. The 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points. The updated tracking sequence is decoded to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and the target category corresponding to the tracking sequence.

[0029] The pedestrian target screening module is used to screen pedestrian target categories according to a preset confidence threshold, and retain the tracking sequences that are higher than the preset confidence threshold and add them to the next moment tracking sequence set; otherwise, they are eliminated;

[0030] The trajectory prediction module is used to perform trajectory prediction based on the tracking sequence set and generate the target prediction trajectory in a certain future time domain.

[0031] Following the above technical solution, the multi-target tracking module specifically uses two multi-layer perceptron networks to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

[0032] The present invention also provides a computer storage medium storing a computer program executable by a processor, which executes the pedestrian trajectory prediction method based on multi-camera BEV perspective feature fusion based on the attention mechanism described in the above technical solution.

[0033] The beneficial effects of the present invention are as follows: the present invention designs a pedestrian trajectory prediction method and system based on the attention mechanism and the fusion of multi-camera BEV perspective features. Cameras are arranged in different directions in a specific scene (such as a construction site), and multi-scale and multi-perspective feature map sequences under different camera images are extracted. The information related to pedestrian movement is extracted by tracking sequence to complete multi-target tracking. The long short-term memory network is then used to extract the motion features in the historical trajectories of multiple targets in the scene and model the spatial interaction relationship of multiple targets. Finally, the pedestrian trajectory prediction in the specific scene is completed to serve more proactive and positive risk prediction.

[0034] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 Flowchart of a pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on an attention mechanism according to an embodiment of the present invention;

[0037] Figure 2This is a flow chart of a pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on an attention mechanism according to another embodiment of the present invention;

[0038] Figure 3 This is the flow chart of the multi-target tracking module;

[0039] Figure 4 This is the flow chart of the pedestrian trajectory prediction module. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] It should be noted that the illustrations provided in the embodiments of the present invention are only schematic illustrations of the basic concept of the present invention. Therefore, the drawings only show components related to the present invention and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0042] In the present invention, it should also be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" and the like are used to indicate positions or locations based on those shown in the accompanying drawings. These terms are intended solely to facilitate the description of the present application and to simplify the description. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present application. Furthermore, the terms "first" and "second" are used solely for descriptive and distinguishing purposes and should not be construed as indicating or implying relative importance.

[0043] like Figure 1 As shown, the pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on the attention mechanism in an embodiment of the present invention includes the following steps:

[0044] S1. Arrange M cameras at different positions in a specified scene to perform bird's-eye global perception and obtain multi-view images. Simultaneously, perform image calibration and timestamp alignment on the M cameras to obtain a homogeneous projection matrix from spatial points to image pixels in a unified reference frame.

[0045] S2. Input the multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features from low-dimensional image features, and use the feature pyramid network to output multi-scale feature maps;

[0046] S3, performing multi-target motion tracking on the multi-scale feature map to generate a target tracking sequence and a 3D reference point representing the spatial position of the corresponding target;

[0047] S4. According to the homogeneous projection matrix, the 3D reference point is projected onto the multi-view image of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points.

[0048] S5. Decode the updated tracking sequence to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence, which represent target motion characteristics;

[0049] S6. Filter pedestrian target categories based on a preset confidence threshold, and retain tracking sequences with confidence levels above the preset confidence threshold and add them to the next tracking sequence set; otherwise, remove them;

[0050] S7. Perform trajectory prediction based on the tracking sequence set to generate a target prediction trajectory in a certain future time domain.

[0051] Furthermore, the pre-trained image feature extraction network in step S2 may be a ResNet-50 network.

[0052] In step S3, a certain number of tracking sequences are randomly initialized to represent all observed moving targets in the scene, and then a multi-layer perceptron is used to decode the tracking sequences to generate corresponding 3D reference points representing the spatial position of the target.

[0053] In step S5, two multi-layer perceptron networks are used to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

[0054] The tracking sequence eliminated in step S6 no longer participates in the tracking and trajectory prediction at the next moment.

[0055] Step S7 is specifically as follows:

[0056] After accumulating the multi-target tracking sequence of a certain observation time, the tracking sequence of a single target of a certain observation time is encoded using the long short-term memory network to obtain the encoding vector, and further obtain the hidden state vector;

[0057] The hidden state vectors of multiple moving targets at the previous moment are accumulated and combined with the hidden state vectors of other targets to generate the influence tensor at time t. This influence tensor implicitly contains the spatial interaction features between targets and is input into the next layer of LSTM module together with the encoding vector at time t.

[0058] Obtain the hidden state output by the last LSTM module and map it into a one-dimensional ternary array (which may include the mean, standard deviation, and correlation coefficient of the Gaussian distribution). Use the two-dimensional Gaussian distribution to predict the coordinates at time t+1, and further iterate to generate the target prediction trajectory in a certain future time domain.

[0059] This embodiment can arrange cameras at different positions in the construction site scene, and extract multi-scale and multi-view feature map sequences under different camera images, and use the tracking sequence method to extract information related to pedestrian movement to complete multi-target tracking; then use the long short-term memory network to extract the motion features in the historical trajectories of multiple targets in the scene and model the spatial interaction relationship of multiple targets, and finally complete the pedestrian trajectory prediction in the construction site scene to serve more proactive and positive risk prediction.

[0060] In another embodiment of the present invention, Figure 2 As shown in FIG, the pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on the attention mechanism includes the following steps:

[0061] S1. Arrange M cameras at different locations in the scene to achieve global perception. At the same time, perform image calibration and timestamp alignment on the M cameras to obtain the homogeneous projection matrix from spatial points to image pixels in a unified reference system. ,in , used in the subsequent multi-target tracking module; is the set of real numbers.

[0062] S2. Input the multi-view images into the pre-trained image feature extraction network ResNet-50 in parallel to extract the high-dimensional semantic features of the low-dimensional image features, and use the Feature Pyramid Network (FPN) to output the multi-scale feature map. The multi-scale feature of the m-th view is represented as , and are the length and width of the i-th scale feature map, respectively, and C is the number of feature map channels.

[0063] S3, input the obtained multi-view multi-scale feature map into the multi-target motion tracking module. Figure 3 As shown in the figure, a certain number of tracking sequences are first randomly initialized to represent all observed moving targets in the scene, and then a multi-layer perceptron is used to decode the tracking sequences to generate corresponding 3D reference points representing the spatial position of the target.

[0064] S4. Next, the 3D reference point is projected onto the multi-view feature map of the corresponding camera using the homogeneous projection matrix to obtain a set of 2D sampling points. The multi-view 2D sampling point features are then aggregated using the attention mechanism to update the tracking sequence features and their 3D reference points.

[0065] S5. Use two multi-layer perceptron (MLP) networks to decode the updated tracking sequence features respectively. The first MLP network decodes the target 3D prediction box center point that represents the target motion characteristics. , 3D prediction box size , target speed in pixel coordinate system and the target's orientation information , another MLP network decodes to obtain the category corresponding to the sequence;

[0066] S6. Pedestrian target categories are screened based on a preset confidence threshold, and sequences corresponding to those above the threshold are retained and added to the next moment tracking sequence set; otherwise, they are removed and no longer participate in the next moment tracking and prediction;

[0067] S7, the multi-target tracking sequence features obtained by parallel updating are input into the trajectory prediction module. After accumulating the multi-target tracking sequence features of a certain observation time, the trajectory prediction module uses the Long Short-Term Memory (LSTM) network to encode the tracking sequence features of a single target with a certain observation time into a vector To obtain the hidden state vector . The hidden state vectors of multiple moving targets at the previous moment are Accumulate and combine the hidden state vectors of other targets To generate the influence tensor at time t , which implicitly contains the spatial interaction features between targets and is the same as the encoding vector at time t , and input the next layer of LSTM module together; the LSTM network in the present invention is a sequence model containing multiple LSTM modules, which is used to process time series data. The LSTM module is the basic unit for executing the LSTM mechanism, and multiple modules can be stacked to form a complete LSTM network.

[0068] S8. After executing S7 multiple times, obtain the hidden state of the last layer of LSTM module output ,Will Mapped into a one-dimensional three-element array: , which represent the mean, standard deviation and correlation coefficient in the Gaussian distribution, and use the two-dimensional Gaussian distribution to predict the coordinates at time t+1 , further iteratively generates the target prediction trajectory in a certain future time domain.

[0069] Specifically, in step S1, a fixed platform can be set up on the construction site, and M cameras can be arranged on the platform. By adjusting the camera shooting angle and height, the multi-view images can cover the entire scene. In step S2, the collected RGB image can be imported into the pre-trained ResNet-50 backbone feature extraction network, and the FPN outputs a multi-scale feature map. The multi-scale feature output of a single camera is represented as , is the feature map length, is the feature map width, is the number of feature map channels.

[0070] In step S3, the multi-target tracking module first randomly initializes the sequence ,in Indicates the observed Input it into the MLP network to obtain the initial 3D reference point sequence , represents the target observed in the scene Further, using the pre-calibrated Homogeneous projection matrix of the cameras Project the 3D reference points corresponding to each tracking sequence into the feature map coordinate system under different viewing angles to obtain the 2D sampling point set corresponding to each sequence. , the calculation expression is as follows:

[0071]

[0072] Among them, Figure 3 As shown, the multi-target tracking module includes Layer attention module, only the first layer of 2D sampling points are obtained by projection transformation as 3D reference points, and the remaining layers of 2D sampling points are obtained by feature update completed by the network. The input of the attention module is: the current tracking sequence set output by the attention module of the previous layer, the current sequence of each attention module output by the attention module of the previous layer Corresponding to the set of 2D sampling points for each view , and the current multi-view multi-scale image features First, the self-attention module in the attention module of this layer is used to update the tracking sequence features, and the weight coefficient of the current layer is obtained by MLP encoding. Then, the weight system and the multi-view and multi-scale image features of the current moment are used to update the tracking sequence features. , the current sequence of the output of the previous attention module Corresponding to the set of 2D sampling points for each view , output The current moment tracking sequence set and 2D sampling point set of the layer are calculated as follows, where the moment Omit to simplify:

[0073]

[0074]

[0075]

[0076] in Indicates the layer number of the current attention module, Indicates the The first camera Layer feature map, The current layer weight coefficient obtained by MLP encoding after the self-attention module updates the tracking sequence features, is the sigmoid function, is the position encoding function.

[0077] Furthermore, in step S5, two MLP networks are used to decode the last layer of tracking sequence features output by the attention module. One MLP network decodes and generates the ID label and confidence value of the target corresponding to the sequence, and the other MLP network decodes and generates a 10-dimensional vector expressing the target motion characteristics, including: 3D position , the 3D size of the target , target speed in pixel coordinate system , and the target's orientation information The calculation expression of the target position in 3D space is as follows:

[0078]

[0079] After the position information of all targets in the observation time domain is accumulated in step S7, the historical motion information of multiple targets is formed as the input of the trajectory prediction model. The moment goals Mapping to generate encoding vector , which is calculated as:

[0080]

[0081] in, is the mapping function using the ReLu nonlinear function, For its weight.

[0082] exist At this moment, the encoding vector , influence tensor and the hidden state vector at the previous moment , are input into the LSTM module together to obtain the hidden state vector corresponding to time t , calculated as:

[0083]

[0084] Influence Tensor It is used to implicitly represent the spatial interaction relationship between targets and is calculated as follows:

[0085]

[0086] in, Represents the total number of other targets around target i.

[0087] Specifically, in step S8, the hidden state output by the last LSTM module in step S7 is obtained. ,Will Mapped into a one-dimensional three-element array: , which represent the mean, standard deviation and correlation coefficient in the Gaussian distribution, and use the two-dimensional Gaussian distribution to predict the coordinates at time t+1 .

[0088] The calculation method is:

[0089]

[0090]

[0091] Furthermore, the coordinates predicted at time t+1 are input into step S7 to predict the trajectory point at time t+2. This process is repeated until all the predicted trajectory points within the prediction time are output.

[0092] After multiple iterative operations of the trajectory prediction module, this embodiment effectively realizes pedestrian trajectory prediction in the construction site scenario, and achieves full coverage of the scene by arranging multi-view cameras; constructing a multi-target tracking and trajectory prediction module based on multi-view image features can achieve more proactive and active safety prediction to assess the random and changeable potential risks in the construction site scenario, thereby improving the intelligence level of the construction site safety monitoring system.

[0093] The present invention also provides a pedestrian trajectory prediction system based on multi-camera BEV view feature fusion based on an attention mechanism, which is mainly used to implement the above method embodiment. The system includes:

[0094] The multi-view image acquisition module is used to arrange M cameras at different positions in a specified scene to perform bird's-eye view global perception and obtain multi-view images;

[0095] The calibration module is used to perform image calibration and timestamp alignment on M cameras to obtain the homogeneous projection matrix from spatial points to image pixels in a unified reference system;

[0096] The multi-scale feature map acquisition module is used to input multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features from low-dimensional image features, and output multi-scale feature maps using the feature pyramid network;

[0097] The multi-target tracking module is used to track multi-target motion using multi-scale feature maps, generating a target tracking sequence and 3D reference points representing the spatial positions of the corresponding targets. Based on the homogeneous projection matrix, the 3D reference points are projected onto the multi-view images of the corresponding camera to obtain a set of 2D sampling points. The 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points. The updated tracking sequence is decoded to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and the target category corresponding to the tracking sequence.

[0098] The pedestrian target screening module is used to screen pedestrian target categories according to a preset confidence threshold, and retain the tracking sequences that are higher than the preset confidence threshold and add them to the next moment tracking sequence set; otherwise, they are eliminated;

[0099] The trajectory prediction module is used to perform trajectory prediction based on the tracking sequence set and generate the target prediction trajectory in a certain future time domain.

[0100] Furthermore, the multi-target tracking module specifically uses two multi-layer perceptron networks to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

[0101] The various modules of the system are mainly used to implement the various steps of the method embodiment, which will not be described in detail here.

[0102] The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic storage device, a disk, an optical disk, a server, an App store, etc., storing a computer program that implements corresponding functions when executed by a processor. When executed by a processor, the computer-readable storage medium of this embodiment implements the pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on an attention mechanism according to the method embodiment.

[0103] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, or two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0104] The size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0105] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism, characterized by: The following steps are involved: S1. Arrange M cameras at different positions in a specified scene to perform bird's-eye global perception and obtain multi-view images. Simultaneously, perform image calibration and timestamp alignment on the M cameras to obtain a homogeneous projection matrix from spatial points to image pixels in a unified reference frame. S2. Input the multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features from low-dimensional image features, and use the feature pyramid network to output multi-scale feature maps; S3, performing multi-target motion tracking on the multi-scale feature map to generate a target tracking sequence and a 3D reference point representing the spatial position of the corresponding target; S4. According to the homogeneous projection matrix, the 3D reference point is projected onto the multi-view image of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points. S5. Decode the updated tracking sequence to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence, which represent target motion characteristics; S6. Filter pedestrian target categories based on a preset confidence threshold, and retain tracking sequences that exceed the preset confidence threshold and add them to the next moment tracking sequence set; Otherwise, it will be eliminated; S7. Perform trajectory prediction based on the tracking sequence set to generate a target prediction trajectory in a certain future time domain.

2. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: The pre-trained image feature extraction network is the ResNet-50 network.

3. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: In step S3, a certain number of tracking sequences are randomly initialized to represent all observed moving targets in the scene, and then a multi-layer perceptron is used to decode the tracking sequences to generate corresponding 3D reference points representing the spatial position of the target.

4. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: In step S5, two multi-layer perceptron networks are used to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

5. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: The tracking sequence eliminated in step S6 no longer participates in the tracking and trajectory prediction at the next moment.

6. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: Step S7 is specifically as follows: After accumulating the multi-target tracking sequence of a certain observation time, the tracking sequence of a single target of a certain observation time is encoded using the long short-term memory network to obtain the encoding vector, and further obtain the hidden state vector; The hidden state vectors of multiple moving targets at the previous moment are accumulated and combined with the hidden state vectors of other targets to generate the influence tensor at time t. This influence tensor implicitly contains the spatial interaction features between targets and is input into the next layer of LSTM module together with the encoding vector at time t. Obtain the hidden state output by the last LSTM module, map it into a one-dimensional ternary array, and use the two-dimensional Gaussian distribution to predict the coordinates at time t+1, and further iterate to generate the target prediction trajectory in a certain future time domain.

7. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 6 is characterized in that: A three-element array containing the mean, standard deviation, and correlation coefficient of a Gaussian distribution.

8. A pedestrian trajectory prediction system based on multi-camera BEV view feature fusion based on attention mechanism, characterized by: include: The multi-view image acquisition module is used to arrange M cameras at different positions in a specified scene to perform bird's-eye view global perception and obtain multi-view images; The calibration module is used to perform image calibration and timestamp alignment on M cameras to obtain the homogeneous projection matrix from spatial points to image pixels in a unified reference system; The multi-scale feature map acquisition module is used to input multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features from low-dimensional image features, and output multi-scale feature maps using the feature pyramid network; The multi-target tracking module is used to track multi-target motion using multi-scale feature maps, generating a target tracking sequence and 3D reference points representing the spatial positions of the corresponding targets. Based on the homogeneous projection matrix, the 3D reference points are projected onto the multi-view images of the corresponding camera to obtain a set of 2D sampling points. The 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points. The updated tracking sequence is decoded to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and the target category corresponding to the tracking sequence. The pedestrian target screening module is used to screen the pedestrian target categories according to the preset confidence threshold, and retain the tracking sequences that are higher than the preset confidence threshold and add them to the next moment tracking sequence set; Otherwise, it will be eliminated; The trajectory prediction module is used to perform trajectory prediction based on the tracking sequence set and generate the target prediction trajectory in a certain future time domain.

9. The pedestrian trajectory prediction system based on multi-camera BEV view feature fusion based on attention mechanism according to claim 8 is characterized in that: The multi-target tracking module specifically uses two multi-layer perceptron networks to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

10. A computer storage medium, characterized in that A computer program executable by a processor is stored therein, which executes the pedestrian trajectory prediction method based on multi-camera BEV perspective feature fusion based on the attention mechanism as described in any one of claims 1-7.

Images