Pedestrian trajectory prediction method and system based on attention mechanism multi-camera BEV visual angle feature fusion

Abstract

The invention discloses a pedestrian trajectory prediction method for multi-camera BEV visual angle feature fusion based on an attention mechanism, and the method comprises the following steps: S1, arranging M cameras to carry out bird's-eye view global perception, and obtaining a multi-visual angle image; s2, extracting high-dimensional semantic features of the multi-view image, and outputting a multi-scale feature map by using a feature pyramid network; s3, performing multi-target motion tracking on the multi-scale feature map; s4, performing aggregation by using an attention mechanism to obtain 2D sampling point features, and updating the tracking sequence and the 3D reference points; s5, decoding the updated tracking sequence; s6, screening pedestrian target categories according to a preset reliability threshold value, and reserving the tracking sequences higher than the preset reliability threshold value and adding the tracking sequences into a tracking sequence set at the next moment; otherwise, rejecting; and S7, performing trajectory prediction according to the tracking sequence set, and generating a target prediction trajectory of a certain future time domain. According to the invention, pedestrian trajectory prediction in a specific scene can be realized, so that on-site monitoring can be better realized.

Description

Technical Field

[0001] The present invention relates to the field of trajectory prediction, and in particular to a pedestrian trajectory prediction method and system based on multi-camera BEV view feature fusion based on an attention mechanism. Background Art

[0002] Risk monitoring is an important task to ensure the safety of personnel and normal production operations in construction site scenarios. Existing risk monitoring methods can be roughly divided into two types: non-manual methods based on machine learning and manual monitoring. Existing non-manual methods mainly use algorithms based on deep learning to monitor the immediate status of workers. For example, when a person enters a prohibited area, an early warning is issued. Due to the lack of prediction of the future status of pedestrians, it is difficult to achieve more proactive risk monitoring. Manual monitoring mainly includes two methods: on-site patrols and video monitoring. Since both methods are highly dependent on manual experience, there are problems such as poor real-time performance, low supervision efficiency, and high early warning error rate.

[0003] Pedestrian trajectory prediction is an important research direction in the field of computer vision. By analyzing the environment and the pedestrian's historical motion information, the real-time prediction of the pedestrian's future trajectory can be achieved. Therefore, it can be used in active risk monitoring. At present, pedestrian trajectory prediction methods are mainly divided into two categories: model-driven and data-driven. Model-driven pedestrian trajectory prediction methods mainly use artificially designed prediction models to simulate pedestrian movement behaviors and predict their movement patterns. Common ones include pedestrian trajectory prediction models based on Markov decision processes and trajectory prediction models based on kinematic models. Model-driven pedestrian trajectory prediction methods have weak generalization capabilities, and with the increase in the number of targets, they are prone to problems of increased computational complexity and decreased computational efficiency. With the development of deep learning, data-driven pedestrian trajectory prediction methods have developed rapidly. Such methods use a large amount of training data to learn the motion characteristics of pedestrians. Their prediction performance is more suitable for handling pedestrian prediction tasks in complex environments than rule-based methods.

[0004] There are a large number of high-speed mechanical equipment and prohibited areas on construction sites. There are a large number of potential risks in these dangerous areas. Therefore, accurate pedestrian trajectory prediction to prevent them from entering dangerous areas is of great significance to ensuring the safety of construction workers and normal production operations. Summary of the invention

[0005] The main purpose of the present invention is to provide a pedestrian trajectory prediction method and system based on multi-camera BEV perspective feature fusion based on attention mechanism, which can effectively monitor potential risks in specific scenes.

[0006] The technical solution adopted by the present invention is:

[0007] A pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism is provided, comprising the following steps:

[0008] S1. Arrange M cameras at different positions of the specified scene to perform bird's-eye view global perception to obtain multi-view images. At the same time, perform image calibration and timestamp alignment on the M cameras to obtain a homogeneous projection matrix from spatial points to image pixels in a unified reference system.

[0009] S2, respectively input the multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features of low-dimensional image features, and use the feature pyramid network to output multi-scale feature maps;

[0010] S3, performing multi-target motion tracking on the multi-scale feature map, generating a target tracking sequence and a 3D reference point representing the spatial position of the corresponding target;

[0011] S4. According to the camera external parameter projection matrix, the 3D reference point is projected onto the multi-view image of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points.

[0012] S5, decoding the updated tracking sequence to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence that characterize the target motion characteristics;

[0013] S6. Filter out pedestrian target categories according to a preset confidence threshold, and retain tracking sequences with confidence levels higher than the preset confidence threshold and add them to the next moment tracking sequence set; otherwise, remove them;

[0014] S7. Perform trajectory prediction based on the tracking sequence set to generate a target prediction trajectory in a certain future time domain.

[0015] Following the above technical solution, the pre-trained image feature extraction network is a ResNet-50 network.

[0016] Following the above technical solution, in step S3, a certain number of tracking sequences are first randomly initialized to represent all observed moving targets in the scene, and then a multi-layer perceptron is used to decode the tracking sequences to generate corresponding 3D reference points representing the spatial position of the target.

[0017] Following the above technical solution, in step S5, two multi-layer perceptron networks are specifically used to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that characterize the target motion characteristics, and the other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

[0018] According to the above technical solution, the tracking sequence eliminated in step S6 no longer participates in the tracking and trajectory prediction at the next moment.

[0019] Following the above technical solution, step S7 is specifically as follows:

[0020] After accumulating the multi-target tracking sequence of a certain observation time, the tracking sequence of a single target of a certain observation time is encoded using the long short-term memory network to obtain the encoding vector, and further obtain the hidden state vector;

[0021] The hidden state vectors of multiple moving targets at the previous moment are accumulated, and the hidden state vectors of other targets are combined to generate the influence tensor at time t. The influence tensor implicitly contains the spatial interaction features between targets and is input into the next layer of LSTM module together with the encoding vector at time t.

[0022] The hidden state output by the last layer of LSTM modules is obtained, mapped into a one-dimensional ternary array, and the coordinates at time t+1 are predicted using a two-dimensional Gaussian distribution, and the target prediction trajectory in a certain future time domain is further iterated.

[0023] Following the above technical solution, the ternary array includes the Gaussian distribution mean, standard deviation and correlation coefficient.

[0024] The present invention also provides a pedestrian trajectory prediction system based on multi-camera BEV view feature fusion based on attention mechanism, comprising:

[0025] The multi-view image acquisition module is used to arrange M cameras at different positions of a specified scene to perform bird's-eye view global perception and obtain multi-view images;

[0026] The calibration module is used to calibrate the images and align the timestamps of the M cameras, and obtain the homogeneous projection matrix from the spatial points to the image pixels in a unified reference system;

[0027] A multi-scale feature map acquisition module is used to input multi-view images in parallel into a pre-trained image feature extraction network to extract high-dimensional semantic features of low-dimensional image features, and output a multi-scale feature map using a feature pyramid network;

[0028] The multi-target tracking module is used to track multi-target motion on multi-scale feature maps, generate target tracking sequences and 3D reference points representing the spatial positions of corresponding targets; according to the camera extrinsic projection matrix, the 3D reference points are projected onto the multi-view images of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points; the updated tracking sequence is decoded to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence that represent the target motion characteristics;

[0029] The pedestrian target screening module is used to screen the pedestrian target categories according to the preset confidence threshold, and retain the tracking sequence that is higher than the preset confidence threshold and add it to the next moment tracking sequence set; otherwise, it will be eliminated;

[0030] The trajectory prediction module is used to perform trajectory prediction based on the tracking sequence set and generate the target prediction trajectory in a certain future time domain.

[0031] Following the above technical solution, the multi-target tracking module specifically uses two multi-layer perceptron networks to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

[0032] The present invention also provides a computer storage medium storing a computer program executable by a processor, wherein the computer program executes the pedestrian trajectory prediction method based on multi-camera BEV perspective feature fusion based on the attention mechanism as described in the above technical solution.

[0033] The beneficial effects of the present invention are as follows: the present invention designs a pedestrian trajectory prediction method and system based on multi-camera BEV perspective feature fusion of an attention mechanism, arranges cameras in different positions in a specific scene (such as a construction site), and extracts multi-scale and multi-perspective feature map sequences under different camera images, and uses the tracking sequence method to extract information related to pedestrian motion to complete multi-target tracking, and then uses a long short-term memory network to extract motion features in the historical trajectories of multiple targets in the scene and model the multi-target spatial interaction relationship, and finally completes pedestrian trajectory prediction in a specific scene to serve more proactive risk prediction.

[0034] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 is a flow chart of a pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on an attention mechanism in an embodiment of the present invention;

[0037] Figure 2is a flow chart of a pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on an attention mechanism in another embodiment of the present invention;

[0038] Figure 3 It is the flow chart of multi-target tracking module;

[0039] Figure 4 It is the flow chart of pedestrian trajectory prediction module. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] It should be noted that the illustrations provided in the embodiments of the present invention are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout type may also be more complicated.

[0042] In the present invention, it is also necessary to explain that, if the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, if the terms "first" and "second" appear, they are only used for description and distinction purposes, and cannot be understood as indicating or implying relative importance.

[0043] like Figure 1 As shown, the pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on the attention mechanism in an embodiment of the present invention includes the following steps:

[0044] S1. Arrange M cameras at different positions of the specified scene to perform bird's-eye view global perception to obtain multi-view images. At the same time, perform image calibration and timestamp alignment on the M cameras to obtain a homogeneous projection matrix from spatial points to image pixels in a unified reference system.

[0045] S2, respectively input the multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features of low-dimensional image features, and use the feature pyramid network to output multi-scale feature maps;

[0046] S3, performing multi-target motion tracking on the multi-scale feature map, generating a target tracking sequence and a 3D reference point representing the spatial position of the corresponding target;

[0047] S4. According to the camera external parameter projection matrix, the 3D reference point is projected onto the multi-view image of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points.

[0048] S5, decoding the updated tracking sequence to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence that characterize the target motion characteristics;

[0049] S6. Filter out pedestrian target categories according to a preset confidence threshold, and retain tracking sequences with confidence levels higher than the preset confidence threshold and add them to the next moment tracking sequence set; otherwise, remove them;

[0050] S7. Perform trajectory prediction based on the tracking sequence set to generate a target prediction trajectory in a certain future time domain.

[0051] Furthermore, the pre-trained image feature extraction network in step S2 may be a ResNet-50 network.

[0052] In step S3, a certain number of tracking sequences are first randomly initialized to represent all observed moving targets in the scene, and then a multi-layer perceptron is used to decode the tracking sequences to generate corresponding 3D reference points representing the spatial positions of the targets.

[0053] In step S5, two multi-layer perceptron networks are specifically used to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that characterize the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

[0054] The tracking sequence eliminated in step S6 no longer participates in the tracking and trajectory prediction at the next moment.

[0055] Step S7 is specifically as follows:

[0056] After accumulating the multi-target tracking sequence of a certain observation time, the tracking sequence of a single target of a certain observation time is encoded using the long short-term memory network to obtain the encoding vector, and further obtain the hidden state vector;

[0057] The hidden state vectors of multiple moving targets at the previous moment are accumulated, and the hidden state vectors of other targets are combined to generate the influence tensor at time t. The influence tensor implicitly contains the spatial interaction features between targets and is input into the next layer of LSTM module together with the encoding vector at time t.

[0058] The hidden state of the output of the last layer of LSTM module is obtained and mapped into a one-dimensional ternary array (the ternary array may include the mean, standard deviation and correlation coefficient of the Gaussian distribution), and the coordinates at time t+1 are predicted using the two-dimensional Gaussian distribution, and the target prediction trajectory in a certain future time domain is further iterated.

[0059] This embodiment can arrange cameras at different positions in the construction site scene, and extract multi-scale and multi-view feature map sequences under different camera images, and use the tracking sequence method to extract information related to pedestrian movement to complete multi-target tracking; then use the long short-term memory network to extract the motion features in the historical trajectories of multiple targets in the scene and model the spatial interaction relationship of multiple targets, and finally complete the pedestrian trajectory prediction in the construction site scene to serve more proactive risk prediction.

[0060] In another embodiment of the present invention, Figure 2 As shown in FIG, the pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on the attention mechanism includes the following steps:

[0061] S1. Arrange M cameras at different positions of the scene to achieve global perception. At the same time, perform image calibration and timestamp alignment on the M cameras to obtain the homogeneous projection matrix from spatial points to image pixels in a unified reference system. in Used in subsequent multi-target tracking modules; is the set of real numbers.

[0062] S2. Input the multi-view images into the pre-trained image feature extraction network ResNet-50 in parallel to extract high-dimensional semantic features of low-dimensional image features, and use the Feature Pyramid Networks (FPN) to output multi-scale feature maps. The multi-scale features of the m-th view are expressed as H i and W i are the length and width of the i-th scale feature map, respectively, and C is the number of feature map channels.

[0063] S3, input the obtained multi-view multi-scale feature map into the multi-target motion tracking module. Figure 3 As shown, firstly, a certain number of tracking sequences are randomly initialized to represent all observed moving targets in the scene, and then a multi-layer perceptron is used to decode the tracking sequences to generate corresponding 3D reference points representing the spatial position of the target.

[0064] S4. Next, the camera calibration matrix is ​​used to project the 3D reference points onto the multi-view feature map of the corresponding camera to obtain a set of 2D sampling points. Then, the attention mechanism is used to aggregate the multi-view 2D sampling point features to update the tracking sequence features and their 3D reference points.

[0065] S5. Use two multi-layer perceptron (MLP) networks to decode the updated tracking sequence features respectively. The first MLP network decodes to obtain the center point of the target 3D prediction box that represents the target motion characteristics. 3D prediction box size Target speed in pixel coordinate system and the target's orientation information Another MLP network decodes to obtain the category corresponding to the sequence;

[0066] S6. Pedestrian target categories are screened according to a preset confidence threshold, and the corresponding sequences above the threshold are retained and added to the next moment tracking sequence set; otherwise, they are removed and no longer participate in the next moment tracking and prediction;

[0067] S7, the multi-target tracking sequence features obtained by parallel update are input into the trajectory prediction module. After accumulating the multi-target tracking sequence features of a certain observation time, the trajectory prediction module uses the Long Short-Term Memory (LSTM) network to encode the tracking sequence features of a single target with a certain observation time into a vector To obtain the hidden state vector The hidden state vectors of multiple moving targets at the previous moment Accumulate and combine the hidden state vectors of other targets To generate the influence tensor at time t This tensor implicitly contains the spatial interaction features between objects and is the same as the encoding vector at time t The LSTM network in the present invention is a sequence model including multiple LSTM modules, which is used to process time series data. The LSTM module is the basic unit for executing the LSTM mechanism, and multiple modules can be stacked to form a complete LSTM network.

[0068] S8. After executing S7 multiple times, obtain the hidden state of the output of the last layer of LSTM module Will Mapped into a one-dimensional three-element array: They represent the mean, standard deviation and correlation coefficient in the Gaussian distribution, and use the two-dimensional Gaussian distribution to predict the coordinates at time t+1 Further iteration is performed to generate the target prediction trajectory in a certain future time domain.

[0069] Specifically, in step S1, a fixed platform can be set up in the construction site scene, and M cameras can be arranged on the platform. By adjusting the camera shooting angle and height, the multi-view images can cover the global scene. In step S2, the collected RGB image can be imported into the pre-trained ResNet-50 backbone feature extraction network, and the FPN outputs a multi-scale feature map. The multi-scale feature output of a single camera is represented as H is the feature map length, W is the feature map width, and C is the number of feature map channels.

[0070] In step S3, the multi-target tracking module first randomly initializes the sequence q = q1, q2, q3 ... q i , where i represents the observed i-th target. It is input into the MLP network to obtain the initialized 3D reference point sequence c = c1, c2, c3…c i , represents the spatial position of the observed target i in the scene. Further, using the projection matrix of the pre-calibrated M cameras Project the 3D reference points corresponding to each tracking sequence into the feature map coordinate system under different viewing angles to obtain the 2D sampling point set d corresponding to each sequence. i =d 1i ,d 2i ,…,d mi , the calculation expression is as follows:

[0071]

[0072] Among them, Figure 3 As shown in Figure 1, the multi-target tracking module includes L layers of attention modules. Only the first layer of 2D sampling points are obtained by projection transformation as 3D reference points, and the remaining layers of 2D sampling points are obtained by network feature update. For the l-th layer of attention module, its input includes: the current tracking sequence set output by the previous layer of attention module, the current sequence output by the previous layer of attention module Corresponding to each viewpoint 2D sampling point set d i , and the current multi-view multi-scale image features First, the self-attention module in the attention module of this layer is used to update the tracking sequence features, and the weight coefficient of the current layer is obtained by MLP encoding. Then, the weight system and the multi-view and multi-scale image features at the current moment are used to The current sequence of the output of the previous attention module Corresponding to each viewpoint 2D sampling point set d i , output the current tracking sequence set and 2D sampling point set of the l+1th layer. The calculation expression is as follows, where the time t is omitted for simplicity:

[0073]

[0074] Where l represents the layer number of the current attention module, F km represents the k-th layer feature map of the M-th camera, w kmi l is the weight coefficient of the current layer obtained by MLP encoding after the self-attention module updates the tracking sequence features, σ(·) is the sigmoid function, and PE(·) is the position encoding function.

[0075] Furthermore, in step S5, two MLP networks are used to decode the last layer of tracking sequence features output by the attention module. One MLP network decodes and generates the ID label and confidence value of the target corresponding to the sequence, and the other MLP network decodes and generates a 10-dimensional vector expressing the target motion features, including: 3D position 3D size of the target Target speed in pixel coordinate system And the target's orientation information The calculation expression of the target position in 3D space is as follows:

[0076] p i =c i +MLP(q i )

[0077] After the position information of all targets in the observation time domain is accumulated in step S7, the historical motion information of multiple targets is used as the input of the trajectory prediction model. Mapping to generate encoding vector The calculation method is:

[0078]

[0079] Among them, φ(·) is the mapping function using the ReLu nonlinear function, W r for its weight.

[0080] At time t, the encoding vector Influence Tensor and the hidden state vector at the previous moment Input them into the LSTM module together to obtain the hidden state vector corresponding to time t The calculation method is:

[0081]

[0082] Influence Tensor It is used to implicitly characterize the spatial interaction relationship between targets and is calculated as follows:

[0083]

[0084] Among them, N i Represents the total number of other targets around target i.

[0085] Specifically, in step S8, the hidden state output by the last LSTM module in step S7 is obtained. Will Mapped into a one-dimensional three-element array: They represent the mean, standard deviation and correlation coefficient in the Gaussian distribution, and use the two-dimensional Gaussian distribution to predict the coordinates at time t+1

[0086] The calculation method is:

[0087]

[0088] Furthermore, the predicted coordinates at time t+1 are input into step S7 to predict the trajectory points at time t+2. The process is repeated until all predicted trajectory points within the prediction duration are output.

[0089] After multiple iterative operations of the trajectory prediction module, this embodiment effectively realizes the pedestrian trajectory prediction in the construction site scene, and achieves full coverage of the scene by arranging multi-view cameras; constructing a multi-target tracking and trajectory prediction module based on multi-view image features can achieve more proactive and active safety prediction to assess the random and changeable potential risks in the construction site scene, thereby improving the intelligence level of the construction site safety monitoring system.

[0090] The present invention also provides a pedestrian trajectory prediction system based on multi-camera BEV view feature fusion based on attention mechanism, which is mainly used to implement the above method embodiment. The system includes:

[0091] The multi-view image acquisition module is used to arrange M cameras at different positions of a specified scene to perform bird's-eye view global perception and obtain multi-view images;

[0092] The calibration module is used to calibrate the images and align the timestamps of the M cameras, and obtain the homogeneous projection matrix from the spatial points to the image pixels in a unified reference system;

[0093] A multi-scale feature map acquisition module is used to input multi-view images in parallel into a pre-trained image feature extraction network to extract high-dimensional semantic features of low-dimensional image features, and output a multi-scale feature map using a feature pyramid network;

[0094] The multi-target tracking module is used to track multi-target motion on multi-scale feature maps, generate target tracking sequences and 3D reference points representing the spatial positions of corresponding targets; according to the camera extrinsic projection matrix, the 3D reference points are projected onto the multi-view images of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points; the updated tracking sequence is decoded to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence that represent the target motion characteristics;

[0095] The pedestrian target screening module is used to screen the pedestrian target categories according to the preset confidence threshold, and retain the tracking sequence that is higher than the preset confidence threshold and add it to the next moment tracking sequence set; otherwise, it will be eliminated;

[0096] The trajectory prediction module is used to perform trajectory prediction based on the tracking sequence set and generate the target prediction trajectory in a certain future time domain.

[0097] Furthermore, the multi-target tracking module specifically uses two multi-layer perceptron networks to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

[0098] The various modules of the system are mainly used to implement the various steps of the method embodiment, which will not be described one by one here.

[0099] The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and the program realizes corresponding functions when executed by a processor. The computer-readable storage medium of this embodiment realizes the pedestrian trajectory prediction method based on the multi-camera BEV perspective feature fusion of the method embodiment based on the attention mechanism when executed by the processor.

[0100] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0101] The order of execution of each step in the above embodiment does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0102] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism, characterized in that: The following steps are involved: S1. Arrange M cameras at different positions of the specified scene to perform bird's-eye view global perception to obtain multi-view images. At the same time, perform image calibration and timestamp alignment on the M cameras to obtain a homogeneous projection matrix from spatial points to image pixels in a unified reference system. S2, respectively input the multi-view images into the pre-trained image feature extraction network in parallel to extract high-dimensional semantic features of low-dimensional image features, and use the feature pyramid network to output multi-scale feature maps; S3, performing multi-target motion tracking on the multi-scale feature map, generating a target tracking sequence and a 3D reference point representing the spatial position of the corresponding target; S4. According to the camera external parameter projection matrix, the 3D reference point is projected onto the multi-view image of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points. S5, decoding the updated tracking sequence to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence that characterize the target motion characteristics; S6. Filter out pedestrian target categories according to a preset confidence threshold, and retain tracking sequences that are higher than the preset confidence threshold and add them to the next moment tracking sequence set; Otherwise, it will be eliminated; S7. Perform trajectory prediction based on the tracking sequence set to generate a target prediction trajectory in a certain future time domain.

2. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: The pre-trained image feature extraction network is the ResNet-50 network.

3. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: In step S3, a certain number of tracking sequences are first randomly initialized to represent all observed moving targets in the scene, and then a multi-layer perceptron is used to decode the tracking sequences to generate corresponding 3D reference points representing the spatial positions of the targets.

4. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: In step S5, two multi-layer perceptron networks are specifically used to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that characterize the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

5. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: The tracking sequence eliminated in step S6 no longer participates in the tracking and trajectory prediction at the next moment.

6. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 1 is characterized in that: Step S7 is specifically as follows: After accumulating the multi-target tracking sequence of a certain observation time, the tracking sequence of a single target of a certain observation time is encoded using the long short-term memory network to obtain the encoding vector, and further obtain the hidden state vector; The hidden state vectors of multiple moving targets at the previous moment are accumulated, and the hidden state vectors of other targets are combined to generate the influence tensor at time t. The influence tensor implicitly contains the spatial interaction features between targets and is input into the next layer of LSTM module together with the encoding vector at time t. The hidden state output by the last layer of LSTM modules is obtained, mapped into a one-dimensional ternary array, and the coordinates at time t+1 are predicted using a two-dimensional Gaussian distribution, and the target prediction trajectory in a certain future time domain is further iterated.

7. The pedestrian trajectory prediction method based on multi-camera BEV view feature fusion based on attention mechanism according to claim 6 is characterized in that: The three-element array includes the Gaussian distribution mean, standard deviation, and correlation coefficient.

8. A pedestrian trajectory prediction system based on multi-camera BEV view feature fusion based on attention mechanism, characterized in that: include: The multi-view image acquisition module is used to arrange M cameras at different positions of a specified scene to perform bird's-eye view global perception and obtain multi-view images; The calibration module is used to calibrate the images and align the timestamps of the M cameras, and obtain the homogeneous projection matrix from the spatial points to the image pixels in a unified reference system; A multi-scale feature map acquisition module is used to input multi-view images in parallel into a pre-trained image feature extraction network to extract high-dimensional semantic features of low-dimensional image features, and output a multi-scale feature map using a feature pyramid network; The multi-target tracking module is used to track multi-target motion on multi-scale feature maps, generate target tracking sequences and 3D reference points representing the spatial positions of corresponding targets; according to the camera extrinsic projection matrix, the 3D reference points are projected onto the multi-view images of the corresponding camera to obtain a set of 2D sampling points, and the 2D sampling point features are aggregated using the attention mechanism to update the tracking sequence and 3D reference points; the updated tracking sequence is decoded to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, target orientation information, and target category corresponding to the tracking sequence that represent the target motion characteristics; A pedestrian target screening module is used to screen pedestrian target categories according to a preset confidence threshold, and retain the tracking sequence above the preset confidence threshold and add it to the next moment tracking sequence set; Otherwise, it will be eliminated; The trajectory prediction module is used to perform trajectory prediction based on the tracking sequence set and generate the target prediction trajectory in a certain future time domain.

9. The pedestrian trajectory prediction system based on multi-camera BEV view feature fusion based on attention mechanism according to claim 8 is characterized in that: The multi-target tracking module specifically uses two multi-layer perceptron networks to decode the updated tracking sequence respectively. The first multi-layer perceptron network decodes to obtain the target 3D prediction box center point, 3D prediction box size, target speed in the pixel coordinate system, and target orientation information that represent the target motion characteristics. The other multi-layer perceptron network decodes to obtain the target category corresponding to the tracking sequence.

10. A computer storage medium, characterized in that: A computer program executable by a processor is stored therein, and the computer program executes the pedestrian trajectory prediction method based on multi-camera BEV perspective feature fusion based on the attention mechanism as described in any one of claims 1-7.

Images