A pedestrian trajectory prediction method and device, electronic equipment and storage medium

Abstract

The application discloses a kind of pedestrian trajectory prediction method, device, equipment and medium.The method comprises: obtaining the input data of behavior prediction model;Wherein, the input data includes to be predicted pedestrian data and environmental data;According to input data and pre-trained behavior prediction model, the movement intention and movement orientation angle of to-be-predicted pedestrian are determined;According to movement intention, movement orientation angle and pre-determined movement speed of to-be-predicted pedestrian, the trajectory prediction result of to-be-predicted pedestrian is determined.The technical scheme solves the problems, such as low accuracy of pedestrian trajectory prediction and poor reliability, can improve the accuracy of pedestrian trajectory prediction while enhancing the explainability of pedestrian trajectory generation, guarantee the safety of pedestrian and autonomous vehicle.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, apparatus, device, and medium for predicting pedestrian trajectories. Background Technology

[0002] In the field of autonomous driving, accurately predicting the impact of pedestrian movement on autonomous vehicles in the driving environment is of great significance. The core task of pedestrian trajectory prediction is to extract features from the historical movement trajectory of pedestrians through trajectory prediction models, and then predict the future movement trajectory of pedestrians based on the extracted movement trajectory features. Pedestrian trajectory prediction faces many challenges: (1) Compared with obstacles such as motor vehicles, pedestrians move more subjectively and freely, and are not restricted by road lane lines. (2) Pedestrian movement is not only related to their own historical trajectory, but also to the state of surrounding pedestrians or vehicles. (3) Pedestrian movement is also constrained by the surrounding physical environment, and the obstruction effect of surrounding buildings, etc., needs to be considered when predicting pedestrian trajectory.

[0003] Existing technologies for pedestrian trajectory prediction primarily define it as a sequence generation problem—that is, how to predict future locations based on historical location information. These methods rely heavily on pedestrians' historical location perception information, neglecting behavioral information such as orientation and posture. Therefore, there is an urgent need to design a pedestrian trajectory prediction scheme that considers pedestrian behavioral information to improve the reliability and accuracy of trajectory prediction. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for predicting pedestrian trajectories to solve problems such as low accuracy and poor reliability in pedestrian trajectory prediction. It can improve the accuracy of pedestrian trajectory prediction while enhancing the interpretability of pedestrian trajectory generation, thus ensuring the safety of pedestrians and autonomous vehicles.

[0005] According to one aspect of the present invention, a method for predicting pedestrian trajectories is provided, the method comprising:

[0006] Obtain input data for the behavior prediction model; wherein, the input data includes pedestrian data to be predicted and environmental data;

[0007] Based on the input data and the pre-trained behavior prediction model, determine the movement intention and movement direction angle of the pedestrian to be predicted;

[0008] Based on the movement intention, the movement orientation angle, and the pre-determined movement speed of the pedestrian to be predicted, the trajectory prediction result of the pedestrian to be predicted is determined.

[0009] According to another aspect of the present invention, a pedestrian trajectory prediction device is provided, the device comprising:

[0010] The input data acquisition module is used to acquire input data for the behavior prediction model; wherein, the input data includes pedestrian data to be predicted and environmental data;

[0011] The intent and orientation angle determination module is used to determine the movement intent and orientation angle of the pedestrian to be predicted based on the input data and the pre-trained behavior prediction model.

[0012] The trajectory prediction result determination module is used to determine the trajectory prediction result of the pedestrian to be predicted based on the movement intention, movement orientation angle, and the pre-determined movement speed of the pedestrian to be predicted.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the pedestrian trajectory prediction method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the pedestrian trajectory prediction method according to any embodiment of the present invention.

[0018] The technical solution of this invention acquires input data from a behavior prediction model, and determines the movement intention and orientation angle of the pedestrian to be predicted based on the input data and the pre-trained behavior prediction model. Then, based on the movement intention, orientation angle, and a pre-determined movement speed of the pedestrian to be predicted, the trajectory prediction result of the pedestrian is determined. This solution solves the problems of low accuracy and poor reliability in pedestrian trajectory prediction, and can improve the accuracy of pedestrian trajectory prediction while enhancing the interpretability of pedestrian trajectory generation, thus ensuring the safety of pedestrians and autonomous vehicles.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1A This is a flowchart of a pedestrian trajectory prediction method provided according to Embodiment 1 of the present invention;

[0022] Figure 1B This is a schematic diagram of the current location of the pedestrian to be predicted according to Embodiment 1 of the present invention;

[0023] Figure 1C This is a schematic diagram of the historical trajectory of the pedestrian to be predicted according to Embodiment 1 of the present invention;

[0024] Figure 1D This is a schematic diagram of the location of the obstacle provided in Embodiment 1 of the present invention;

[0025] Figure 1E This is a schematic diagram of a road configuration provided in Embodiment 1 of the present invention;

[0026] Figure 1F This is a schematic diagram of road traffic rights provided according to Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart of a pedestrian trajectory prediction method provided according to Embodiment 2 of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a pedestrian trajectory prediction device provided in Embodiment 3 of the present invention;

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the pedestrian trajectory prediction method of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0032] Example 1

[0033] Figure 1A This is a flowchart illustrating a method for predicting pedestrian trajectories according to Embodiment 1 of the present invention. This embodiment is applicable to pedestrian trajectory prediction in autonomous driving scenarios. The method can be executed by a pedestrian trajectory prediction device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1A As shown, the method includes:

[0034] S110. Obtain the input data for the behavior prediction model.

[0035] This solution can be executed by autonomous vehicles, which can acquire input data for the behavior prediction model through sensors such as radar and cameras. The input data may include data on the pedestrian to be predicted and environmental data. The pedestrian data may include the pedestrian's current position, direction of movement, speed, and acceleration. The environmental data may include data related to the road, road conditions, and obstacles associated with the pedestrian.

[0036] Autonomous vehicles can predict pedestrian trajectories according to a preset cycle. Based on the input data of the behavior prediction model obtained in the current cycle, they can predict the pedestrian's movement trajectory in the next cycle.

[0037] In this scheme, optionally, the pedestrian data to be predicted includes the location data of the pedestrian to be predicted at the current time, and the location data of the pedestrian to be predicted at least two times within the current period; the environmental data includes the location data of obstacles within a preset range at least two times within the current period, and the road data associated with the pedestrian to be predicted within the current period; wherein, the location data is determined based on the target detection results of the driving environment image at the target time; the road data includes road type, road reference lines, and road access permissions at least two times.

[0038] To ensure a detailed depiction of traffic scenarios, location and road data can be image data. Autonomous vehicles can mark the current location of pedestrians to be predicted in the images. Figure 1B This is a schematic diagram of the current location of the pedestrian to be predicted according to Embodiment 1 of the present invention, as shown below. Figure 1B As shown, the white rectangle represents the pedestrian to be predicted. The location data of the pedestrian to be predicted can be determined based on the pedestrian detection results in the driving environment image at the current moment. Based on the location data of the pedestrian to be predicted at least two historical moments within the current period, the autonomous vehicle can determine the historical trajectory of the pedestrian to be predicted. Figure 1C This is a schematic diagram of the historical trajectory of the pedestrian to be predicted according to Embodiment 1 of the present invention, such as... Figure 1C As shown, the historical trajectory of the pedestrian to be predicted can be obtained by connecting the positions of the pedestrian to be predicted at least two historical moments within the current period.

[0039] Autonomous vehicles can determine the positions of obstacles within a preset range in the current period at at least two moments based on the location of the pedestrian to be predicted, and mark them in the driving environment image. Figure 1D This is a schematic diagram of the location of an obstacle according to Embodiment 1 of the present invention, as shown below. Figure 1D As shown, each time point corresponds to a schematic diagram of the obstacle's location. The obstacle's location data can be determined based on obstacle detection results in the driving environment image at that time. The autonomous vehicle can acquire road data associated with the pedestrian to be predicted in the current cycle, such as road data of the path the pedestrian may traverse in the current cycle. Figure 1E This is a schematic diagram of a road configuration provided according to Embodiment 1 of the present invention, such as... Figure 1E As shown, autonomous vehicles can acquire position data of road edges and reference lines to generate road morphology images, which are used to characterize road types and lane divisions. Figure 1F This is a schematic diagram of road traffic rights provided according to Embodiment 1 of the present invention, as shown below. Figure 1F As shown, each moment corresponds to a road access permission diagram, and autonomous vehicles can use curves, arrows, and colors to indicate the road access permission at that moment.

[0040] This solution can generate pedestrian data and environmental data from multiple dimensions, which is beneficial for accurate prediction of pedestrian trajectories and ensures the reliability and stability of pedestrian trajectory prediction.

[0041] S120. Based on the input data and the pre-trained behavior prediction model, determine the movement intention and movement direction angle of the pedestrian to be predicted.

[0042] Autonomous vehicles can feed input data into a pre-trained behavior prediction model to obtain the pedestrian's movement intention and orientation angle. This behavior prediction model can be built based on deep learning algorithms. In a specific example, the input data may include, for instance... Figures 1B-1F The image data shown can be used to construct an action prediction model based on a convolutional neural network. This model can include five network input branches, a feature fusion structure, and two network output branches. The five network input branches are used to extract features such as... Figures 1B-1F The five image categories shown have their own features. Two network output branches are used to output the pedestrian's motion intention and orientation angle, respectively. The feature fusion structure fuses the features of the five image categories and passes them to the two network output branches for predicting motion intention and orientation angle.

[0043] Based on the above scheme, the step of determining the movement intention and movement orientation angle of the pedestrian to be predicted according to the input data and the pre-trained behavior prediction model includes:

[0044] The input data is fed into a pre-trained behavior prediction model to determine the movement intention and direction angle of the pedestrian to be predicted at the current moment; wherein, the movement intention includes going straight, turning left, turning right, moving backward, and stopping.

[0045] Compared to simply focusing on the historical trajectory of the pedestrian to be predicted, determining the pedestrian's movement intention and direction angle based on both pedestrian data and environmental data can more accurately predict the pedestrian's trajectory, ensuring both pedestrian safety and the driving safety of autonomous vehicles. Movement intention can include states such as going straight, turning left, turning right, reversing, and stopping, which facilitates finer-grained movement intention segmentation, thereby improving the accuracy of pedestrian trajectory prediction.

[0046] In one feasible approach, the behavior prediction model includes an encoder and a decoder; wherein the encoder is constructed based on an attention mechanism structure; and the decoder includes a motion intention prediction branch and a motion orientation angle prediction branch.

[0047] Understandably, the encoder extracts features from the pedestrian data and environmental data, and fuses these features using an attention mechanism to obtain fused features. The decoder's motion intent prediction branch determines the pedestrian's motion intent based on the fused features. The motion orientation angle prediction branch determines the pedestrian's motion orientation angle based on the fused features. It's important to note that the network structure and parameters of the motion intent prediction and motion orientation angle prediction branches can be the same or different. For example, the motion intent prediction branch can use a classification loss function to evaluate the decoder's output motion intent, while the motion orientation angle prediction branch can select a loss function that fits the scene requirements from the regression loss to evaluate the decoder's output motion orientation angle.

[0048] S130. Based on the movement intention, movement orientation angle, and the predetermined movement speed of the pedestrian to be predicted, determine the trajectory prediction result of the pedestrian to be predicted.

[0049] Understandably, the speed of the pedestrian to be predicted can be obtained through the speed sensors of the autonomous vehicle, or it can be calculated based on the pedestrian's position data at least two moments within the current period. Based on the pedestrian's current movement intention and orientation angle, the autonomous vehicle can determine the pedestrian's direction of movement. Based on the direction and speed of movement, the autonomous vehicle can generate the pedestrian's trajectory, thus obtaining the predicted trajectory result.

[0050] Optionally, in this scheme, determining the trajectory prediction result of the pedestrian to be predicted based on the movement intention, the movement orientation angle, and the pre-determined movement speed of the pedestrian to be predicted includes:

[0051] Based on the predetermined speed of the pedestrian to be predicted, the pedestrian's current movement intention and direction angle, the trajectory prediction result of the pedestrian in the current cycle is determined based on the motion model.

[0052] In this scheme, the motion model may include models such as the Dynamic Window Approach (DWA), Curvature Velocity Method (CVM), Probabilistic Road Map (PRM), and Rapidly-exploring Random Tree (RRT). The autonomous vehicle can calculate the trajectory prediction result for the current cycle of the pedestrian based on the motion model, considering the pedestrian's speed, the pedestrian's current movement intention, and the direction angle.

[0053] This technical solution acquires input data from a behavior prediction model, and based on the input data and the pre-trained behavior prediction model, determines the movement intention and orientation angle of the pedestrian to be predicted. Then, based on the movement intention, orientation angle, and the pre-determined movement speed of the pedestrian to be predicted, the trajectory prediction result is determined. This solution solves the problems of low accuracy and poor reliability in pedestrian trajectory prediction, improving the accuracy of pedestrian trajectory prediction while enhancing the interpretability of pedestrian trajectory generation, thus ensuring the safety of pedestrians and autonomous vehicles.

[0054] Example 2

[0055] Figure 2 This is a flowchart of a pedestrian trajectory prediction method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and refines the process of determining the trajectory prediction result of the pedestrian in the current period. For example... Figure 2 As shown, the method includes:

[0056] S210. Obtain the input data for the behavior prediction model.

[0057] S220. Input the input data into the pre-trained behavior prediction model to determine the movement intention and movement orientation angle of the pedestrian to be predicted at the current moment.

[0058] S230. Determine the movement speed of the pedestrian to be predicted based on the position data of the pedestrian at least two times within the current period.

[0059] Autonomous vehicles can acquire location data from at least two predetermined moments within the current period, and use bitwise operations divided by time to calculate the predicted speed of a pedestrian. For example, an autonomous vehicle can calculate the pedestrian's displacement based on the current and previous moment's location data, and also calculate the time difference between the current and previous moments. Based on the displacement and time difference, the predicted speed of the pedestrian is determined.

[0060] S240. Based on the movement speed of the pedestrian to be predicted, the current position data of the pedestrian to be predicted, the movement intention matched by the pedestrian to be predicted at the current moment, and the movement orientation angle, determine the position data of the pedestrian to be predicted at the next moment.

[0061] Understandably, based on the current location data of the pedestrian to be predicted, and according to the pedestrian's speed, the pedestrian's current movement intention, and the direction angle of movement, the autonomous vehicle can determine the location data of the pedestrian to be predicted at the next moment.

[0062] S250. If the next moment is within the current cycle, then the next moment is determined as the current moment, and the process returns to obtain the input data of the behavior prediction model until the next moment is not in the current cycle. Then, the trajectory prediction result of the pedestrian to be predicted in the current cycle is output.

[0063] In this scheme, each period can include multiple moments, and the time intervals between these moments can be the same or different. The trajectory prediction result for one period is generated based on the position data of the pedestrian to be predicted at each moment within that period. If the next moment is not within the current period, it means that the next moment is the moment of the period following the current period, and the complete motion trajectory of the pedestrian to be predicted in the current period has been generated. Based on the position data of the pedestrian to be predicted at each moment in the current period, the trajectory prediction result for the current period can be obtained. If the next moment is within the current period, the position data of the pedestrian to be predicted at the next moment can be added to the already determined motion trajectory of the current period, and the next moment can be used as the current moment to continue determining the position data of the pedestrian to be predicted at the next moment after the current moment, until the next moment exceeds the current period, and the trajectory prediction result of the pedestrian to be predicted in the current period is output.

[0064] In a feasible solution, the steps for pedestrian trajectory prediction are as follows:

[0065] Step 1: Determine the intention of the movement and the direction of the movement.

[0066] Step 1.1: For the positional data in the input data Feature extraction is performed using convolutional structures and Feature Pyramid Networks (FPN) to obtain features. obs As shown in the formula below:

[0067]

[0068] Where T represents a prediction period. It includes the location data of pedestrians to be predicted within a period and the location data of obstacles within a preset range.

[0069] Step 1.2: For the road data I in the input data T Features are extracted using convolutional structures to obtain the feature. map The feature extraction process can be represented as: feature map =CNN(I T ).

[0070] Step 1.3: Based on the fusion network, extract the features of the road data. map Features fused into location data obsIn the process, the fusion feature is obtained. fus The feature fusion process can be represented as: feature fus =FusionNet(feature) obs feature map ).

[0071] Specifically, the refinement process of feature fusion can include:

[0072] Step 1.3.1: Based on the attention mechanism, fuse features obs To feature map In, it is represented as:

[0073]

[0074] Where i represents the feature index of the road data, j represents the feature index of the location data, and W0, W1, and W2 are the feature indices, respectively. as well as Matching weight, Δ i,j =MLP(v i ,v j ), v i This indicates the location of the first node, which is a road location selected from the road data according to preset rules. j The second node position is the target position selected from the location data of pedestrians and obstacles to be predicted according to preset rules. MLP (Multi-Layer Perceptron) represents a multi-layer perceptron.

[0075] Step 1.3.2: Based on the extended convolutional structure, fuse various... The calculation process is expressed as follows:

[0076]

[0077] Step 1.3.3: Utilize the attention mechanism to extract the features from the road data. map Features fused into location data obs In this context, the fusion process can be represented as:

[0078]

[0079] Where i represents the feature index of the road data, j represents the feature index of the location data, and W3, W4, and W5 are the feature indices, respectively. as well as Matching weight, Δ i,j =MLP(v i ,v j), v i This indicates the location of the first node, which is a road location selected from the road data according to preset rules. j The second node position is the target position selected from the location data of pedestrians and obstacles to be predicted according to preset rules. MLP (Multi-Layer Perceptron) represents a multi-layer perceptron.

[0080] Step 1.3.4: Based on the attention mechanism, [the following steps are performed]... To merge, that is:

[0081]

[0082] Where j and k both represent feature indices of the location data, Δ k,j =MLP(v k ,v j ), v k ,v j All represent the second node position, which is the target position selected from the location data of pedestrians and obstacles to be predicted according to preset rules. MLP stands for Multilayer Perceptron; W6, W7, and W8 are features. as well as Matching weights.

[0083] Step 1.3.5: Determine the motion intention and motion orientation angle, using the ReLU function as the activation function, as shown:

[0084]

[0085] Here, intent represents the intention to move, and θ represents the direction angle of the movement.

[0086] It should be noted that during the training of the behavior prediction model, the regression error of the motion orientation angle is calculated using the smooth-L1 loss, as shown in the following formula. The classification error of the motion intent is calculated using the cross-entropy loss. The smooth-L1 loss is expressed as:

[0087]

[0088] Where, θ pre θ represents the direction of motion. gt The orientation angle label is used to represent the cross-entropy loss L, which is expressed as: L = -[ylogy] 0 +(1-y)log(1-y 0 ); where y 0 y represents the intention of the movement.

[0089] Step 2: Determine the trajectory prediction results.

[0090] Step 2.1: Calculate the obstacle's velocity v based on the position data of the pedestrian to be predicted at least two times within the current period. agent The calculation formula is expressed as:

[0091]

[0092] in, This indicates the current position coordinates of the pedestrian to be predicted; This represents the position coordinates of the pedestrian to be predicted at the previous moment; This indicates the displacement of the pedestrian to be predicted between the current moment and the previous moment; It can be represented as It can be represented as The calculation formula can be expressed as:

[0093]

[0094] Step 2.2: Based on the predicted pedestrian's speed v agent Given the motion orientation angle θ and the motion intention, calculate the next time step t. f Position coordinates And save the trajectory points to the motion trajectory traj that has been determined in the current cycle. and The calculation formulas are as follows:

[0095]

[0096]

[0097] Step 2.3: Determine t f If tf is in the prediction period T, proceed to step 2.2; otherwise, proceed to step 3.

[0098] Step 3: Output the trajectory prediction result traj.

[0099] This scheme breaks down the pedestrian prediction problem into two parts: intent prediction and trajectory generation, making the trajectory generation process more interpretable. Simultaneously, it combines learning-based and motion-based methods to overcome the limitation of learning-based trajectory prediction methods in accurately conforming to kinematic laws. When predicting pedestrian intent, it utilizes surrounding environmental information and pedestrian motion information, and employs multi-scale convolutional operations to extract various types of features, improving the accuracy of pedestrian intent prediction and providing a good reference for pedestrian trajectory generation.

[0100] This technical solution acquires input data from a behavior prediction model, and based on the input data and the pre-trained behavior prediction model, determines the movement intention and orientation angle of the pedestrian to be predicted. Then, based on the movement intention, orientation angle, and the pre-determined movement speed of the pedestrian to be predicted, the trajectory prediction result is determined. This solution solves the problems of low accuracy and poor reliability in pedestrian trajectory prediction, improving the accuracy of pedestrian trajectory prediction while enhancing the interpretability of pedestrian trajectory generation, thus ensuring the safety of pedestrians and autonomous vehicles.

[0101] Example 3

[0102] Figure 3 This is a schematic diagram of the structure of a pedestrian trajectory prediction device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0103] The input data acquisition module 310 is used to acquire input data for the behavior prediction model; wherein, the input data includes pedestrian data to be predicted and environmental data;

[0104] The intent and orientation angle determination module 320 is used to determine the movement intent and orientation angle of the pedestrian to be predicted based on the input data and the pre-trained behavior prediction model.

[0105] The trajectory prediction result determination module 330 is used to determine the trajectory prediction result of the pedestrian to be predicted based on the movement intention, movement orientation angle and the pre-determined movement speed of the pedestrian to be predicted.

[0106] In this scheme, optionally, the pedestrian data to be predicted includes the location data of the pedestrian to be predicted at the current time, and the location data of the pedestrian to be predicted at least two times within the current period; the environmental data includes the location data of obstacles within a preset range at least two times within the current period, and the road data associated with the pedestrian to be predicted within the current period; wherein, the location data is determined based on the target detection results of the driving environment image at the target time; the road data includes road type, road reference lines, and road access permissions at least two times.

[0107] Based on the above solution, optionally, the intent and orientation angle determination module 320 is specifically used for:

[0108] The input data is fed into a pre-trained behavior prediction model to determine the movement intention and direction angle of the pedestrian to be predicted at the current moment; wherein, the movement intention includes going straight, turning left, turning right, moving backward, and stopping.

[0109] In one feasible approach, the behavior prediction model includes an encoder and a decoder; wherein the encoder is constructed based on an attention mechanism structure; and the decoder includes a motion intention prediction branch and a motion orientation angle prediction branch.

[0110] In this embodiment, optionally, the trajectory prediction result determination module 330 is specifically used for:

[0111] Based on the predetermined speed of the pedestrian to be predicted, the pedestrian's current movement intention and direction angle, the trajectory prediction result of the pedestrian in the current cycle is determined based on the motion model.

[0112] Based on the above scheme, optionally, the device further includes a motion speed determination module, used to determine the motion speed of the pedestrian to be predicted based on the position data of the pedestrian to be predicted at least two times within the current period;

[0113] The trajectory prediction result determination module 330 is specifically used for:

[0114] Based on the predetermined speed of the pedestrian to be predicted, the pedestrian's current movement intention and direction angle, the trajectory prediction result of the pedestrian in the current cycle is determined based on the curvature velocity model.

[0115] In a preferred embodiment, the trajectory prediction result determination module 330 is specifically used for:

[0116] Based on the pedestrian's movement speed, the pedestrian's current position data, the matching movement intention and movement direction angle at the current moment, the position data of the pedestrian to be predicted at the next moment is determined;

[0117] If the next moment is within the current cycle, then the next moment is determined as the current moment, and the process returns to obtain the input data for the behavior prediction model. This continues until the next moment is no longer within the current cycle, at which point the trajectory prediction result for the pedestrian in the current cycle is output.

[0118] The pedestrian trajectory prediction device provided in this embodiment of the invention can execute the pedestrian trajectory prediction method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0119] Example 4

[0120] Figure 4A schematic diagram of an electronic device 410 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0121] like Figure 4 As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0122] Multiple components in electronic device 410 are connected to I / O interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of displays, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0123] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as methods for predicting pedestrian trajectories.

[0124] In some embodiments, the pedestrian trajectory prediction method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded into and / or mounted on electronic device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the pedestrian trajectory prediction method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured to perform the pedestrian trajectory prediction method by any other suitable means (e.g., by means of firmware).

[0125] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0129] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0130] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0131] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0132] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for predicting pedestrian trajectories, characterized in that, The method includes: The input data for the behavior prediction model is obtained; wherein, the input data includes pedestrian data to be predicted and environmental data; the pedestrian data to be predicted includes the location data of the pedestrian to be predicted at the current time, and the location data of the pedestrian to be predicted at least two times within the current period; the environmental data includes the location data of obstacles within a preset range within the current period at at least two times, and the road data associated with the pedestrian to be predicted within the current period; the location data is a location image determined based on the target detection results of the driving environment image at the target time; the road data includes road type, road reference lines, and road access permissions at least two times; Input data is fed into a pre-trained behavior prediction model to determine the movement intention and orientation angle of the pedestrian to be predicted at the current moment. The movement intention includes going straight, turning left, turning right, moving backward, and stopping. The behavior prediction model includes five network input branches, a feature fusion structure, and two network output branches. The five network input branches are used to extract features from the first to fifth images: the first image is the position image of the pedestrian to be predicted at the current moment; the second image is the historical trajectory image of the pedestrian to be predicted, generated based on the position data of the pedestrian to be predicted at least two moments within the current period; the third image is the position image of obstacles within a preset range at various moments within the current period; the fourth image is the road morphology image generated based on the road type and road reference line associated with the pedestrian to be predicted within the current period; and the fifth image is the road access permission image generated based on the road access permission associated with the pedestrian to be predicted at various moments within the current period. The two network output branches are used to output the movement intention and orientation angle of the pedestrian to be predicted. The feature fusion structure is used to fuse the features of the first to fifth images and pass them to the two network output branches. The position data of the pedestrian to be predicted at the next moment is determined based on the pedestrian's movement speed, the pedestrian's current position data, the matching movement intention of the pedestrian at the current moment, and the movement orientation angle; the pedestrian's movement speed is determined after acquiring the input data of the behavior prediction model, based on the position data of the pedestrian to be predicted at at least two moments within the current period. If the next moment is within the current cycle, then the next moment is determined as the current moment, and the process returns to obtain the input data for the behavior prediction model. This continues until the next moment is no longer within the current cycle, at which point the trajectory prediction result for the pedestrian in the current cycle is output.

2. A pedestrian trajectory prediction device, characterized in that, The device includes: The input data acquisition module is used to acquire input data for the behavior prediction model. The input data includes pedestrian data to be predicted and environmental data. The pedestrian data to be predicted includes the current location data of the pedestrian to be predicted, and the location data of the pedestrian to be predicted at least two times within the current period. The environmental data includes the location data of obstacles within a preset range at least two times within the current period, and the road data associated with the pedestrian to be predicted within the current period. The location data is a location image determined based on target detection results from the driving environment image at the target time. The road data includes road type, road reference lines, and road access permissions at least two times. An intent and orientation angle determination module is used to input input data into a pre-trained behavior prediction model to determine the movement intent and orientation angle of the pedestrian to be predicted at the current moment. The movement intent includes going straight, turning left, turning right, moving backward, and stopping. The behavior prediction model includes five network input branches, a feature fusion structure, and two network output branches. The five network input branches are used to extract features from the first to fifth images: the first image is the position image of the pedestrian to be predicted at the current moment; the second image is the historical trajectory image of the pedestrian to be predicted, generated based on the position data of the pedestrian to be predicted at least two moments within the current period; the third image is the position image of obstacles within a preset range at various moments within the current period; the fourth image is the road morphology image generated based on the road type and road reference line associated with the pedestrian to be predicted within the current period; and the fifth image is the road access permission image generated based on the road access permission associated with the pedestrian to be predicted at various moments within the current period. The two network output branches are used to output the movement intent and orientation angle of the pedestrian to be predicted. The feature fusion structure is used to fuse the features of the first to fifth images and pass them to the two network output branches. The trajectory prediction result determination module is used to determine the position data of the pedestrian to be predicted at the next moment based on the pedestrian's movement speed, the pedestrian's current position data, the matching movement intention of the pedestrian at the current moment, and the movement orientation angle. The movement speed of the pedestrian to be predicted is determined based on the position data of the pedestrian to be predicted at at least two moments within the current period after acquiring the input data of the behavior prediction model. If the next moment is within the current period, the next moment is determined as the current moment, and the process returns to acquire the input data of the behavior prediction model until the next moment is not within the current period, at which point the trajectory prediction result of the pedestrian to be predicted for the current period is output.

3. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the pedestrian trajectory prediction method of claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the pedestrian trajectory prediction method of claim 1.

Images