A key point-based pedestrian trajectory prediction method and device

Abstract

The application discloses a kind of pedestrian trajectory prediction method and device based on key point, it is related to automatic driving technical field.The specific embodiment of the method includes: extracting the key point feature of pedestrian from pedestrian key point detection information;From multiple historical images, extract multiple sets of historical features;The historical features include: the position of the pedestrian, the speed of the pedestrian and the key point feature;Based on the multiple sets of historical features, generate the fusion feature of the pedestrian;From lane information, extract the feature of lane;Based on self-attention mechanism, fuse the fusion feature of the pedestrian and the feature of lane, obtain pedestrian lane fusion feature;Based on the pedestrian lane fusion feature, determine whether the pedestrian is stationary or moving, if it is moving, then the pedestrian lane fusion feature is input into decoder, obtain the trajectory of the pedestrian.This embodiment can more accurately predict the trajectory of pedestrian.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and apparatus for predicting pedestrian trajectories based on key points. Background Technology

[0002] Pedestrian intent prediction is a crucial aspect of ensuring the safety and comfort of autonomous driving. When driving on urban roads, it is inevitable that vehicles and pedestrians will share the same road. The main characteristics of pedestrians participating in road traffic are their highly unpredictable walking and changing directions. If pedestrian intent is not accurately predicted, it may cause autonomous vehicles to make incorrect decisions, leading to collisions between vehicles and pedestrians.

[0003] In existing technologies, the focus of prediction technology for autonomous driving is on the prediction of fast-moving obstacles such as vehicles and non-motorized vehicles, but there are few studies specifically for pedestrian trajectory prediction. Although there are many studies on pedestrian prediction in academic research, these studies mainly focus on information obtained by fixed perception systems, and the effect is poor when transferred to mobile systems such as autonomous driving. Summary of the Invention

[0004] In view of the above, the present invention provides a method and apparatus for predicting pedestrian trajectories based on key points, so as to achieve effective prediction of pedestrian trajectories.

[0005] Firstly, this specification provides a pedestrian trajectory prediction method based on key points, including:

[0006] Extract pedestrian key point features from pedestrian key point detection information;

[0007] Multiple sets of historical features are extracted from multiple frames of historical images; the historical features include: the position of the pedestrian, the speed of the pedestrian, and the key point features;

[0008] Based on the aforementioned multiple sets of historical features, the fusion features of the pedestrian are generated;

[0009] Extract lane features from lane information;

[0010] Based on a self-attention mechanism, the fusion features of the pedestrian and the features of the lane are fused to obtain pedestrian-lane fusion features;

[0011] Based on the pedestrian lane fusion features, it is determined whether the pedestrian is stationary or moving. If the pedestrian is moving, the pedestrian lane fusion features are input into the decoder to obtain the pedestrian's trajectory.

[0012] Secondly, this specification provides a pedestrian trajectory prediction device based on key points, including:

[0013] The first extraction module is configured to extract key point features of pedestrians from pedestrian key point detection information; and extract multiple sets of historical features from multiple frames of historical images; the historical features include: the position of the pedestrian, the speed of the pedestrian, and the key point features.

[0014] The fusion module is configured to generate the fused features of the pedestrian based on the multiple sets of historical features;

[0015] The second extraction module is configured to extract lane features from lane information;

[0016] The self-attention mechanism module is configured to fuse the pedestrian's fusion features and the lane's features based on the self-attention mechanism to obtain pedestrian-lane fusion features;

[0017] The decoding module is configured to determine whether the pedestrian is stationary or moving based on the pedestrian lane fusion features. If the pedestrian is moving, the pedestrian lane fusion features are input into the decoder to obtain the pedestrian's trajectory.

[0018] Thirdly, embodiments of this specification provide an electronic device, including:

[0019] One or more processors;

[0020] Storage device for storing one or more programs.

[0021] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any of the above embodiments.

[0022] Fourthly, embodiments of the present invention provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above embodiments.

[0023] One embodiment of the above invention has the following advantages or beneficial effects: the method focuses on the influence of pedestrian key point detection information on future behavior in order to obtain more comprehensive and effective information about pedestrians. Combined with the self-attention mechanism, the autonomous vehicle can pre-determine whether the pedestrian is in a stationary or moving mode. If the pedestrian is determined to be in a stationary mode, the program exits. If the pedestrian is determined to be in a moving mode, the future trajectory of the pedestrian is further output. This method enables the autonomous vehicle to accurately predict the future behavior of pedestrians and adjust the travel route in time to avoid collisions. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below, constituting a part of the present invention. The illustrative embodiments of the present invention and their descriptions explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0025] Figure 1 A flowchart illustrating a pedestrian trajectory prediction method based on key points provided in one embodiment of the present invention;

[0026] Figure 2 A flowchart of a pedestrian trajectory prediction method based on key points provided in another embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a pedestrian trajectory prediction device based on key points provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0029] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] As key participants in traffic scenarios, the accurate reasoning and prediction of pedestrians' future trajectories are crucial for autonomous driving and road safety. Existing pedestrian trajectory prediction methods suffer from the following drawbacks:

[0031] (1) The impact of key points on the future intentions of pedestrians is not taken into account. The actions of pedestrians can indicate many intentions. They are not just simple points. Therefore, it is not enough to only consider information such as position and speed in pedestrian characteristics.

[0032] (2) Existing pedestrian trajectory prediction methods mainly predict moving obstacles, but do not predict stationary obstacles, which will lead to autonomous vehicles not reacting in time to pedestrians who suddenly start moving.

[0033] All of the above shortcomings will cause existing pedestrian trajectory prediction to be unable to make accurate predictions.

[0034] In view of this, according to Figure 1 This specification provides a method for predicting pedestrian trajectories based on key points, including:

[0035] Step 100: Extract pedestrian key point features from pedestrian key point detection information.

[0036] Step 101: Extract multiple sets of historical features from multiple frames of historical images; the historical features include: pedestrian position, pedestrian speed, and key point features.

[0037] Autonomous driving is a highly complex system engineering project composed of multiple sensors and subsystems, including LiDAR, millimeter-wave radar, cameras, ultrasonic radar, and GPS. In this specification, pedestrian key point detection information is mainly obtained through cameras. Other historical features, such as pedestrian position and speed, are mainly obtained through perception systems such as LiDAR, ultrasonic radar, and millimeter-wave radar, but are not limited to these.

[0038] The task of pedestrian keypoint detection is to detect the human body and its corresponding keypoint locations from the input image. Keypoint features include, but are not limited to, the nose, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0039] The historical features in this specification include, but are not limited to, pedestrian location, pedestrian speed, and key point features.

[0040] Step 102: Generate pedestrian fusion features based on multiple sets of historical features.

[0041] Step 103: Extract lane features from lane information.

[0042] Step 104: Based on the self-attention mechanism, the pedestrian fusion features and lane features are fused to obtain pedestrian lane fusion features.

[0043] Step 105: Determine whether the pedestrian is stationary or moving based on the pedestrian lane fusion features. If the pedestrian is moving, input the pedestrian lane fusion features into the decoder to obtain the pedestrian's trajectory.

[0044] This specification addresses the problem of poor trajectory prediction performance in existing pedestrian trajectory prediction technologies that do not consider key human body points. It proposes a key point-based pedestrian trajectory prediction method. First, pedestrian key point information in both intersection and non-intersection scenarios is used as the feature input for pedestrians. This key point feature is fused with multiple sets of historical features to obtain more detailed and effective pedestrian information. Second, a self-attention mechanism is used to fuse pedestrian and lane features to obtain the correlation between them. Finally, the correlation between lane and pedestrian features is used to determine whether the pedestrian is moving. If movement is detected, the predicted pedestrian trajectory is output. This method emphasizes the influence of pedestrian key point detection information on future behavior to obtain more comprehensive and effective pedestrian information. Combined with the self-attention mechanism, it allows autonomous vehicles to pre-determine whether the pedestrian is stationary or moving. If the pedestrian is determined to be stationary, the program exits; if the pedestrian is determined to be moving, the future trajectory is output. This method enables autonomous vehicles to accurately predict future pedestrian behavior and adjust their routes in a timely manner to avoid collisions.

[0045] In one embodiment of the present invention, step 100 specifically includes:

[0046] The pedestrian keypoint detection information is input into the first self-attention model to obtain the association information between keypoints;

[0047] The correlation information between key points is input into the first multilayer perceptron to obtain the key point features of the pedestrian.

[0048] Specifically, the three-dimensional spatial coordinates of pedestrian key points are taken as input and mapped to obtain Query, Key, and Value, where Query, Key, and Value all represent the three-dimensional spatial coordinates of the key points.

[0049] The product of the query and the key is passed to the softmax layer based on the self-attention model to obtain the feature similarity weight matrix;

[0050] The value is weighted and summed based on the feature similarity weight matrix, and a vector sequence is output to obtain the correlation information between key points.

[0051] Alternatively, cross-attention models can also be used to obtain the correlation information between key points, but this description is not limited to these two calculation methods.

[0052] Finally, the above output vector sequence is input into the first multilayer perceptron to obtain the key point features of the pedestrian.

[0053] This specification does not specifically refer to a first multilayer perceptron, but may also refer to one or more of the following network structures: multilayer perceptron, convolutional neural network, long short-term memory network, and recurrent neural network, but is not limited to these.

[0054] In practical applications, during the operation of an autonomous vehicle, its perception system continuously senses surrounding pedestrians and promptly detects human bodies and their corresponding key point locations. This key point detection information is input into a first self-attention model and a first multilayer perceptron, or into a cross-attention mechanism and a first multilayer perceptron for calculation to obtain the pedestrian's key point features. This specification employs a combination of the first self-attention model and the first multilayer perceptron, which can more effectively capture the internal correlations of pedestrian features, thereby obtaining more efficient pedestrian key point features.

[0055] In one embodiment of the present invention, step 101 specifically includes:

[0056] Autonomous vehicles are equipped with various types of perception systems, such as cameras, ultrasonic radar, and lidar, to continuously observe pedestrians in intersection or non-intersection scenarios and accurately calculate the position, speed, and key features of pedestrians in various ranges.

[0057] In this specification, multiple types of perception systems may work together to leverage their respective strengths and provide redundant, complete, accurate, and faulty target information, thereby improving the correctness and safety of autonomous driving system decisions.

[0058] In one embodiment of the present invention, step 102 specifically includes:

[0059] By inputting multiple sets of historical features into the second self-attention model, the correlation information between the historical features is obtained;

[0060] The correlation information between historical features is input into the second multilayer perceptron to obtain the fused features of pedestrians.

[0061] Specifically, historical features mainly refer to the features of a pedestrian in each frame over a historical period, where the features of each frame include the pedestrian's position, speed, and key point features.

[0062] After mapping the features of each historical frame, we obtain the query, key, and value, where Query, Key, and Value all represent the features of each historical frame.

[0063] The features of each historical frame are input as nodes into the second self-attention model for calculation, thereby obtaining the correlation information between historical features. The calculation method is the same as that of the first self-attention model described above, and will not be repeated here.

[0064] This specification is not limited to the self-attention calculation method; cross-attention mechanism can also be used to obtain the correlation information between historical features of each frame, but it is not limited to this.

[0065] The features of each historical frame are then used as nodes and input into the second self-attention model for calculation. The correlation information between the obtained historical features is input into the second multilayer perceptron in vector form to obtain the fused features of the pedestrian.

[0066] This specification does not specifically refer to a second multilayer perceptron, but may also refer to one or more of the following network structures: multilayer perceptron, convolutional neural network, long short-term memory network, and recurrent neural network, but is not limited to these.

[0067] In practical applications, during the driving process of autonomous vehicles, their perception system constantly senses pedestrians around them. The perception system needs to input important information such as the location, speed, and key features of the captured pedestrians into the second self-attention model for calculation, and then input the output association structure into the second multilayer perceptron for calculation, so that the obtained pedestrian fusion features are more comprehensive and more efficient.

[0068] In one embodiment of the present invention, step 103 specifically includes:

[0069] Lane information is input into a convolutional neural network to obtain lane features.

[0070] Specifically, the perception system of autonomous vehicles uses one or more sensors, such as lidar, millimeter-wave radar, cameras, ultrasonic radar, and GPS, to acquire lane information.

[0071] In this specification, lane information includes, but is not limited to, one or more of the following: lane centerline information, lane width, and lane type in non-intersection scenarios. Intersection scenarios include cross-shaped, T-shaped, Y-shaped, staggered intersections, and complex intersections; non-intersection scenarios include one-way streets, two-way streets, and multi-lane streets. Lane centerline information is an important feature line in road geometry design and a crucial control line in road traffic management, used to separate opposing traffic flows. It is categorized as double solid center lines, single solid center lines, dashed center lines, and double solid center lines. Lane width refers to the width required for a single vehicle to travel safely and smoothly on the road, including the designed vehicle's outer width and the necessary clearance for passing, overtaking, or parallel driving. Lane type typically refers to bicycle lanes, bus lanes, and overtaking lanes.

[0072] The aforementioned lane information is input into a convolutional neural network (CNN). The CNN then sequentially extracts and maps features from the lane information to obtain lane features. In this specification, using a CNN to extract lane features effectively learns the corresponding features from a large number of samples, avoiding a complex feature extraction process.

[0073] This specification does not specifically refer to convolutional neural networks. It can also be one or more of the following network structures: multilayer perceptron, convolutional neural network, long short-term memory network, and recurrent neural network, but is not limited to these, in order to leverage their respective advantages and thereby improve the accuracy and safety of autonomous driving system decision-making.

[0074] In practical applications, before predicting pedestrian trajectories, it is necessary to train self-attention models, multilayer perceptrons, and decoders based on intersection and non-intersection scenarios. Pedestrian historical features include pedestrian position, speed, and key point features. For example, a prediction model can be trained based on pedestrians in intersection A or non-intersection B scenarios within a month. Specifically, key point features from intersection A or non-intersection B scenarios within a month are input into a first self-attention model and a first multilayer perceptron; historical pedestrian features from intersection A or non-intersection B scenarios within a month are input into a second self-attention model and a second multilayer perceptron; and lane information from intersection A or non-intersection B scenarios within a month is input into a convolutional neural network, resulting in a trained prediction model. The prediction model consists of a first self-attention model, a first multilayer perceptron, a second self-attention model, a second multilayer perceptron, and a convolutional neural network. Based on the historical pedestrian features and lane information in intersection A or non-intersection B scenarios at the current moment, the trained prediction model predicts the pedestrian trajectory in intersection A or non-intersection B scenarios.

[0075] In real-world applications, autonomous vehicles need to perceive not only pedestrians but also lane information, such as lane type, lane width, and lane line type, during operation. This lane information is then input into a second-layer sensor for feature vectorization to obtain lane features. This allows the autonomous vehicle to combine lane information with the pedestrian's future trajectory and further plan the driving path, thereby improving driving smoothness.

[0076] In one embodiment of the present invention, step 104 specifically includes:

[0077] The pedestrian and lane fusion features are input into the third self-attention model to obtain the pedestrian and lane fusion features.

[0078] Specifically, the fused features of pedestrians are used as the query, and the features of lanes are used as <key, value> data pairs, where Query = Key = Value.

[0079] Based on the self-attention model, the product of the query and the key is passed to the softmax layer to obtain the feature similarity weight matrix;

[0080] The value terms are weighted and summed based on the feature similarity weight matrix to obtain the vehicle-pedestrian lane fusion features.

[0081] The principles and calculation methods of the first self-attention model, the second self-attention model, and the third self-attention model are the same in this specification.

[0082] In one embodiment of the present invention, step 105 specifically includes:

[0083] The pedestrian lane fusion features are input into a binary classification multilayer perception network to obtain the probability that the pedestrian is stationary and the probability that the pedestrian is moving.

[0084] If the probability that a pedestrian is stationary is greater than the probability that a pedestrian is moving, then the pedestrian is stationary.

[0085] If the probability that a pedestrian is moving is greater than the probability that a pedestrian is stationary, then the pedestrian is moving.

[0086] This specification does not specifically refer to a binary classification multilayer perceptron. It can also be a network with classification function that is a combination of one or more of the following network structures: multilayer perceptron, convolutional neural network, long short-term memory network, and recurrent neural network, but is not limited to these.

[0087] In real-world applications, each pedestrian in intersection and non-intersection scenarios has a different movement pattern. For example, some pedestrians are stationary while others are moving. Autonomous vehicles can make timely adjustments based on the different movement patterns of pedestrians to avoid collisions.

[0088] according to Figure 2 This specification provides a method for predicting pedestrian trajectories based on key points, including:

[0089] Step 200: Input the pedestrian key point detection information into the first self-attention model to obtain the correlation information between key points; input the correlation information between key points into the first multilayer perceptron to obtain the pedestrian key point features.

[0090] Step 201: Extract multiple sets of historical features from multiple frames of historical images; the historical features include: pedestrian position, pedestrian speed, and key point features.

[0091] Step 202: Input multiple sets of historical features into the second self-attention model to obtain the correlation information between historical features; input the correlation information between historical features into the second multilayer perceptron to obtain the fused features of the pedestrian.

[0092] Step 203: Input the lane information into the convolutional neural network to obtain the lane features.

[0093] Step 204: Input the pedestrian fusion features and lane features into the third self-attention model to obtain pedestrian lane fusion features.

[0094] Step 205: Input the pedestrian lane fusion features into a binary classification multilayer perceptron to obtain the probability that the pedestrian is stationary and the probability that the pedestrian is moving.

[0095] Step 206: Determine if the probability that the pedestrian is stationary is greater than the probability that the pedestrian is moving. If yes, exit the program; otherwise, proceed to step 207.

[0096] Step 207: Input the pedestrian lane fusion features into the decoder to obtain the pedestrian's trajectory.

[0097] In the embodiments of this specification, when autonomous vehicles are driving in intersection and non-intersection scenarios and encounter pedestrians with different movement patterns, not only is pedestrian key point detection information taken into account, but attention model calculations are also used, which further enables autonomous vehicles to make accurate decisions in a timely and effective manner.

[0098] according to Figure 3 This specification provides a pedestrian trajectory prediction device based on key points, comprising:

[0099] The first extraction module 300 is configured to extract pedestrian key point features from pedestrian key point detection information; and extract multiple sets of historical features from multiple frames of historical images; the historical features include: pedestrian position, pedestrian speed and key point features.

[0100] The fusion module 301 is configured to generate fused features of pedestrians based on multiple sets of historical features.

[0101] The second extraction module 302 is configured to extract lane features from lane information.

[0102] The self-attention mechanism module 303 is configured to fuse pedestrian and lane features based on the self-attention mechanism to obtain pedestrian and lane fusion features.

[0103] The decoding module 304 is configured to determine whether a pedestrian is stationary or moving based on the pedestrian lane fusion features. If the pedestrian is moving, the pedestrian lane fusion features are input into the decoder to obtain the pedestrian's trajectory.

[0104] In one embodiment of the present invention, the first extraction module 300 is configured to input pedestrian key point detection information into a first self-attention model to obtain the correlation information between key points; and input the correlation information between key points into a first multilayer perceptron to obtain the key point features of the pedestrian.

[0105] In one embodiment of the present invention, the fusion module 301 is configured to input multiple sets of historical features into a second self-attention model to obtain the correlation information between the historical features; and input the correlation information between the historical features into a second multilayer perceptron to obtain the fused features of the pedestrian.

[0106] In one embodiment of the present invention, the second extraction module 302 is configured to input lane information into a convolutional neural network to obtain lane features.

[0107] In one embodiment of the present invention, the self-attention mechanism module 303 is configured to input the fused features of pedestrians and the features of lanes into a third self-attention model to obtain pedestrian-lane fused features.

[0108] In one embodiment of the present invention, the decoding module 304 is configured to input the pedestrian lane fusion features into a binary classification multilayer perception network to obtain the probability that the pedestrian is stationary and the probability that the pedestrian is moving.

[0109] If the probability that a pedestrian is stationary is greater than the probability that a pedestrian is moving, then the pedestrian is stationary.

[0110] If the probability that a pedestrian is moving is greater than the probability that a pedestrian is stationary, then the pedestrian is moving.

[0111] In one embodiment of the present invention, the decoder includes one or more of the following: a multilayer perceptron, a convolutional neural network, a long short-term memory network, and a recurrent neural network.

[0112] This invention provides an electronic device, comprising:

[0113] One or more processors;

[0114] Storage device for storing one or more programs.

[0115] When one or more programs are executed by one or more processors, the one or more processors implement the methods as described in any of the above embodiments.

[0116] This invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method of any of the above embodiments.

[0117] The following is for reference. Figure 4 A schematic diagram of the structure of a computer system 400 suitable for implementing an embodiment of the present invention is shown. Figure 4 The terminal device described is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0118] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0119] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0120] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined above in the system of this invention.

[0121] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a sending module, an acquisition module, a determining module, and a first processing module. The names of these modules do not necessarily limit the module itself; for example, the sending module can also be described as "a module that sends an image acquisition request to a connected server."

[0124] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A keypoint-based pedestrian trajectory prediction method, characterized in that, The method comprises the following steps: extracting key point features of pedestrians from pedestrian key point detection information; extracting multiple sets of historical features from multiple historical images; the historical features include the position of the pedestrian, the speed of the pedestrian and the key point features; generating a fusion feature of the pedestrian based on the multiple sets of historical features; extracting lane features from lane information; fusing the fusion feature of the pedestrian and the lane feature based on a self-attention mechanism to obtain a pedestrian-lane fusion feature; determining whether the pedestrian is stationary or moving based on the pedestrian-lane fusion feature, and if the pedestrian is moving, inputting the pedestrian-lane fusion feature into a decoder to obtain the trajectory of the pedestrian; extracting key point features of pedestrians from pedestrian key point detection information, comprising: inputting the pedestrian key point detection information into a first self-attention model to obtain correlation information between key points; inputting the correlation information between the key points into a first multi-layer perceptron to obtain the key point features of the pedestrian; the step of inputting the pedestrian key point detection information into the first self-attention model to obtain the correlation information between the key points comprises: taking the pedestrian key point three-dimensional space coordinate point as input, and after mapping, obtaining query Query, key value Key and value item Value, wherein Query=Key=Value all represent the three-dimensional space coordinate point of the key point; based on the self-attention model, the product of the query Query and the key value Key is transmitted to a softmax layer to obtain a feature similarity weight matrix; the value item Value is weighted and summed according to the feature similarity weight matrix to output a vector sequence, and the correlation information between the key points is obtained.

2. The method of claim 1, wherein: based on the multiple sets of historical features, generating a fusion feature of the pedestrian comprises: inputting the multiple sets of historical features into a second self-attention model to obtain correlation information between historical features; inputting the correlation information between the historical features into a second multi-layer perceptron to obtain the fusion feature of the pedestrian.

3. The method of claim 1, wherein: extracting lane features from lane information comprises: inputting the lane information into a convolutional neural network to obtain the lane features.

4. The method of claim 1, wherein: fusing the fusion feature of the pedestrian and the lane feature based on a self-attention mechanism to obtain a pedestrian-lane fusion feature comprises: inputting the fusion feature of the pedestrian and the lane feature into a third self-attention model to obtain the pedestrian-lane fusion feature.

5. The method of any one of claims 1-4, wherein: determining whether the pedestrian is stationary or moving based on the pedestrian-lane fusion feature comprises: inputting the pedestrian-lane fusion feature into a binary classification multi-layer perceptron network to obtain a probability that the pedestrian is stationary and a probability that the pedestrian is moving; if the probability that the pedestrian is stationary is greater than the probability that the pedestrian is moving, the pedestrian is stationary; if the probability that the pedestrian is moving is greater than the probability that the pedestrian is stationary, the pedestrian is moving.

6. The method of claim 1, wherein the decoder comprises one or a combination of more than one of a multi-layer perceptron, a convolutional neural network, a long short-term memory network, and a recurrent neural network.

7. A keypoint-based pedestrian trajectory prediction apparatus, characterized by, comprising: a first extraction module configured to extract a key point feature of a pedestrian from pedestrian key point detection information; extracting a plurality of sets of historical features from a plurality of historical images; the historical features include a position of the pedestrian, a speed of the pedestrian, and the key point feature; a fusion module configured to generate a fusion feature of the pedestrian based on the plurality of sets of historical features; a second extraction module configured to extract a feature of a lane from lane information; a self-attention mechanism module configured to fuse the fusion feature of the pedestrian and the feature of the lane based on a self-attention mechanism to obtain a pedestrian-lane fusion feature; a decoding module configured to determine whether the pedestrian is stationary or moving based on the pedestrian-lane fusion feature, and if moving, input the pedestrian-lane fusion feature into a decoder to obtain a trajectory of the pedestrian; the first extraction module is configured to input the pedestrian key point detection information into a first self-attention model to obtain association information between key points, and input the association information between the key points into a first multi-layer perceptron to obtain the key point feature of the pedestrian; the first extraction module is configured to: take the three-dimensional space coordinate points of the pedestrian key points as input, and after mapping, obtain a query Query, a key value Key, and a value item Value, wherein Query = Key = Value all represent the three-dimensional space coordinate points of the key points; based on a self-attention model, transmit the product of the query Query and the key value Key to a softmax layer to obtain a feature similarity weight matrix; and according to the feature similarity weight matrix, perform weighted summation on the value item Value and output a vector sequence, obtaining the association information between the key points.

8. An electronic device, comprising: comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

9. A computer readable medium having stored thereon a computer program, characterized in that, the program is executed by the processor to implement the method of any one of claims 1-6.

Images