Pedestrian trajectory prediction method, device and server

Abstract

The present invention provides a pedestrian trajectory prediction method, device, and server for use in the field of artificial intelligence technology. The method obtains a preset number of continuous frame images containing a pedestrian to be predicted and the pedestrian's historical motion trajectory. Based on the continuous frame images and historical motion trajectory, the method obtains the pedestrian's historical trajectory encoding features, intimacy information encoding features, and visual encoding features. Finally, the historical trajectory encoding features, intimacy information encoding features, and visual encoding features are input into a pre-trained trajectory prediction model to obtain a final prediction result for the pedestrian's motion trajectory. This method predicts pedestrian trajectories based on three conditions, effectively improving the accuracy of pedestrian trajectory prediction and thus meeting practical application needs.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a pedestrian trajectory prediction method, device and server. Background Art

[0002] The main goal of pedestrian trajectory prediction is to predict a pedestrian's future location at different moments based on a given pedestrian's historical trajectory. Because pedestrian trajectory prediction can determine a pedestrian's likely future location within a certain period of time, it helps avoid collisions during movement and has therefore become widely used.

[0003] Many applications of pedestrian trajectory prediction, such as smart cities, intelligent surveillance, and autonomous driving, place high demands on pedestrian trajectory prediction accuracy. Especially in the field of autonomous driving, the accuracy of pedestrian trajectory prediction not only affects the normal operation of the vehicle but also the personal safety of pedestrians.

[0004] However, the pedestrian trajectory prediction methods in the existing technology generally have low accuracy in pedestrian trajectory prediction and are difficult to meet the needs of practical applications. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a pedestrian trajectory prediction method, apparatus, and terminal device, which predict pedestrian trajectories based on multiple conditions, effectively improve the accuracy of pedestrian trajectory prediction, and meet practical application needs. The specific scheme is as follows:

[0006] In a first aspect, the present invention provides a pedestrian trajectory prediction method, comprising:

[0007] Obtaining a preset number of consecutive frame images containing a pedestrian to be predicted and a historical motion trajectory of the pedestrian to be predicted;

[0008] Inputting the historical motion trajectory of the pedestrian to be predicted into a pre-trained historical trajectory encoding model to obtain the historical trajectory encoding features of the pedestrian to be predicted, wherein the historical trajectory encoding model is obtained based on LSTM network training;

[0009] Inputting the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images into the LSTM network for encoding to obtain an intimacy information encoding feature, wherein the intimacy value is used to represent the degree of influence of the movement trajectories between pedestrians;

[0010] Extract visual features from each frame of the image to obtain visual coding features containing obstacle information;

[0011] Inputting the historical trajectory coding features, the intimacy information coding features, and the visual coding features of the pedestrian to be predicted into a pre-trained trajectory prediction model to obtain a final prediction result of the motion trajectory of the pedestrian to be predicted;

[0012] The trajectory prediction model is obtained by training the LSTM network with the output results of the historical trajectory encoding features, the intimacy information encoding features and the visual encoding features being close to the actual movement trajectory of the pedestrian.

[0013] Optionally, the step of inputting the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images into an LSTM network for encoding to obtain encoding features of the intimacy information includes:

[0014] Calculate the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images according to a preset algorithm;

[0015] Each of the intimacy values ​​is input into the LSTM network for encoding to obtain the intimacy information encoding feature.

[0016] Optionally, the calculating of the intimacy value between the pedestrian to be predicted and other pedestrians in each of the pictures according to a preset algorithm includes:

[0017] For each frame of the picture, perform the following operations to obtain the intimacy value between the pedestrian to be predicted and other pedestrians in each picture:

[0018] Obtaining the position coordinates of the pedestrian to be predicted and the other pedestrians in the image;

[0019] The position coordinates of the pedestrian to be predicted and the position coordinates of the other pedestrians are input into the following formula to obtain the intimacy value between the pedestrian to be predicted and the other pedestrians:

[0020]

[0021] Among them, a i，j Indicates the intimacy value between the pedestrian to be predicted and the other pedestrians;

[0022]

[0023]

[0024]

[0025]

[0026] represents the position coordinates of the pedestrian to be predicted in the t-th frame image, Indicates the position coordinates of the pedestrian to be predicted in the t-1th frame image;

[0027] represents the position coordinates of the other pedestrians in the t-th frame image, Indicates the position coordinates of the other pedestrians in the t-1th frame image.

[0028] Optionally, extracting visual features from each frame of the image to obtain visual coding features containing obstacle information includes:

[0029] Visual feature extraction is performed on each frame of the image based on the VGG-Net convolutional neural network;

[0030] Performing pooling processing on the visual features corresponding to each frame of the image to obtain pooled visual features;

[0031] The pooled visual features are respectively input into the LSTM network for encoding to obtain visual coding features.

[0032] Optionally, during the process of training the trajectory prediction model, the training end condition is that the error between the predicted trajectory and the actual trajectory is within a preset range.

[0033] In a second aspect, the present invention provides a pedestrian trajectory prediction device, comprising:

[0034] An acquisition unit, configured to acquire a preset number of consecutive frame images containing a pedestrian to be predicted, and a historical motion trajectory of the pedestrian to be predicted;

[0035] a trajectory encoding feature unit, configured to input the historical motion trajectory of the pedestrian to be predicted into a pre-trained historical trajectory encoding model to obtain the historical trajectory encoding features of the pedestrian to be predicted, wherein the historical trajectory encoding model is obtained based on LSTM network training;

[0036] an intimacy encoding feature unit, configured to input the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images into an LSTM network for encoding, thereby obtaining an intimacy information encoding feature, wherein the intimacy value is used to characterize the degree of influence of the movement trajectories between the pedestrians;

[0037] A visual coding feature unit is used to extract visual features from each frame of the image to obtain visual coding features containing obstacle information;

[0038] a final prediction unit, configured to input the historical trajectory encoding features, the intimacy information encoding features, and the visual encoding features of the pedestrian to be predicted into a pre-trained trajectory prediction model to obtain a final prediction result of the motion trajectory of the pedestrian to be predicted;

[0039] The trajectory prediction model is obtained by training the LSTM network with the output results of the historical trajectory encoding features, the intimacy information encoding features and the visual encoding features being close to the actual movement trajectory of the pedestrian.

[0040] Optionally, the intimacy encoding feature unit is configured to input the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images into an LSTM network for encoding, and to obtain the intimacy information encoding feature, specifically including:

[0041] Calculate the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images according to a preset algorithm;

[0042] Each of the intimacy values ​​is input into the LSTM network for encoding to obtain the intimacy information encoding feature.

[0043] Optionally, the intimacy encoding feature unit is configured to calculate the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images according to a preset algorithm, specifically including:

[0044] For each frame of the picture, perform the following operations to obtain the intimacy value between the pedestrian to be predicted and other pedestrians in each picture:

[0045] Obtaining the position coordinates of the pedestrian to be predicted and the other pedestrians in the image;

[0046] The position coordinates of the pedestrian to be predicted and the position coordinates of the other pedestrians are input into the following formula to obtain the intimacy value between the pedestrian to be predicted and the other pedestrians:

[0047]

[0048] Among them, a i，j Indicates the intimacy value between the pedestrian to be predicted and the other pedestrians;

[0049]

[0050]

[0051]

[0052]

[0053] represents the position coordinates of the pedestrian to be predicted in the t-th frame image, Indicates the position coordinates of the pedestrian to be predicted in the t-1th frame image;

[0054] represents the position coordinates of the other pedestrians in the t-th frame image, Indicates the position coordinates of the other pedestrians in the t-1th frame image.

[0055] Optionally, the visual coding feature unit is configured to extract visual features from each frame of the image to obtain visual coding features containing obstacle information, specifically including:

[0056] Visual feature extraction is performed on each frame of the image based on the VGG-Net convolutional neural network;

[0057] Performing pooling processing on the visual features corresponding to each frame of the image to obtain pooled visual features;

[0058] The pooled visual features are respectively input into the LSTM network for encoding to obtain visual coding features.

[0059] In a third aspect, the present invention provides a terminal device comprising: a memory and a processor; the memory stores a program suitable for execution by the processor to implement the pedestrian trajectory prediction method described in any one of the first aspects of the present invention.

[0060] Based on the pedestrian trajectory prediction method provided by the present invention, first obtain a preset number of continuous frame images containing the pedestrian to be predicted and the historical motion trajectory of the pedestrian to be predicted, then input the historical motion trajectory of the pedestrian to be predicted into the historical trajectory coding model to obtain the historical trajectory coding features of the pedestrian to be predicted; input the intimacy value between the pedestrian to be predicted and other pedestrians in each image into the LSTM network for encoding to obtain the intimacy information coding features; further perform visual feature extraction on each frame of the image to obtain the visual coding features containing obstacle information; finally, input the historical trajectory coding features, intimacy information coding features, and visual coding features of the pedestrian to be predicted into the pre-trained trajectory prediction model to obtain the final prediction result of the motion trajectory of the pedestrian to be predicted. The pedestrian trajectory prediction method provided by the present invention predicts pedestrian trajectories based on the three conditions of the historical trajectory coding features, intimacy information coding features, and visual coding features of the pedestrian to be predicted, which can effectively improve the accuracy of pedestrian trajectory prediction, thereby meeting the needs of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 This is a flow chart of a pedestrian trajectory prediction method provided by an embodiment of the present invention;

[0063] Figure 2 Schematic diagram of the process of obtaining visual coding features in the pedestrian trajectory prediction method provided by an embodiment of the present invention;

[0064] Figure 3 This is a flow chart of an implementation method of a pedestrian trajectory prediction method provided by an embodiment of the present invention;

[0065] Figure 4 This is a structural block diagram of a pedestrian trajectory prediction device provided by an embodiment of the present invention;

[0066] Figure 5 This is a structural block diagram of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0068] See also Figure 1 , the pedestrian trajectory prediction method provided by the embodiment of the present invention is a flow chart. The method can be applied to electronic devices that can run preset programs and process images or data. The electronic devices can be laptops, PCs, mobile phones, etc., or they can be control devices integrated into vehicles, such as control devices of automatic driving systems. Of course, in some cases, the pedestrian trajectory prediction method provided by the embodiment of the present invention can also be applied to servers on the network side. Figure 1 The pedestrian trajectory prediction method provided by the embodiment of the present invention may include:

[0069] S100: Obtain a preset number of consecutive frame images containing pedestrians to be predicted, and historical movement trajectories of the pedestrians to be predicted.

[0070] First, continuous frame images can be acquired using existing image acquisition methods. For example, a video can be captured, and continuous frame images that are temporally sequential within a specified time period in the video can be used as the images used in this embodiment. Of course, other existing methods can also be used to acquire continuous frame images, and the present invention does not limit the acquisition of continuous frame images.

[0071] For the device that implements the pedestrian trajectory prediction method provided by the embodiment of the present invention, after receiving the picture, it is necessary to identify the content in the picture, that is, to identify pedestrians and objects other than pedestrians, such as stationary obstacles, other vehicles, etc. On the basis of identifying pedestrians, the position coordinates of the pedestrians in the picture should also be obtained synchronously. Assume that there are N pedestrians P1, P2, ... P N , t is the current frame time, and the position of the i-th pedestrian at time t is Its movement trajectory can be expressed as in (x i ,y i ) represents the position coordinates of the pedestrian in the image. The method for identifying pedestrians from images can be implemented with reference to the prior art, and the present invention does not limit this.

[0072] In most cases, a picture includes multiple pedestrians. In this case, the pedestrian to be predicted can be any pedestrian in each picture. Therefore, in practical applications, all pedestrians in the obtained continuous frames of pictures can be used as pedestrians to be predicted. Accordingly, when executing the pedestrian trajectory prediction algorithm provided by the embodiment of the present invention, the trajectory of each pedestrian in the picture can be predicted simultaneously; if the computing power of the device is limited, the trajectory of the pedestrians in the picture can also be predicted in a certain order. It can be imagined that in the field of intelligent driving, most people require the ability to quickly and accurately predict pedestrian trajectories. Therefore, efforts should be made to improve the computing power of related equipment and strive to make trajectory predictions for all pedestrians in the picture in the shortest time.

[0073] The historical motion trajectory of the pedestrian to be predicted can be obtained based on a tracker in the prior art. The present invention does not limit the specific method for obtaining the historical motion trajectory of the pedestrian to be predicted.

[0074] S110 , inputting the historical motion trajectory of the pedestrian to be predicted into a pre-trained historical trajectory encoding model to obtain the historical trajectory encoding features of the pedestrian to be predicted.

[0075] This embodiment of the present invention provides a pre-trained historical trajectory encoding model based on an LSTM network. The process of training the LSTM network to obtain the historical trajectory encoding model, using motion trajectories as input and historical trajectory encoding features as output, can be implemented based on existing technologies and is not limited by this invention.

[0076] In practical applications, it is only necessary to call the trained historical trajectory encoding model and input the historical motion trajectory of the pedestrian to be predicted obtained in the above steps into the historical trajectory encoding model to obtain the historical trajectory encoding features of the pedestrian to be predicted.

[0077] S120: Input the intimacy value between the pedestrian to be predicted and other pedestrians in each picture into the LSTM network for encoding to obtain the intimacy information encoding feature.

[0078] In an embodiment of the present invention, the intimacy value is used to characterize the degree of influence of the movement trajectories between pedestrians. For example, if pedestrian A and pedestrian B have the same moving direction, the intimacy value is relatively high; if the moving directions are inconsistent, the intimacy value is relatively low. The higher the intimacy value between pedestrians, the greater the influence between them.

[0079] Optionally, the present invention provides a preset algorithm for calculating the intimacy value between pedestrians. Specifically, for each frame of the image obtained in the above steps, the following operations are performed to obtain the intimacy value between the pedestrian to be predicted and other pedestrians in each image:

[0080] First, the position coordinates of the pedestrian to be predicted and other pedestrians in the image are obtained. The position coordinates can be directly obtained by using the recognition results in the aforementioned S100. Accordingly, if the operation of recognizing the position coordinates is not performed in S100, the recognition method in the prior art can also be used in this step.

[0081] After obtaining the position coordinates of the pedestrian to be predicted and other pedestrians in the currently processed image, input the position coordinates of the pedestrian to be predicted and the position coordinates of other pedestrians into the following formula to obtain the closeness value of the pedestrian to be predicted and other pedestrians:

[0082]

[0083] Among them, a i，j Indicates the intimacy value between the pedestrian to be predicted and the other pedestrians;

[0084]

[0085]

[0086]

[0087]

[0088] represents the position coordinates of the pedestrian to be predicted in the t-th frame image, Indicates the position coordinates of the pedestrian to be predicted in the t-1th frame image;

[0089] represents the position coordinates of the other pedestrians in the t-th frame image, Indicates the position coordinates of the other pedestrians in the t-1th frame image.

[0090] For each frame of the image, the above calculation is performed to obtain the intimacy value between the pedestrian to be predicted and other pedestrians in each image.

[0091] Then, each intimacy value is input into the LSTM network for encoding to obtain the intimacy information encoding feature. The encoding process based on the LSTM network can be implemented with reference to the existing technology, and the present invention does not limit the specific encoding process.

[0092] S130 , extracting visual features from each frame of the image to obtain visual coding features containing obstacle information.

[0093] Optionally, in an embodiment of the present invention, the visual feature extraction of each frame of the image is implemented based on the VGG-Net convolutional neural network. First, the visual features of each frame of the image are extracted separately through the VGG-Net convolutional neural network. Then, the visual features corresponding to each frame of the image are pooled to obtain the pooled visual features. Finally, each pooled visual feature is input into the LSTM network for encoding to obtain the visual coding features.

[0094] Specifically, in order to better illustrate the process of visual feature extraction, you can refer to Figure 2 The schematic diagram of the visual encoding feature process shown in the figure, the embodiment of the present invention uses VGG-Net

[13] to extract the Conv4_3 layer feature F, assuming that its corresponding dimension is: w*h*c, first through channel average pooling, to obtain the feature F with a dimension of w*h*1 1 ,To better understand channel average pooling, the mathematical formula is expressed as: Then input the fully connected layer with the corresponding number of nodes 1024,256 to obtain the 256-dimensional feature F 2 Finally, use an LSTM to 2 It can be understood that the pooling process can clearly highlight the presence of obstacles. Specifically, the pooled feature values ​​are more obvious than those in non-obstacle areas. Therefore, the visual coding features obtained in this step can effectively reflect the obstacle information in each image, providing more accurate basic data for the subsequent steps to accurately predict pedestrian trajectories.

[0095] It should be particularly emphasized that after executing S100 and obtaining a preset number of continuous frame images containing the pedestrian to be predicted, S110, S120, and S130 can be executed in sequence. If the electronic device that executes the pedestrian trajectory prediction method provided by the embodiment of the present invention has sufficient computing power, it is obvious that S110, S120, and S130 can also be executed simultaneously, thereby obtaining the historical trajectory coding features, intimacy information coding features, and visual coding features in a shorter computing time, and then continuing to execute S140. Optionally, for the implementation method of synchronously executing S110, S120, and S130, please refer to Figure 3 , Figure 3 FIG. 1 is a flow chart of an optional implementation of the pedestrian trajectory prediction method provided by an embodiment of the present invention. Figure 3 As shown, the historical trajectory encoding features, intimacy information encoding features, and visual encoding features are obtained simultaneously and then synchronously input into the trajectory prediction model, that is, executing S140. Therefore, the present invention does not limit the specific execution order of S110, S120, and S130. This embodiment describes the execution process of this embodiment according to S110, S120, and S130 only for the purpose of facilitating the implementation process of the present invention and does not limit the specific execution process of the technical solution.

[0096] S140 , inputting the historical trajectory coding features, intimacy information coding features, and visual coding features of the pedestrian to be predicted into a pre-trained trajectory prediction model to obtain a final prediction result of the motion trajectory of the pedestrian to be predicted.

[0097] Optionally, embodiments of the present invention also include a pre-trained trajectory prediction model. This trajectory prediction model is obtained by training an LSTM network based on the output of historical trajectory encoding features, intimacy information encoding features, and visual encoding features, with the goal of approximating the actual movement trajectory of the pedestrian. Similar to the historical trajectory encoding model, the trajectory prediction model training process also terminates when the error between the predicted trajectory and the actual trajectory falls within a preset range.

[0098] After the above steps, the historical trajectory encoding features, intimacy information encoding features, and visual encoding features of the pedestrian to be predicted have been obtained. By inputting this information into the trajectory prediction model, the final prediction result of the movement trajectory of the pedestrian to be predicted can be obtained.

[0099] In summary, the pedestrian trajectory prediction method provided by the present invention predicts pedestrian trajectories based on three conditions: the historical trajectory coding features, intimacy information coding features, and visual coding features of the pedestrian to be predicted. This can effectively improve the accuracy of pedestrian trajectory prediction and thus meet practical application needs.

[0100] The following is an introduction to the pedestrian trajectory prediction device provided in an embodiment of the present invention. The pedestrian trajectory prediction device described below can be considered as the functional module architecture that needs to be set in the central device to implement the pedestrian trajectory prediction method provided in an embodiment of the present invention; the description below can be cross-referenced with the above.

[0101] Figure 4 A structural block diagram of a pedestrian trajectory prediction device provided by an embodiment of the present invention, referring to Figure 4 , the apparatus may include:

[0102] An acquisition unit 10 is configured to acquire a preset number of consecutive frame images containing a pedestrian to be predicted, and a historical motion trajectory of the pedestrian to be predicted;

[0103] The trajectory encoding feature unit 20 is used to input the historical motion trajectory of the pedestrian to be predicted into a pre-trained historical trajectory encoding model to obtain the historical trajectory encoding feature of the pedestrian to be predicted, wherein the historical trajectory encoding model is obtained based on LSTM network training;

[0104] The intimacy encoding feature unit 30 is used to input the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images into the LSTM network for encoding to obtain an intimacy information encoding feature, wherein the intimacy value is used to represent the degree of influence of the movement trajectories between pedestrians;

[0105] A visual coding feature unit 40 is configured to extract visual features from each frame of the image to obtain visual coding features containing obstacle information;

[0106] A final prediction unit 50 is configured to input the historical trajectory encoding features, the intimacy information encoding features, and the visual encoding features of the pedestrian to be predicted into a pre-trained trajectory prediction model to obtain a final prediction result of the motion trajectory of the pedestrian to be predicted;

[0107] The trajectory prediction model is obtained by training the LSTM network with the output results of the historical trajectory encoding features, the intimacy information encoding features and the visual encoding features being close to the actual movement trajectory of the pedestrian.

[0108] Optionally, the intimacy encoding feature unit 30 is configured to input the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images into an LSTM network for encoding, and to obtain the intimacy information encoding feature, specifically including:

[0109] Calculate the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images according to a preset algorithm;

[0110] Each of the intimacy values ​​is input into the LSTM network for encoding to obtain the intimacy information encoding feature.

[0111] Optionally, the intimacy encoding feature unit 30 is configured to calculate the intimacy value between the pedestrian to be predicted and other pedestrians in each of the images according to a preset algorithm, specifically including:

[0112] For each frame of the picture, perform the following operations to obtain the intimacy value between the pedestrian to be predicted and other pedestrians in each picture:

[0113] Obtaining the position coordinates of the pedestrian to be predicted and the other pedestrians in the image;

[0114] The position coordinates of the pedestrian to be predicted and the position coordinates of the other pedestrians are input into the following formula to obtain the intimacy value between the pedestrian to be predicted and the other pedestrians:

[0115]

[0116] Among them, a i，j Indicates the intimacy value between the pedestrian to be predicted and the other pedestrians;

[0117]

[0118]

[0119]

[0120]

[0121] represents the position coordinates of the pedestrian to be predicted in the t-th frame image, Indicates the position coordinates of the pedestrian to be predicted in the t-1th frame image;

[0122] represents the position coordinates of the other pedestrians in the t-th frame image, Indicates the position coordinates of the other pedestrians in the t-1th frame image.

[0123] Optionally, the visual coding feature unit 40 is configured to extract visual features from each frame of the image to obtain visual coding features containing obstacle information, specifically including:

[0124] Visual feature extraction is performed on each frame of the image based on the VGG-Net convolutional neural network;

[0125] Performing pooling processing on the visual features corresponding to each frame of the image to obtain pooled visual features;

[0126] The pooled visual features are respectively input into the LSTM network for encoding to obtain visual coding features.

[0127] Figure 5 For a structural block diagram of a terminal device provided in an embodiment of the present invention, see Figure 5 As shown, it may include: at least one processor 100, at least one communication interface 200, at least one memory 300 and at least one communication bus 400;

[0128] In the embodiment of the present invention, the number of the processor 100, the communication interface 200, the memory 300, and the communication bus 400 is at least one, and the processor 100, the communication interface 200, and the memory 300 communicate with each other through the communication bus 400; obviously, Figure 5 The communication connections shown for the processor 100, communication interface 200, memory 300, and communication bus 400 are merely optional;

[0129] Optionally, the communication interface 200 may be an interface of a communication module, such as an interface of a GSM module;

[0130] The processor 100 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0131] The memory 300 stores application programs and may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0132] The processor 100 is specifically configured to execute an application program in the memory to implement any embodiment of the pedestrian trajectory prediction method described above.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0134] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0135] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0136] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the core concept or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pedestrian trajectory prediction method, characterized in that: include: Obtaining a preset number of consecutive frame images containing a pedestrian to be predicted and a historical motion trajectory of the pedestrian to be predicted; Inputting the historical motion trajectory of the pedestrian to be predicted into a pre-trained historical trajectory encoding model to obtain the historical trajectory encoding features of the pedestrian to be predicted, wherein the historical trajectory encoding model is obtained based on LSTM network training; For each frame of the picture, perform the following operations to obtain the intimacy value between the pedestrian to be predicted and other pedestrians in each picture: Obtaining the position coordinates of the pedestrian to be predicted and the other pedestrians in the image; The position coordinates of the pedestrian to be predicted and the position coordinates of the other pedestrians are input into the following formula to obtain the intimacy value between the pedestrian to be predicted and the other pedestrians: in, Indicates the intimacy value between the pedestrian to be predicted and the other pedestrians; represents the position coordinates of the pedestrian to be predicted in the t-th frame image, Indicates the position coordinates of the pedestrian to be predicted in the t-1th frame image; represents the position coordinates of the other pedestrians in the t-th frame image, Indicates the position coordinates of the other pedestrians in the t-1th frame image; Inputting each of the intimacy values ​​into an LSTM network for encoding to obtain an intimacy information encoding feature, wherein the intimacy value is used to characterize the degree of influence of the movement trajectories between pedestrians; Extract visual features from each frame of the image to obtain visual coding features containing obstacle information; Inputting the historical trajectory coding features, the intimacy information coding features, and the visual coding features of the pedestrian to be predicted into a pre-trained trajectory prediction model to obtain a final prediction result of the motion trajectory of the pedestrian to be predicted; The trajectory prediction model is obtained by training the LSTM network with the output results of the historical trajectory encoding features, the intimacy information encoding features and the visual encoding features being close to the actual movement trajectory of the pedestrian.

2. The pedestrian trajectory prediction method according to claim 1, characterized in that The visual feature extraction is performed on each frame of the image to obtain visual coding features containing obstacle information, including: Visual feature extraction is performed on each frame of the image based on the VGG-Net convolutional neural network; Performing pooling processing on the visual features corresponding to each frame of the image to obtain pooled visual features; The pooled visual features are respectively input into the LSTM network for encoding to obtain visual coding features.

3. The pedestrian trajectory prediction method according to claim 1, characterized in that During the process of training the trajectory prediction model, the training ends when the error between the predicted trajectory and the actual trajectory is within a preset range.

4. A pedestrian trajectory prediction device, characterized in that: include: An acquisition unit, configured to acquire a preset number of consecutive frame images containing a pedestrian to be predicted, and a historical motion trajectory of the pedestrian to be predicted; a trajectory encoding feature unit, configured to input the historical motion trajectory of the pedestrian to be predicted into a pre-trained historical trajectory encoding model to obtain the historical trajectory encoding features of the pedestrian to be predicted, wherein the historical trajectory encoding model is obtained based on LSTM network training; The intimacy encoding feature unit is configured to perform the following operations for each frame of the image to obtain the intimacy value between the pedestrian to be predicted and other pedestrians in each frame of the image: Obtaining the position coordinates of the pedestrian to be predicted and the other pedestrians in the image; The position coordinates of the pedestrian to be predicted and the position coordinates of the other pedestrians are input into the following formula to obtain the intimacy value between the pedestrian to be predicted and the other pedestrians: in, Indicates the intimacy value between the pedestrian to be predicted and the other pedestrians; represents the position coordinates of the pedestrian to be predicted in the t-th frame image, Indicates the position coordinates of the pedestrian to be predicted in the t-1th frame image; represents the position coordinates of the other pedestrians in the t-th frame image, Indicates the position coordinates of the other pedestrians in the t-1th frame image; Inputting each of the intimacy values ​​into an LSTM network for encoding to obtain an intimacy information encoding feature, wherein the intimacy value is used to characterize the degree of influence of the movement trajectories between pedestrians; A visual coding feature unit is used to extract visual features from each frame of the image to obtain visual coding features containing obstacle information; a final prediction unit, configured to input the historical trajectory encoding features, the intimacy information encoding features, and the visual encoding features of the pedestrian to be predicted into a pre-trained trajectory prediction model to obtain a final prediction result of the motion trajectory of the pedestrian to be predicted; The trajectory prediction model is obtained by training the LSTM network with the output results of the historical trajectory encoding features, the intimacy information encoding features and the visual encoding features being close to the actual movement trajectory of the pedestrian.

5. The pedestrian trajectory prediction device according to claim 4, characterized in that: The visual coding feature unit is configured to extract visual features from each frame of the image to obtain visual coding features containing obstacle information, specifically comprising: Visual feature extraction is performed on each frame of the image based on the VGG-Net convolutional neural network; Performing pooling processing on the visual features corresponding to each frame of the image to obtain pooled visual features; The pooled visual features are respectively input into the LSTM network for encoding to obtain visual coding features.

6. A terminal device, characterized in that: include: memory and processor; The memory stores a program suitable for execution by the processor to implement the pedestrian trajectory prediction method according to any one of claims 1 to 3.

Images