Pedestrian trajectory prediction method and device based on multiple spatial graphs and time fusion

Abstract

The present invention discloses a method and device for pedestrian trajectory prediction based on multiple spatial graphs and temporal fusion. The method comprises: obtaining a set of pedestrian trajectories for each pedestrian in a target scene, wherein the pedestrian trajectories include the pedestrian's spatial position information and temporal information; extracting spatial local information, spatial global information, temporal local information, and temporal global information from the pedestrian trajectory set; performing a first fusion of the spatial local information and the spatial global information to obtain multiple spatial features; performing a second fusion of the temporal local information and the temporal global information to obtain temporal features; and determining trajectory prediction information for the target pedestrian based on the multiple spatial features and the temporal features. The present invention improves the accuracy of pedestrian trajectory prediction and can be widely applied in the field of artificial intelligence technology.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a pedestrian trajectory prediction method and device based on multiple spatial graphs and time fusion. Background Art

[0002] When people walk in public places, they interact with other pedestrians around them. And this interaction follows a kind of common sense and social customs shared by people. Therefore, in order to accurately predict the future trajectory of pedestrians, it is necessary to accurately simulate the interaction between pedestrians. Figure 1 As shown in the figure, the related art method mainly consists of two parts: a spatial transformer and a temporal transformer. The spatial transformer input is the position features of all pedestrians in a scene at the same time, which is used to extract the spatial interactions between all pedestrians at the same time. The temporal transformer input is the position features of a pedestrian's trajectory over a period of time, and outputs the updated trajectory features, which represent a description of the pedestrian's possible future trajectory based on the pedestrian's trajectory over this period of time.

[0003] However, although this approach recognizes that pedestrian trajectories are affected by both time and space, and therefore runs the spatial and temporal transformers in parallel, combining their features before feeding them into the connected spatial and temporal transformers to ultimately determine the future trajectory, this approach still inherently separates temporal and spatial features, failing to effectively simulate the spatiotemporal interactions between pedestrians. This is because pedestrians, when planning their own routes, also consider the historical trajectories of surrounding pedestrians and infer possible future trajectories to adjust their subsequent actions. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a pedestrian trajectory prediction method and device with high accuracy based on multiple spatial graphs and time fusion.

[0005] One aspect of the present invention provides a pedestrian trajectory prediction method based on multiple spatial graphs and time fusion, comprising:

[0006] Obtaining a set of pedestrian trajectories of each pedestrian in a target scene, wherein the pedestrian trajectory includes spatial position information and time information of the pedestrian;

[0007] Extracting spatial local information, spatial global information, temporal local information and temporal global information from the pedestrian trajectory set;

[0008] Performing a first fusion of the spatial local information and the spatial global information to obtain multiple spatial features;

[0009] Performing a second fusion of the local time series information and the global time series information to obtain a time series feature;

[0010] The target pedestrian's trajectory prediction information is determined based on the multiple spatial features and the temporal features.

[0011] Optionally, obtaining a pedestrian trajectory set of each pedestrian in the target scene includes:

[0012] Obtain the observed position information of each pedestrian in the target scene from the first moment to the second moment;

[0013] Connecting any two pedestrians in the target scene that meet a preset distance condition through an undirected edge to determine the interaction relationship between the two pedestrians;

[0014] A pedestrian trajectory set is constructed according to the observation position information and the interaction relationship.

[0015] Optionally, obtaining a pedestrian trajectory set of each pedestrian in the target scene further includes:

[0016] Mapping the pedestrian trajectory set into a query vector, a key vector, and a value vector;

[0017] Processing the query vector through a query mapping function to construct a pedestrian query matrix;

[0018] Processing the key vector through a key mapping function to construct a key matrix for the pedestrian;

[0019] The value vector is processed by a value mapping function to construct a value matrix of the pedestrian.

[0020] Optionally, extracting spatial local information and spatial global information from the pedestrian trajectory set includes:

[0021] Obtaining position information of each pedestrian at consecutive moments from the pedestrian trajectory set;

[0022] Obtain the adjacency matrix of the interaction relationships between pedestrians;

[0023] The influence of other pedestrians around each pedestrian is determined according to the position information and the adjacency matrix.

[0024] Optionally, the method further includes:

[0025] The adjacent pedestrian set of each pedestrian is determined by graph convolution, and the adjacent pedestrian set includes the adjacency matrix of all pedestrians.

[0026] Optionally, determining the influence of other pedestrians around each pedestrian based on the position information and the adjacency matrix includes:

[0027] Get the target image frame;

[0028] Calculating the influence of other pedestrians around each pedestrian in the target image frame according to the position information and the adjacency matrix;

[0029] The influence of other pedestrians around each pedestrian in the target image frame is generalized to multiple other image frames to obtain the influence of other pedestrians around each pedestrian at the current moment and historical moments.

[0030] Optionally, the calculation formula for the influence is:

[0031]

[0032] Among them, Attention j→i represents the influence of pedestrian j on pedestrian i; q i represents the query vector of pedestrian i; k j represents the key vector of pedestrian j; v j represents the value vector of pedestrian j; d k The problem dimensions representing the query vector, key vector, and value vector; softmax() is used to normalize the weights of different pedestrians for pedestrian i so that their sum is 1;

[0033] The calculation formula of the multiple spatial features is:

[0034] Total(i)=Global(i)+Local(i)

[0035] Among them, Total(i) represents multiple spatial features; Global(i) represents the global information of pedestrian i; Local(i) represents the local information of pedestrian i.

[0036] Another aspect of the present invention provides a pedestrian trajectory prediction device based on multiple spatial graphs and time fusion, comprising:

[0037] The first module is used to obtain a set of pedestrian trajectories of each pedestrian in the target scene, wherein the pedestrian trajectory includes the spatial position information and time information of the pedestrian;

[0038] The second module is used to extract spatial local information, spatial global information, temporal local information and temporal global information from the pedestrian trajectory set;

[0039] A third module is configured to perform a first fusion of the spatial local information and the spatial global information to obtain multiple spatial features;

[0040] A fourth module is configured to perform a second fusion of the local time series information and the global time series information to obtain a time series feature;

[0041] The fifth module is used to determine the trajectory prediction information of the target pedestrian based on the multiple spatial features and the temporal features.

[0042] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0043] The memory is used to store programs;

[0044] The processor executes the program to implement the method described above.

[0045] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0046] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0047] Embodiments of the present invention obtain a set of pedestrian trajectories for each pedestrian in a target scene, wherein the pedestrian trajectories include the pedestrian's spatial position information and time information; extract spatial local information, spatial global information, temporal local information, and temporal global information from the pedestrian trajectory set; perform a first fusion of the spatial local information and the spatial global information to obtain multiple spatial features; perform a second fusion of the temporal local information and the temporal global information to obtain temporal features; and determine trajectory prediction information for the target pedestrian based on the multiple spatial features and the temporal features. This invention improves the accuracy of pedestrian trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1This is a flowchart of related technologies when predicting pedestrian trajectories;

[0050] Figure 2 A flowchart of the steps of a pedestrian trajectory prediction method according to an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the structure of a multi-spatial Transformer module provided in an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a multiple adjacency matrix defined by fusing multiple spatial graphs provided in an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of the spatial and temporal correspondence between global information and local information provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] To address the problems of existing technologies and to simultaneously simulate the temporal and spatial influences on pedestrians during their movements to obtain more accurate pedestrian trajectory prediction, embodiments of the present invention provide a pedestrian trajectory prediction method based on multiple spatial graphs and temporal fusion, including:

[0056] Obtaining a set of pedestrian trajectories of each pedestrian in a target scene, wherein the pedestrian trajectory includes spatial position information and time information of the pedestrian;

[0057] Extracting spatial local information, spatial global information, temporal local information and temporal global information from the pedestrian trajectory set;

[0058] Performing a first fusion of the spatial local information and the spatial global information to obtain multiple spatial features;

[0059] Performing a second fusion of the local time series information and the global time series information to obtain a time series feature;

[0060] The target pedestrian's trajectory prediction information is determined based on the multiple spatial features and the temporal features.

[0061] Optionally, obtaining a pedestrian trajectory set of each pedestrian in the target scene includes:

[0062] Obtain the observed position information of each pedestrian in the target scene from the first moment to the second moment;

[0063] Connecting any two pedestrians in the target scene that meet a preset distance condition through an undirected edge to determine the interaction relationship between the two pedestrians;

[0064] A pedestrian trajectory set is constructed according to the observation position information and the interaction relationship.

[0065] Optionally, obtaining a pedestrian trajectory set of each pedestrian in the target scene further includes:

[0066] Mapping the pedestrian trajectory set into a query vector, a key vector, and a value vector;

[0067] Processing the query vector through a query mapping function to construct a pedestrian query matrix;

[0068] Processing the key vector through a key mapping function to construct a key matrix for the pedestrian;

[0069] The value vector is processed by a value mapping function to construct a value matrix of the pedestrian.

[0070] Optionally, extracting spatial local information and spatial global information from the pedestrian trajectory set includes:

[0071] Obtaining position information of each pedestrian at consecutive moments from the pedestrian trajectory set;

[0072] Obtain the adjacency matrix of the interaction relationships between pedestrians;

[0073] The influence of other pedestrians around each pedestrian is determined according to the position information and the adjacency matrix.

[0074] Optionally, the method further includes:

[0075] The adjacent pedestrian set of each pedestrian is determined by graph convolution, and the adjacent pedestrian set includes the adjacency matrix of all pedestrians.

[0076] Optionally, determining the influence of other pedestrians around each pedestrian based on the position information and the adjacency matrix includes:

[0077] Get the target image frame;

[0078] Calculating the influence of other pedestrians around each pedestrian in the target image frame according to the position information and the adjacency matrix;

[0079] The influence of other pedestrians around each pedestrian in the target image frame is generalized to multiple other image frames to obtain the influence of other pedestrians around each pedestrian at the current moment and historical moments.

[0080] Optionally, the calculation formula for the influence is:

[0081]

[0082] Among them, Attention j→i represents the influence of pedestrian j on pedestrian i; q i represents the query vector of pedestrian i; k j represents the key vector of pedestrian j; v j represents the value vector of pedestrian j; d k The problem dimensions representing the query vector, key vector, and value vector; softmax() is used to normalize the weights of different pedestrians for pedestrian i so that their sum is 1;

[0083] The calculation formula of the multiple spatial features is:

[0084] Total(i)=Global(i)+Local(i)

[0085] Among them, Total(i) represents multiple spatial features; Global(i) represents the global information of pedestrian i; Local(i) represents the local information of pedestrian i.

[0086] Another aspect of the present invention provides a pedestrian trajectory prediction device based on multiple spatial graphs and time fusion, comprising:

[0087] The first module is used to obtain a set of pedestrian trajectories of each pedestrian in the target scene, wherein the pedestrian trajectory includes the spatial position information and time information of the pedestrian;

[0088] The second module is used to extract spatial local information, spatial global information, temporal local information and temporal global information from the pedestrian trajectory set;

[0089] A third module is configured to perform a first fusion of the spatial local information and the spatial global information to obtain multiple spatial features;

[0090] A fourth module is configured to perform a second fusion of the local time series information and the global time series information to obtain a time series feature;

[0091] The fifth module is used to determine the trajectory prediction information of the target pedestrian based on the multiple spatial features and the temporal features.

[0092] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0093] The memory is used to store programs;

[0094] The processor executes the program to implement the method described above.

[0095] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0096] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0097] The specific implementation process of the present invention is described in detail below with reference to the accompanying drawings:

[0098] This embodiment assumes that there are N pedestrians in a scene, given their positions from time T0 to T obs The problem to be solved is to predict their observation positions at T obs The future trajectory from time +1 to time T. At each time t, the position of each person is represented as Where i∈{1,…,N}. Assume that the distance between two pedestrians is less than d and There is an undirected edge between Connect these two people to show that there is an interactive relationship between them. This will give us an undirected graph G t =(V t ,E t ) represents the relationship between pedestrians in the scene at time t, where represents pedestrians in the scene, Represents the edges between pedestrians.

[0099] To solve this problem, the present invention proposes a method that combines time and space to directly extract the influence of pedestrians on surrounding pedestrians at different times. It mainly includes three parts: global and local information extraction, multi-space fusion and time information extraction. The flow chart is as follows Figure 2 As shown, the specific steps include:

[0100] The reason for introducing both global and local information is that, in real-world scenarios, their combined effectiveness is maximized. For example, in temporal feature extraction, the influence of historical coordinates far from the current moment on a pedestrian's subsequent trajectory can be neglected. Therefore, local information processing is incorporated into the temporal processing. Specifically, only the pedestrian's historical trajectory from a few moments prior to the current moment is selected to extract the impact of previous trajectories on future trajectories. The extracted feature vectors are then combined with the feature vectors obtained from the global context to produce the final temporal feature representation. In spatial feature extraction, typical models only consider pedestrians within a certain range to calculate their interactions with the pedestrian. Therefore, individuals outside this range are not considered in the overall scene. However, in some cases, a pedestrian may have a relationship with someone who is relatively distant, such as an acquaintance. This relationship can have a significant impact on the current individual, and in these cases, global spatial information should be considered. Therefore, combining local and global information is essential.

[0101] Multi-spatial fusion combines undirected graphs from different moments and feeds them into a multi-spatial transformer. This transformer receives the position information of each pedestrian in a continuous scene and determines the influence of surrounding pedestrians on each pedestrian at both the current and previous moments. This fusion of spatial information across different moments can better characterize the influence of surrounding pedestrians at different moments over a period of time, compared to separate temporal and spatial features.

[0102] The temporal information extraction part uses the temporal transformer to process temporal local information and temporal global information, and then combines the processed features to obtain the impact of the pedestrian's past historical trajectory on the future motion trajectory.

[0103] Finally, the features output by multiple spatial transformers and temporal transformers are combined and passed through a spatiotemporal decoder to obtain the final predicted trajectory.

[0104] It should be noted that the temporal Transformer module in the embodiment of the present invention uses the trajectory features of pedestrians As input, then output the updated trajectory features These trajectory features contain the dependency information between each pedestrian’s historical trajectories. In the self-attention module, the pedestrian’s trajectory features are mapped into three feature vectors. For each pedestrian i, there are:

[0105]

[0106] in, is the query vector, is the key vector, is a value vector, and f q , f k , f v These are the query mapping function, key mapping function, and value mapping function. The q, k, and v vectors at different times are combined to generate the Q, K, and V matrices. This module can simultaneously apply the historical trajectories of different pedestrians to calculate the Q, K, and V matrices of different pedestrians in parallel.

[0107] Multiple spatial Transformer modules such as Figure 3 As shown, it mainly receives the position information of each pedestrian in a continuous moment scene, as well as the adjacency matrix indicating whether there is an interactive relationship between them, and obtains the influence of the pedestrians around each pedestrian on him at the current and previous moments.

[0108] The Multi-Spatial Transformer module generally calculates the interaction information between pedestrians in a single frame and extends it to multiple frames. In practical applications, a person's walking trajectory is not only affected by other pedestrians at the current moment, but also by the historical trajectories of other pedestrians at previous moments. That is, the position probability of pedestrian i at time t+1 can be written as Where Nb(i) represents the set of pedestrians that have interaction relationships with pedestrian i.

[0109] For pedestrians at the same moment, the pedestrian i in the scene at a certain moment is the same as the pedestrian’s feature vector {h i The same process is performed on |i∈{1,…,N}} to obtain the query vector q i =f q (h i ), key vector k i =f k (h i ) and the value vector v i =f v (h i ). Next, we can define the influence of pedestrian j on pedestrian i in this case

[0110]

[0111] Then, the Transformer mechanism is extended to the entire image using the graph convolution method. Assume that the set of adjacent pedestrians connected to pedestrian i is Nb(i). This set can be represented by the adjacency matrix A of all pedestrians in the image. t To express, where A t The elements are determined by the following formula

[0112]

[0113] Then in the entire scene, the interaction relationship between all people connected to pedestrian i and pedestrian i can be calculated at the same time:

[0114]

[0115] When several images at different times are merged together, the pedestrians at different times can be connected in the same way as described above, and the influence of pedestrians at previous times on pedestrians at later times can be calculated. At this time, a larger adjacency matrix will be obtained, which can be used in Figure 4 It can be seen intuitively that the elements in this adjacency matrix can be obtained by generalizing the previous formula and defining Elements in

[0116]

[0117] This embodiment defines The influence of pedestrians connected to pedestrian i at time t2 on pedestrian i at time t1 can be calculated by the following formula:

[0118]

[0119] It can be understood that the set of adjacent pedestrians Nb(i) of pedestrian i is determined based on the previously calculated adjacency matrix. In Nb(i), if the value in row i and column j is 1, then person j is in Nb(i).

[0120] Global information and local information can be obtained through Figure 5 Intuitively, the final features can be expressed as follows:

[0121] Total(i)=Global(i)+Local(i)

[0122] Among them, Global(i) represents the global information of pedestrian i (including spatial global information or temporal global information), and Local(i) represents the local information of pedestrian i (including spatial local information or spatial global information).

[0123] In summary, compared to existing technologies, this method first incorporates both global and local information into pedestrian trajectory prediction, fully leveraging information from pedestrians' historical trajectories. It then merges multiple spatial graphs, extracting information that is fused together temporally and spatially. Because pedestrians are subject to synchronous temporal and spatial influences during movement, this method can better simulate pedestrian interactions in real-world scenarios, resulting in more accurate pedestrian trajectory predictions.

[0124] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0125] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0126] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0127] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0128] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0129] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0130] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0131] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0132] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. Pedestrian trajectory prediction method based on multiple spatial graphs and time fusion, characterized by: include: Obtaining a set of pedestrian trajectories of each pedestrian in a target scene, wherein the pedestrian trajectory includes spatial position information and time information of the pedestrian; Extracting spatial local information, spatial global information, temporal local information and temporal global information from the pedestrian trajectory set; performing a first fusion of the spatial local information and the spatial global information at different times to obtain multiple spatial features, wherein the multiple spatial features represent the influence of surrounding pedestrians at different times on the pedestrian at the current and previous moments; Performing a second fusion of the local time series information and the global time series information to obtain a time series feature; Determining trajectory prediction information of a target pedestrian based on the multiple spatial features and the temporal features; The step of extracting spatial local information and spatial global information from the pedestrian trajectory set includes: Obtaining position information of each pedestrian at consecutive moments from the pedestrian trajectory set; Obtain the adjacency matrix of the interaction relationships between pedestrians; Determine the influence of other pedestrians around each pedestrian based on the position information and the adjacency matrix; The first fusing of the spatial local information and the spatial global information at different times to obtain multiple spatial features includes the following steps: Merge the undirected graphs constructed at different times and input them into multiple spatial transformers; The multi-spatial Transformer calculates the influence of pedestrians at previous moments on pedestrians at later moments between multiple frames based on the position information of each pedestrian at consecutive moments, thereby obtaining the multi-spatial features. The determining, based on the position information and the adjacency matrix, the influence of other pedestrians around each pedestrian includes: Get the target image frame; Calculating the influence of other pedestrians around each pedestrian in the target image frame according to the position information and the adjacency matrix; The influence of other pedestrians around each pedestrian in the target image frame is extended to other multiple image frames by defining Calculate the influence of pedestrians in the previous moment on pedestrians in the later moment, and obtain the influence of each pedestrian on other pedestrians around him at the current moment and historical moments; The calculation formula of the influence of the pedestrians at the previous moment on the pedestrians at the later moment is: ； in, represent Pedestrians at all times Pedestrians at all times The size of the influence, Represents the previous moment, Represents the later moment, represent Pedestrians at all times The query vector, represent Pedestrians The key vector of Representative pedestrians The value vector of The problem dimensions representing the query vector, key vector, and value vector, It is used to normalize the weights of different pedestrians on pedestrian i so that their sum is 1. Indicates pedestrian A collection of pedestrians with interactive relationships.

2. The pedestrian trajectory prediction method based on multiple spatial graphs and time fusion according to claim 1 is characterized in that: The step of obtaining a set of pedestrian trajectories of each pedestrian in the target scene includes: Obtain the observed position information of each pedestrian in the target scene from the first moment to the second moment; Connecting any two pedestrians in the target scene that meet a preset distance condition through an undirected edge to determine the interaction relationship between the two pedestrians; A pedestrian trajectory set is constructed according to the observation position information and the interaction relationship.

3. The pedestrian trajectory prediction method based on multiple spatial graphs and time fusion according to claim 2 is characterized in that: The step of obtaining a set of pedestrian trajectories of each pedestrian in the target scene further includes: Mapping the pedestrian trajectory set into a query vector, a key vector, and a value vector; Processing the query vector through a query mapping function to construct a pedestrian query matrix; Processing the key vector through a key mapping function to construct a key matrix for the pedestrian; The value vector is processed by a value mapping function to construct a value matrix of the pedestrian.

4. The pedestrian trajectory prediction method based on multiple spatial graphs and time fusion according to claim 1, characterized in that: The method further comprises: The adjacent pedestrian set of each pedestrian is determined by graph convolution, and the adjacent pedestrian set includes the adjacency matrix of all pedestrians.

5. The pedestrian trajectory prediction method based on multiple spatial graphs and time fusion according to claim 1 is characterized in that: The calculation formula for the influence of other pedestrians around each pedestrian in the target image frame is: in, Representative pedestrians For pedestrians the size of its influence; Representative pedestrians The query vector of Representative pedestrians The key vector of Representative pedestrians A vector of values ​​for ; The question dimensions representing the query vector, key vector, and value vector; Used to normalize the weights of different pedestrians on pedestrian i so that their sum is 1; The calculation formula of the multiple spatial features is: in, Represents multiple spatial features; Indicates pedestrians Global information; Indicates pedestrians local information.

6. Pedestrian trajectory prediction device based on multiple spatial graphs and time fusion, characterized by: include: The first module is used to obtain a set of pedestrian trajectories of each pedestrian in the target scene, wherein the pedestrian trajectory includes the spatial position information and time information of the pedestrian; The second module is used to extract spatial local information, spatial global information, temporal local information and temporal global information from the pedestrian trajectory set; A third module is configured to first fuse the spatial local information and the spatial global information at different times to obtain multiple spatial features, wherein the multiple spatial features represent the influence of surrounding pedestrians at different times on the pedestrian at the current and previous moments; A fourth module is configured to perform a second fusion of the local time series information and the global time series information to obtain a time series feature; A fifth module is configured to determine trajectory prediction information of a target pedestrian based on the multiple spatial features and the temporal features; The step of extracting spatial local information and spatial global information from the pedestrian trajectory set includes: Obtaining position information of each pedestrian at consecutive moments from the pedestrian trajectory set; Obtain the adjacency matrix of the interaction relationships between pedestrians; Determine the influence of other pedestrians around each pedestrian based on the position information and the adjacency matrix; The first fusing of the spatial local information and the spatial global information at different times to obtain multiple spatial features includes the following steps: Merge the undirected graphs constructed at different times and input them into multiple spatial transformers; The multi-spatial Transformer calculates the influence of pedestrians at previous moments on pedestrians at later moments between multiple frames based on the position information of each pedestrian at consecutive moments, thereby obtaining the multi-spatial features. The determining, based on the position information and the adjacency matrix, the influence of other pedestrians around each pedestrian includes: Get the target image frame; Calculating the influence of other pedestrians around each pedestrian in the target image frame according to the position information and the adjacency matrix; The influence of other pedestrians around each pedestrian in the target image frame is extended to other multiple image frames by defining Calculate the influence of pedestrians in the previous moment on pedestrians in the later moment, and obtain the influence of each pedestrian on other pedestrians around him at the current moment and historical moments; The calculation formula of the influence of the pedestrians at the previous moment on the pedestrians at the later moment is: ； in, represent Pedestrians at all times Pedestrians The size of the influence, Represents the previous moment, Represents the later moment, represent Pedestrians The query vector, represent Pedestrians The key vector of Representative pedestrians The value vector of The problem dimensions representing the query vector, key vector, and value vector, It is used to normalize the weights of different pedestrians on pedestrian i so that their sum is 1. Indicates pedestrian A collection of pedestrians with interactive relationships.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images