Motion trajectory generation method and device, computer device, and storage medium

Abstract

The application relates to a motion trajectory generation method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring multiple images, generating spatial position representations of multiple objects in each image, nodes of the spatial position representations are used to represent initial position features of the objects in the images, and corresponding spatial adjacency matrices of the spatial position representations are used to represent spatial position relationships between the objects; taking each image and a corresponding adjacent image of each image as an image group, fusing multiple spatial position representations corresponding to each image group, and generating a space-time representation of each image group; performing graph convolution processing on the space-time representation of each image group to generate target position features of the multiple objects; and generating a motion trajectory of each object within a future time according to the target position features. The method can make the target position features have spatial features and time features of the objects at the same time, thereby improving the accuracy of motion trajectory generation.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a motion trajectory generation method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] Autonomous driving technology is a technology that can sense and navigate the dynamic changes in the surrounding environment of transportation equipment. Dynamic changes in the surrounding environment include the position changes of pedestrians over time. Therefore, it is necessary to predict the position changes of pedestrians and generate their movement trajectories.

[0003] In the traditional method, the motion trajectories of all pedestrians in the scene are encoded by using a long short-term memory network, the motion features of the motion trajectories are obtained, and the adjacency matrix of the motion trajectories is constructed. The motion features and the adjacency matrix of the motion trajectories are input into the graph convolutional network to obtain the interaction features of the motion trajectories. The motion features and interaction features of the motion trajectories are decoded using a long short-term memory network to obtain the motion trajectories of the pedestrians.

[0004] However, with the development of deep learning technology, there is an urgent need for a motion trajectory generation method with high accuracy. Summary of the Invention

[0005] Based on this, it is necessary to provide a motion trajectory generation method, device, computer equipment, computer-readable storage medium and computer program product with high accuracy to address the above technical problems.

[0006] In a first aspect, the present application provides a method for generating a motion trajectory. The method comprises:

[0007] Acquire multiple frames of images and generate spatial position representations of multiple objects in each frame of the image, wherein nodes of the spatial position representations are used to represent initial position features of the objects in the image, and a spatial adjacency matrix corresponding to the spatial position representations is used to represent spatial position relationships between the objects;

[0008] taking each frame of the image and the adjacent images corresponding to each frame of the image as an image group, fusing the multiple spatial position representations corresponding to each image group, and generating a spatiotemporal representation of each image group;

[0009] performing graph convolution processing on the spatiotemporal representation of each of the image groups to generate target position features of a plurality of the objects;

[0010] A motion trajectory of each of the objects in the future is generated according to the target position features.

[0011] In one embodiment, fusing the multiple spatial position representations corresponding to each image group to generate a spatiotemporal representation of each image group includes:

[0012] fusing the plurality of spatial adjacency matrices corresponding to each of the image groups to obtain a fused spatial adjacency matrix;

[0013] Processing the fused spatial adjacency matrix according to the number of images in the image group to obtain a spatiotemporal adjacency matrix corresponding to each image group, wherein the spatiotemporal adjacency matrix is ​​used to represent the spatiotemporal positional relationship between nodes on different frame images in the image group;

[0014] The spatiotemporal representation corresponding to each image group is generated according to the nodes included in each image group and the spatiotemporal adjacency matrix corresponding to each image group.

[0015] In one embodiment, the step of forming each frame of the image and the adjacent images corresponding to each frame of the image as an image group includes:

[0016] Determine a first adjacent image captured before the capture moment of each frame of the image, and a second adjacent image captured after the capture moment of each frame of the image, and take each frame of the image, the first adjacent image corresponding to each frame of the image, and the second adjacent image as a group of the image group, wherein the first adjacent image and the second adjacent image have the same number of frames.

[0017] In one embodiment, generating a motion trajectory of each of the objects in the future based on the target position features includes:

[0018] Performing time convolution processing on the target position features to obtain position distribution features of the object in the future;

[0019] The position distribution features are sampled to generate the motion trajectory of the object in the future.

[0020] In one embodiment, generating a spatial position representation of multiple objects in each frame of the image includes:

[0021] Determining, based on the position information of each object in the multiple frames of the image, displacement information of each object between each frame of the image and the previous frame of the image, and using the displacement information as an initial position feature of the object in each frame of the image;

[0022] Determining distances between two of the objects in each frame of the image based on position information of the plurality of objects in each frame of the image, and generating a spatial adjacency matrix corresponding to each frame of the image based on the distances;

[0023] The spatial position representation of the plurality of objects in each frame of the image is generated according to the initial position features and the spatial adjacency matrix.

[0024] In one embodiment, the method further comprises:

[0025] The multiple objects respectively included in each two adjacent frames of images are compared, and when there is a first object that does not exist in both adjacent frames of images, the position information of the first object is deleted.

[0026] In a second aspect, the present application further provides a motion trajectory generating device. The device comprises:

[0027] a spatial representation generation module, configured to acquire multiple frames of images and generate spatial position representations of multiple objects in each frame of the image, wherein the nodes of the spatial position representations are used to represent the initial position features of the objects in the image, and the spatial adjacency matrix corresponding to the spatial position representations is used to represent the spatial position relationships between the objects;

[0028] a spatiotemporal representation generation module, configured to treat each frame of the image and its corresponding adjacent image as an image group, fuse the multiple spatial position representations corresponding to each image group, and generate a spatiotemporal representation of each image group;

[0029] The motion trajectory generation module is used to perform graph convolution processing on the spatiotemporal representation of each of the image groups to generate target position features of multiple objects, and generate a motion trajectory of each of the objects in the future based on the target position features.

[0030] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the motion trajectory generation method described in any one of the embodiments of the first aspect.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the motion trajectory generation method described in any one of the embodiments of the first aspect.

[0032] In a fifth aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, implements the motion trajectory generation method described in any one of the embodiments of the first aspect.

[0033] The above-mentioned motion trajectory generation method, device, computer equipment, storage medium and computer program product obtain multiple frames of images to generate spatial position representations of multiple objects in each frame of image, take each frame of image and the adjacent image corresponding to each frame of image as an image group, fuse the multiple spatial position representations corresponding to each image group, generate the spatiotemporal representation of each image group, perform graph convolution processing on the spatiotemporal representation of each image group, generate target position features of multiple objects, generate the motion trajectory of each object in the future time based on the target position features, and can fuse the spatial features of the object in each frame of image and the temporal features of the object between multiple frames of image into the spatiotemporal representation corresponding to the image group, so that the target position features have both the spatial features and the temporal features of the object, thereby improving the accuracy of motion trajectory generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A diagram showing an application environment of a motion trajectory generation method according to an embodiment;

[0035] Figure 2 1 is a flow chart of a motion trajectory generating method according to an embodiment;

[0036] Figure 3a 1 is a flow chart of the steps of generating a spatiotemporal representation in one embodiment;

[0037] Figure 3b is a schematic diagram of a step of generating a spatiotemporal representation in one embodiment;

[0038] Figure 4 1 is a flow chart of a step of generating a spatial position representation in one embodiment;

[0039] Figure 5a is a flow chart of a motion trajectory generating method according to another embodiment;

[0040] Figure 5b is a schematic diagram of a motion trajectory in one embodiment;

[0041] Figure 6 is a structural block diagram of a motion trajectory generating device in one embodiment;

[0042] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0044] The motion trajectory generation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 obtains multiple frames of images captured by the terminal 102, generates spatial position representations of multiple objects in each frame of image, and uses each frame of image and the adjacent images corresponding to each frame of image as an image group, fuses the multiple spatial position representations corresponding to each image group, generates a spatiotemporal representation of each image group, performs graph convolution processing on the spatiotemporal representation of each image group, generates target position features of multiple objects, and generates a motion trajectory of each object in the future time based on the target position features.

[0045] The terminal 102 may be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart TVs, smart car devices, etc. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0046] In one embodiment, Figure 2 As shown, a motion trajectory generation method is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0047] Step S202: Acquire multiple frames of images and generate spatial position representations of multiple objects in each frame of image.

[0048] The image may be acquired by a terminal device in real time for a fixed scene, or may be pre-stored in a server or on the Internet.

[0049] Objects can be used to represent objects in motion, such as pedestrians, vehicles, animals, etc.

[0050] The spatial position representation consists of multiple nodes, each corresponding to an object, representing the initial position characteristics of the object in the image. Initial position characteristics may include, but are not limited to, displacement and coordinates. The spatial adjacency matrix corresponding to the spatial position representation is used to represent the spatial positional relationships between multiple objects. Spatial positional relationships may include, but are not limited to, relative positional relationships between multiple objects and distances between multiple objects.

[0051] Specifically, in response to the motion trajectory generation request, the server obtains multiple frames of images, performs target detection on each frame of image, and determines the position information of the object in each frame of image. The position information of the object between two adjacent frames of image is processed to determine the initial position features of the object, and obtain the node corresponding to the object in each frame of image. The initial position features of each pair of objects in each frame of image are processed to obtain the spatial position relationship between each pair of objects, and the spatial position relationship between each pair of objects is used as the edge between each pair of nodes to generate a spatial adjacency matrix corresponding to the image. Based on the multiple nodes corresponding to each frame of image and the spatial adjacency matrix, a spatial position representation corresponding to each frame of image is generated. The motion trajectory generation request can be manually triggered by the user, for example, the user clicks the corresponding motion trajectory generation button on the page to trigger the motion trajectory generation request; it can also be automatically triggered by the server, for example, when the server detects that multiple frames of image have been uploaded, it automatically triggers the motion trajectory generation request.

[0052] Step S204 : taking each frame image and the adjacent images corresponding to each frame image as an image group, fusing the multiple spatial position representations corresponding to each image group to generate a spatiotemporal representation of each image group.

[0053] Among them, the spatiotemporal representation can be used to characterize the temporal characteristics of the object in each frame image and between adjacent images corresponding to each frame image.

[0054] Specifically, the server determines the adjacent images corresponding to each frame based on the acquisition time corresponding to each frame, and forms an image group with each frame and its adjacent images. The server then fuses the spatial position representations corresponding to each frame in the image group, obtaining the relative positional relationships between each pair of nodes corresponding to different frames in the fused spatial positional representations. Based on the relative positional relationships between each pair of nodes, a spatiotemporal representation corresponding to each image group is generated.

[0055] Step S206: performing graph convolution processing on the spatiotemporal representation of each image group to generate target position features of multiple objects.

[0056] The target position feature may be used to characterize the distribution characteristics of a plurality of initial position features corresponding to each object in the image group.

[0057] Specifically, the server can perform graph convolution processing on the spatiotemporal representation of each image group using a graph convolutional network, obtain the position information of each pair of nodes in the spatiotemporal representation of each image group, and determine the edges between each pair of nodes in the spatiotemporal representation. Graph convolution processing is performed on each node in the spatiotemporal representation and the edges between each pair of nodes to obtain the target position information corresponding to the node after graph convolution processing, and use the target position information corresponding to the node as the target position feature of the object.

[0058] Step S208: Generate a motion trajectory of each object in the future based on the target position features.

[0059] Specifically, the server pre-stores motion trajectory generation logic. This motion trajectory generation logic is used to process the target position features of each object to generate the motion trajectory of each object in the future. The motion trajectory generation logic can be: performing computations on the object's initial position features and target position features to generate the object's motion trajectory in the future; or, it can be: performing temporal convolution processing on the target position features to generate the object's motion trajectory in the future.

[0060] In the above-mentioned motion trajectory generation method, by acquiring multiple frames of images, the spatial position representation of multiple objects in each frame of image is generated, each frame of image and the adjacent image corresponding to each frame of image are taken as an image group, the multiple spatial position representations corresponding to each image group are fused, and the spatiotemporal representation of each image group is generated. The spatiotemporal representation of each image group is subjected to graph convolution processing to generate target position features of multiple objects, and the motion trajectory of each object in the future time is generated based on the target position features. The spatial features of the object in each frame of image and the temporal features of the object between multiple frames of image can be fused into the spatiotemporal representation corresponding to the image group, so that the target position features have both the spatial features and the temporal features of the object, thereby improving the accuracy of motion trajectory generation.

[0061] In one embodiment, Figure 3a As shown, step S204, each frame image and the adjacent images corresponding to each frame image are taken as an image group, and multiple spatial position representations corresponding to each image group are fused to generate a spatiotemporal representation of each image group, including:

[0062] Step S302 : Fusing multiple spatial adjacency matrices corresponding to each image group to obtain a fused spatial adjacency matrix.

[0063] Specifically, the server obtains multiple spatial position representations corresponding to the image groups and determines the spatial adjacency matrix corresponding to each spatial position representation. The multiple spatial adjacency matrices corresponding to each image group are fused together to determine the distances between nodes in the spatial position representations corresponding to different frames of imagery, generating a fused spatial adjacency matrix.

[0064] Step S304 : Processing the fused spatial adjacency matrix according to the number of images in the image group to obtain a spatiotemporal adjacency matrix corresponding to each image group.

[0065] Step S306 : generating a spatiotemporal representation corresponding to each image group according to the nodes included in each image group and the spatiotemporal adjacency matrix corresponding to each image group.

[0066] The spatiotemporal adjacency matrix is ​​used to represent the spatiotemporal position relationship between nodes in different frame images in an image group.

[0067] Specifically, the server obtains the number of images in each image group and processes the fused spatial adjacency matrix based on the number of images, so that the dimension of the processed spatial adjacency matrix is ​​the same as the number of images in the image group. The processed spatial adjacency matrix is ​​used as the spatiotemporal adjacency matrix corresponding to each image group. The server obtains multiple nodes contained in each image group and generates a spatiotemporal representation corresponding to each image group based on the nodes contained in each image group and the spatiotemporal adjacency matrix corresponding to each image group.

[0068] In an example, the spatiotemporal adjacency matrix can be obtained by referring to the following formula:

[0069]

[0070] in, is the spatiotemporal adjacency matrix corresponding to an image group (the image group includes

[0071] Frame image to frame images), τ is the number of images in the image group (τ value can be any one of 3, 5, 7, ..., 2n+1, where n is a positive integer greater than 0), t represents the t-th frame image (t is a positive integer greater than 0), Represents the spatial adjacency matrix corresponding to the adjacent image before the t-th frame image, Represents the spatial adjacency matrix corresponding to the adjacent images after the t-th frame image, is a constant, T represents time, N represents the number of objects, and C represents the number of channels.

[0072] In an example, the spatiotemporal representation can be generated according to the following formula:

[0073] G (τ) =(V (τ) ,E (τ) )

[0074] V (τ) =V1…∪V τ

[0075] Among them, G (τ) is the spatiotemporal representation of the image group, V (τ) is the node in the spatial position representation corresponding to each frame image in the image group, E (τ) The spatiotemporal adjacency matrix corresponding to each image group is is a set of images, τ is the number of images in the image group.

[0076] In one example, the server may group three consecutive frames of images (ie, τ=3) into an image group, such as Figure 3b As shown, the t-th frame image, the t-1th frame image corresponding to the t-th frame image, and the t+1th frame image are taken as an image group. The spatial adjacency matrix corresponding to the t-1th frame image is fused The spatial adjacency matrix corresponding to the t-th frame And the spatial adjacency matrix corresponding to the t+1th frame The fused spatial adjacency matrix is ​​obtained. The fused spatial adjacency matrix is ​​processed according to the number of images in the image group so that the processed spatial adjacency matrix is ​​three times the spatial adjacency matrix before processing. The processed spatial adjacency matrix is ​​used as the spatiotemporal adjacency matrix corresponding to the image group.

[0077] In this embodiment, by fusing the spatial adjacency matrix corresponding to the image group, a spatiotemporal adjacency matrix corresponding to the image group is constructed. Based on the nodes contained in the image group and the spatiotemporal adjacency matrix corresponding to the image group, a spatiotemporal representation corresponding to the image group is generated. By constructing the spatiotemporal adjacency matrix, the positional features of objects on multiple frames of images can be fused to generate a spatiotemporal representation that has both the spatial features and the temporal features of the objects.

[0078] In one embodiment, step S204 treats each frame image and the adjacent images corresponding to each frame image as an image group, fuses multiple spatial position representations corresponding to each image group, and generates a spatiotemporal representation of each image group, including: determining a first adjacent image acquired before the acquisition moment of each frame image, and a second adjacent image acquired after the acquisition moment of each frame image, treating each frame image, the first adjacent image corresponding to each frame image, and the second adjacent image as a group of image groups, wherein the number of frames of the first adjacent image and the second adjacent image is the same.

[0079] In one example, the number of frames of the first adjacent images may be one, and an image group consisting of each frame, its corresponding first adjacent image, and its corresponding second adjacent image includes three frames. In another example, the number of frames of the first adjacent images may be two, and an image group consisting of each frame, its corresponding first adjacent image, and its corresponding second adjacent image includes five frames.

[0080] In one example, when the target image is the first frame, a blank frame is used as the first adjacent image corresponding to the target object. When the target image is the last frame, a blank frame is used as the second adjacent image corresponding to the target object.

[0081] In this embodiment, by treating the first adjacent image before each frame image, each frame image, and the second adjacent image after each frame image as a group of image groups, a connection can be established between each frame image and the nodes in the upper and lower frame images corresponding to each frame image, thereby improving the accuracy of motion trajectory generation.

[0082] In one embodiment, step S208 generates the motion trajectory of each object in the future time based on the target position features, including: performing time convolution processing on the target position features, obtaining the position distribution features of the object in the future time, sampling the position distribution features, and generating the motion trajectory of the object in the future time.

[0083] The position distribution feature may be, but is not limited to, any one of a binary Gaussian distribution feature, a Poisson distribution feature, and a Bernoulli distribution feature.

[0084] Specifically, the server can input the target position features of the object into the temporal convolutional network for temporal convolution processing to obtain the position distribution features of the object in the future time. The position distribution features are sampled to determine the position information of the object in the future time, and the movement trajectory of the object in the future time is obtained.

[0085] In an example, the following formula can be used to obtain the location distribution characteristics of an object in the future:

[0086] V pred =PRELU(TCN(V st ))+V st

[0087] Among them, V pred is the position distribution feature of the object in the future time, PRELU is an activation function, TCN is the Temporal Convolutional Network, V st is the target location feature.

[0088] In an example, when the generated position distribution feature is a binary Gaussian distribution feature, the position distribution feature can be sampled according to the following formula to generate the object's motion trajectory in the future:

[0089]

[0090] in, is the motion trajectory of the object in the future, W is the weight (which may include the weight of graph convolution processing and the weight of time convolution processing), is the mean of the location distribution characteristics, is the variance of the location distribution feature, is the covariance of the location distribution features.

[0091] In this embodiment, by performing temporal convolution processing on the target position features, the motion trajectory of the object in the future time is generated, and the target position features of the object at different acquisition moments can be utilized, thereby improving the accuracy of the motion trajectory.

[0092] In one embodiment, Figure 4 As shown, step S202, acquiring multiple frames of images and generating spatial position representations of multiple objects in each frame of image, includes:

[0093] Step S402 : determining the displacement information of each object between each frame and the previous frame according to the position information of each object in the multiple frames of images, and using the displacement information as the initial position feature of the object in each frame of images.

[0094] Specifically, the server acquires multiple frames of images, performs target detection on each frame, obtains the position information of each object in each frame, and generates an identifier corresponding to each object. Based on the identifier, the server determines the position information of the same object between two adjacent frames. This position information between the two adjacent frames is then processed to determine the displacement of the object between each frame and the previous frame. This displacement information is used as the initial position feature of the object in each frame.

[0095] In one example, when the target image is the first frame image, a blank frame image is used as the previous frame image of the frame to determine the displacement information of the object on the first frame image.

[0096] Step S404 : determining the distance between any two objects in each frame of image based on the position information of the multiple objects in each frame of image, and generating a spatial adjacency matrix corresponding to each frame of image based on the distance.

[0097] Specifically, the server obtains the position information of objects in each frame and performs computations on the position information of each pair of objects in each frame to determine the distance between each pair of objects in each frame. The server then weights the distances between each pair of objects in each frame using a preset threshold to determine the correlation between each pair of objects and generate a spatial adjacency matrix corresponding to each frame.

[0098] In one example, to obtain the correlation between objects in an image, the server sets the weights between each object to a value greater than zero. When using the Euclidean norm to weight the distance, the spatial adjacency matrix corresponding to the image can be determined by referring to the following formula:

[0099]

[0100]

[0101] in, represents the correlation between the i-th object and the j-th object on the image obtained by the Euclidean norm, represents the initial position feature of the i-th object on the image, Represents the initial position feature of the jth object on the image, ∈ is the minimum value, which is used to make the denominator non-zero, that is, the correlation between the two objects A t is the spatial adjacency matrix corresponding to the t-th frame image.

[0102] In one example, the server may normalize the spatial adjacency matrix and use the normalized spatial adjacency matrix for subsequent processing.

[0103]

[0104] in, is the spatial adjacency matrix corresponding to the t-th frame image after normalization, is the reversible degree matrix, A t is the spatial adjacency matrix corresponding to the t-th frame image before normalization, and I is the identity matrix.

[0105] Step S406 : Generate spatial position representations of multiple objects in each frame of image based on the initial position features and the spatial adjacency matrix.

[0106] Specifically, the server generates spatial position representations of multiple objects in each frame of image based on the initial position features of multiple objects in each frame of image and the spatial adjacency matrix corresponding to the image.

[0107] In an example, the spatial position representation can be generated according to the following formula:

[0108] G t ={V t ,E t}

[0109]

[0110]

[0111] Among them, G t is the spatial position representation, V t is the initial position feature of the object, is the displacement of the i-th object in the image relative to the previous frame, E t is the spatial adjacency matrix, is the correlation between the i-th object and the j-th object in the image.

[0112] In this embodiment, by using the position information of the object in the image to determine the displacement of the object between two adjacent frames, the movement speed and direction of the object in continuous time can be determined; by determining the correlation between objects based on the distance between each two objects in each frame of the image and generating a spatial adjacency matrix, the interaction process between multiple objects can be learned; by generating the spatial position representation corresponding to the image based on the displacement and the spatial adjacency matrix, the accuracy of the generation of the spatial position representation can be improved.

[0113] In one embodiment, the motion trajectory generation method further includes: comparing multiple objects respectively included in each two adjacent frames of images, and when a first object does not exist in both adjacent frames of images, deleting the position information of the first object.

[0114] Specifically, the server acquires multiple image frames, performs target detection on each frame, identifies the objects in each frame, and generates identifiers corresponding to the objects. It then acquires two adjacent frames and compares the identifiers of multiple objects in the two frames. If the server determines that the identifier corresponding to a first object exists only in one of the frames, it deletes the location information of the first object in that frame. The server then uses the deleted image to generate motion trajectories.

[0115] In one example, the difference between the acquisition times of two adjacent frames of image may be 0.4 seconds.

[0116] In this embodiment, by deleting the position information of objects that do not exist simultaneously in two adjacent frames of images, the position information of the same object in multiple frames of images can be collected in a continuous time, thereby improving the accuracy of the subsequently generated object motion trajectory in the future time.

[0117] In one embodiment, Figure 5a As shown, a motion trajectory generation method is provided, comprising:

[0118] Step S502 : acquiring multiple frames of images, comparing multiple objects respectively included in two adjacent frames of images, and deleting position information of the first object when there is a first object that does not exist in both adjacent frames of images.

[0119] Step S504 : determining displacement information of each deleted object between each frame and the previous frame according to the position information of each deleted object in the multiple frames of images.

[0120] Step S506 , determining the distance between any two objects in each frame of image based on the position information of the deleted objects in each frame of image, and generating a spatial adjacency matrix corresponding to each frame of image based on the distance.

[0121] Step S508 : generating spatial position representations of multiple objects in each frame of image based on the initial position features and the spatial adjacency matrix.

[0122] Specifically, the server obtains multiple frames of images, performs target detection on each frame of image, determines the position information of each object, and generates an identifier corresponding to the object. The identifiers of multiple objects included in two adjacent frames of image are determined, the identifiers of objects that do not exist in two adjacent frames of image at the same time are determined, and the position information of the object is deleted. The position information of the same object in the multiple frames of image after deletion is processed, and the displacement information generated between each frame of image and the previous frame of image of each frame after deletion is determined. The position information of the multiple objects in each frame of image after deletion is processed, and the distance between each two objects in each frame of image is determined, and the correlation between each two objects is obtained. The spatial adjacency matrix corresponding to each frame of image is generated based on the correlation. Based on the initial position features of the multiple objects and the spatial adjacency matrix, the spatial position representation of the multiple objects in each frame of image is generated. The specific spatial position representation generation operation can be implemented with reference to the method provided in the above embodiment, and will not be elaborated here.

[0123] In step S510 , each frame image, the first adjacent image corresponding to each frame image, and the second adjacent image corresponding to each frame image are taken as a group of image groups, and multiple spatial adjacency matrices corresponding to each image group are fused to obtain a fused spatial adjacency matrix.

[0124] Step S512 : processing the fused spatial adjacency matrix according to the number of images in the image group to obtain a spatiotemporal adjacency matrix corresponding to each image group.

[0125] Step S514 : generating a spatiotemporal representation corresponding to each image group according to the nodes included in each image group and the spatiotemporal adjacency matrix corresponding to each image group.

[0126] Specifically, the server takes each frame image, the first adjacent image collected before each frame image, and the second adjacent image collected after each frame image as a group of image groups, fuses multiple spatial adjacency matrices corresponding to each image group, determines the correlation between each pair of nodes on different frame images, and obtains the fused spatial adjacency matrix. According to the number of images in the image group, the fused spatial adjacency matrix is ​​processed to obtain the spatiotemporal adjacency matrix corresponding to each image group. According to the nodes contained in each image group and the spatiotemporal adjacency matrix corresponding to each image group, the spatiotemporal representation corresponding to each image group is generated. The specific spatiotemporal representation generation operation can be implemented with reference to the method provided in the above embodiment and will not be elaborated here.

[0127] Step S516: perform graph convolution processing on the spatiotemporal representation of each image group to generate target position features of multiple objects, and perform time convolution processing on the target position features of each object to generate the motion trajectory of the object in the future time.

[0128] Specifically, the server performs graph convolution on multiple nodes in the spatiotemporal representation with the corresponding spatiotemporal adjacency matrix to generate target position features for multiple objects. Temporal convolution is performed on each object's target position feature to generate a position distribution feature for the object in the future. These position distribution features are then sampled to generate the object's motion trajectory in the future.

[0129] In one example, the server may use image processing technology and video calibration technology to process each frame of image and determine the position information of each object in each frame of image.

[0130] In one example, the server can use a unified spatiotemporal modeling network to replace the convolution part in the traditional graph convolutional network, and combine the replaced graph convolutional network with the temporal convolutional network to obtain a spatiotemporal graph convolutional network that can simultaneously obtain the temporal and spatial features of the object.

[0131] In one example, the server can use the G3D network (a unified spatiotemporal modeling network) to replace the convolution part of the GCN network (graph convolutional network, a graph convolutional network) and combine it with the TCN network (temporal convolutional network, a temporal convolutional network) to obtain a spatiotemporal graph convolutional network. Taking this spatiotemporal graph convolutional network as an example, the network training process is explained:

[0132] Twenty image samples are acquired, and the first eight frames are used to perform the following operations: the displacement of the object sample in each frame relative to the previous frame, as well as the correlation between multiple object samples in each frame, are determined. The displacement and correlation between the object samples in each frame are then fed into an untrained spatiotemporal graph convolutional network (GCN). A spatial position representation corresponding to each frame is then constructed using a GCN network. Each image sample, the first adjacent image sample before the acquisition moment of each frame, and the second adjacent image sample after the acquisition moment of each frame are fused to generate eight image sample groups. Each image sample group is processed using a G3D network to generate a spatiotemporal representation corresponding to each image sample group. Each spatiotemporal representation is then convolved to obtain a target position feature for each object sample. The target position feature for each object sample is then fed into a TCN network to generate a predicted position distribution feature for the twelve frames following the first eight frames. The predicted position distribution feature is then compared with the actual position distribution feature of each object sample in the twelve frames following the first eight frames. The average Euclidean distance error and the final Euclidean distance error between the predicted and actual position distribution features are determined. The weights of the spatiotemporal graph convolutional network are adjusted according to the average Euclidean distance error and the final Euclidean distance error until the distance error between the predicted position distribution features and the actual position distribution features output by the adjusted spatiotemporal graph convolutional network meets the error condition, and the trained spatiotemporal graph convolutional network is obtained.

[0133] In one example, the motion trajectory generation method provided in the embodiment of the present application can simultaneously model all pedestrians in a fixed scene, and generate the weight of the interaction between the first pedestrian far from the target pedestrian and the target pedestrian, and the weight of the interaction between the second pedestrian close to the target pedestrian and the target pedestrian according to the distance between the multiple pedestrians. This makes the motion trajectory generation method provided in the embodiment of the present application suitable for pedestrian motion trajectory prediction in complex and crowded environments, and can effectively solve the problem of difficult pedestrian motion trajectory prediction in environments with large pedestrian flow. Figure 5b As shown, the actual motion trajectories of pedestrians and the predicted motion trajectories generated by different methods are provided under different numbers of pedestrians in multiple fixed scenes. Among them, the straight line is the actual motion trajectory of the pedestrian, the first dotted line composed of short lines represents the motion trajectory generated by the SGAN network (Semi-Supervised Learning with Generative Adversarial Networks, semi-supervised adversarial network) method, the second dotted line composed of short lines and dots represents the motion trajectory generated by the linear method, and the third dotted line composed of dots represents the motion trajectory generated by the method provided in the embodiment of the present application. After comparison, it can be seen that the accuracy of the motion trajectory generated by the method provided in the embodiment of the present application is higher.

[0134] In this embodiment, by acquiring multiple frames of images, spatial position representations of multiple objects in each frame of image are generated, each frame of image and the adjacent images corresponding to each frame of image are taken as an image group, the multiple spatial position representations corresponding to each image group are fused, and the spatiotemporal representation of each image group is generated. The spatiotemporal representation of each image group is subjected to graph convolution processing to generate target position features of multiple objects, and the motion trajectory of each object in the future time is generated based on the target position features. The spatial features of the object in each frame of image and the temporal features of the object between multiple frames of image can be fused into the spatiotemporal representation corresponding to the image group, so that the target position features have both the spatial features and the temporal features of the object, thereby improving the accuracy of motion trajectory generation.

[0135] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0136] Based on the same inventive concept, the present application also provides a motion trajectory generation device for implementing the aforementioned motion trajectory generation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more of the following embodiments of the motion trajectory generation device can be found in the above-mentioned limitations of the motion trajectory generation method and will not be repeated here.

[0137] In one embodiment, Figure 6 As shown, a motion trajectory generation device 600 is provided, comprising: a spatial representation generation module 602, a spatiotemporal representation generation module 604 and a motion trajectory generation module 606, wherein:

[0138] The spatial representation generation module 602 is used to obtain multiple frames of images and generate spatial position representations of multiple objects in each frame of image. The nodes of the spatial position representation are used to represent the initial position features of the objects in the image, and the spatial adjacency matrix corresponding to the spatial position representation is used to represent the spatial position relationship between objects.

[0139] The spatiotemporal representation generation module 604 is configured to treat each frame image and its corresponding adjacent images as an image group, fuse multiple spatial position representations corresponding to each image group, and generate a spatiotemporal representation of each image group.

[0140] The motion trajectory generation module 606 is used to perform graph convolution processing on the spatiotemporal representation of each image group to generate target position features of multiple objects; and generate the motion trajectory of each object in the future time based on the target position features.

[0141] In one embodiment, the spatiotemporal representation generation module 604 includes: a spatial adjacency matrix fusion unit, used to fuse multiple spatial adjacency matrices corresponding to each image group to obtain a fused spatial adjacency matrix; a spatiotemporal adjacency matrix generation unit, used to process the fused spatial adjacency matrix according to the number of images in the image group to obtain a spatiotemporal adjacency matrix corresponding to each image group, and the spatiotemporal adjacency matrix is ​​used to represent the spatiotemporal position relationship between nodes on different frame images in the image group; a spatiotemporal representation generation unit, used to generate a spatiotemporal representation corresponding to each image group based on the nodes contained in each image group and the spatiotemporal adjacency matrix corresponding to each image group.

[0142] In one embodiment, the spatial adjacency matrix fusion unit is further used to: determine the first adjacent image acquired before the acquisition moment of each frame image, and the second adjacent image acquired after the acquisition moment of each frame image, and take each frame image, the first adjacent image and the second adjacent image corresponding to each frame image as a group of image groups, wherein the number of frames of the first adjacent image and the second adjacent image are the same.

[0143] In one embodiment, the motion trajectory generation module 606 includes: a distribution feature generation unit, which is used to perform time convolution processing on the target position feature to obtain the position distribution feature of the object in the future time; and a motion trajectory generation unit, which is used to sample the position distribution feature to generate the motion trajectory of the object in the future time.

[0144] In one embodiment, the spatial representation generation module 602 includes: an initial position feature generation unit, which is used to determine the displacement information generated by each object between each frame image and the previous frame image of each frame based on the position information of each object in multiple frames of images, and use the displacement information as the initial position feature of the object in each frame image; a spatial adjacency matrix generation unit, which is used to determine the distance between two objects in each frame image based on the position information of multiple objects in each frame image, and generate a spatial adjacency matrix corresponding to each frame image based on the distance; and a spatial position representation generation unit, which is used to generate a spatial position representation of multiple objects in each frame image based on the initial position feature and the spatial adjacency matrix.

[0145] In one embodiment, the spatial representation generation module 602 further includes: a position information deletion unit for comparing multiple objects included in each two adjacent frames of images, and deleting the position information of the first object when there is a first object that does not exist in both adjacent frames of images.

[0146] Each module in the motion trajectory generation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0147] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store weight data of a graph convolutional network and weight data of a time convolutional network. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a motion trajectory generation method is implemented.

[0148] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0149] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0150] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0151] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0153] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0154] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A motion trajectory generation method, characterized in that: The method comprises: Acquire multiple frames of images and generate spatial position representations of multiple objects in each frame of the image, wherein nodes of the spatial position representations are used to represent initial position features of the objects in the image, and a spatial adjacency matrix corresponding to the spatial position representations is used to represent spatial position relationships between the objects; taking each frame of the image and the adjacent images corresponding to each frame of the image as an image group, fusing the multiple spatial position representations corresponding to each image group, and generating a spatiotemporal representation of each image group; performing graph convolution processing on the spatiotemporal representation of each of the image groups to generate target position features of a plurality of the objects; generating a motion trajectory of each of the objects in the future according to the target position features; The acquiring of multiple frames of images and generating spatial position representations of multiple objects in each frame of the image includes: Performing target detection on each frame of the image to determine the position information of the object in each frame of the image; Performing computational processing on the position information of the object between two adjacent frames of images to determine the initial position features of the object and obtain a node corresponding to the object in each frame of the image; Performing computational processing on initial position features of each pair of objects in the image of each frame to obtain spatial positional relationships between the two objects, using the spatial positional relationships between the two objects as edges between two nodes, and generating a spatial adjacency matrix corresponding to the image; Generate a spatial position representation corresponding to each frame of the image according to a plurality of nodes corresponding to each frame of the image and a spatial adjacency matrix; The fusing of the plurality of spatial position representations corresponding to each of the image groups to generate a spatiotemporal representation of each of the image groups includes: fusing the plurality of spatial adjacency matrices corresponding to each of the image groups to obtain a fused spatial adjacency matrix, including: fusing the spatial position representation corresponding to each frame image in the image group, and obtaining relative positional relationships between nodes corresponding to different frame images in the fused spatial position representation; Processing the fused spatial adjacency matrix according to the number of images in the image group to obtain a spatiotemporal adjacency matrix corresponding to each image group, wherein the spatiotemporal adjacency matrix is ​​used to represent the spatiotemporal positional relationship between nodes on different frame images in the image group; Based on the nodes contained in each of the image groups and the spatiotemporal adjacency matrix corresponding to each of the image groups, the spatiotemporal representation corresponding to each of the image groups is generated, including: based on the relative positional relationship between any two nodes, a spatiotemporal representation corresponding to each of the image groups is generated, the spatiotemporal representation being used to characterize the temporal characteristics of the object between each frame of the image and the adjacent images corresponding to each frame of the image.

2. The method according to claim 1, characterized in that The step of taking each frame of the image and the adjacent images corresponding to each frame of the image as an image group includes: Determine a first adjacent image captured before the capture moment of each frame of the image, and a second adjacent image captured after the capture moment of each frame of the image, and take each frame of the image, the first adjacent image corresponding to each frame of the image, and the second adjacent image as a group of the image group, wherein the first adjacent image and the second adjacent image have the same number of frames.

3. The method according to claim 1, characterized in that Generating a motion trajectory of each object in the future time according to the target position feature includes: Performing time convolution processing on the target position features to obtain position distribution features of the object in the future; The position distribution features are sampled to generate the motion trajectory of the object in the future.

4. The method according to claim 1, wherein Generating spatial position representations of multiple objects in each frame of the image includes: Determining, based on the position information of each object in the multiple frames of the image, displacement information of each object between each frame of the image and the previous frame of the image, and using the displacement information as an initial position feature of the object in each frame of the image; Determining distances between two of the objects in each frame of the image based on position information of the objects in each frame of the image, and generating a spatial adjacency matrix corresponding to each frame of the image based on the distances; The spatial position representation of the plurality of objects in each frame of the image is generated according to the initial position features and the spatial adjacency matrix.

5. The method according to claim 4, characterized in that The method further comprises: The multiple objects respectively included in each two adjacent frames of images are compared, and when there is a first object that does not exist in both adjacent frames of images, the position information of the first object is deleted.

6. A motion trajectory generating device, characterized in that: The device comprises: a spatial representation generation module, configured to acquire multiple frames of images and generate spatial position representations of multiple objects in each frame of the image, wherein the nodes of the spatial position representations are used to represent the initial position features of the objects in the image, and the spatial adjacency matrix corresponding to the spatial position representations is used to represent the spatial position relationships between the objects; a spatiotemporal representation generation module, configured to treat each frame of the image and its corresponding adjacent image as an image group, fuse the multiple spatial position representations corresponding to each image group, and generate a spatiotemporal representation of each image group; a motion trajectory generation module, configured to perform graph convolution processing on the spatiotemporal representation of each of the image groups to generate target position features of a plurality of the objects, and generate a motion trajectory of each of the objects in the future based on the target position features; The spatial representation generation module is further configured to perform target detection on each frame of the image to determine the position information of the object in each frame of the image; perform computational processing on the position information of the object between two adjacent frames of the image to determine the initial position features of the object and obtain nodes corresponding to the object in each frame of the image; perform computational processing on the initial position features of each pair of objects in each frame of the image to obtain the spatial position relationship between the two pairs of objects, use the spatial position relationship between the two pairs of objects as the edge between the two pairs of nodes, and generate a spatial adjacency matrix corresponding to the image; and generate a spatial position representation corresponding to each frame of the image based on the multiple nodes corresponding to each frame of the image and the spatial adjacency matrix. The spatiotemporal representation generation module includes: a spatial adjacency matrix fusion unit, configured to fuse the plurality of spatial adjacency matrices corresponding to each of the image groups to obtain a fused spatial adjacency matrix, comprising: fusing the spatial position representation corresponding to each frame image in the image group to obtain the relative positional relationship between any two nodes corresponding to different frame images in the fused spatial position representation; a spatiotemporal adjacency matrix generating unit, configured to process the fused spatial adjacency matrix according to the number of images in the image group to obtain a spatiotemporal adjacency matrix corresponding to each image group, wherein the spatiotemporal adjacency matrix is ​​used to represent the spatiotemporal positional relationship between nodes on different frame images in the image group; A spatiotemporal representation generation unit is configured to generate the spatiotemporal representation corresponding to each image group based on the nodes contained in each image group and the spatiotemporal adjacency matrix corresponding to each image group, including: generating a spatiotemporal representation corresponding to each image group based on the relative positional relationship between any two nodes, wherein the spatiotemporal representation is used to characterize the temporal characteristics of the object between each frame of the image and the adjacent images corresponding to each frame of the image.

7. The device according to claim 6, characterized in that The spatial adjacency matrix fusion unit is further used to: determine the first adjacent image acquired before the acquisition moment of each frame of the image, and the second adjacent image acquired after the acquisition moment of each frame of the image, and take each frame of the image, the first adjacent image corresponding to each frame of the image, and the second adjacent image as a group of the image group, wherein the number of frames of the first adjacent image and the second adjacent image are the same.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images