A method and device for predicting the motion state of a target object

Abstract

This specification discloses a method and device for predicting the motion state of a target object. When predicting the motion state of each target object in the surrounding environment, each target object is distinguished and the interaction between each target object is determined. The determined interaction between each target object includes: the spatial interaction characteristics between each target object in the surrounding environment and the temporal interaction characteristics of the spatial interaction characteristics of each target object changing over time. Finally, the method in this specification predicts the motion state of each target object based on the determined interaction between each target object, so that the prediction result is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a method and device for predicting the motion state of a target object. Background Art

[0002] At present, vehicles with assisted driving functions and unmanned vehicles have become one of the main directions of development of transportation technology.

[0003] When determining the conditions of their surrounding environment, vehicles with assisted driving functions and unmanned vehicles need to predict the future motion states of various objects in the surrounding environment to reflect possible changes in the surrounding environment conditions in the future.

[0004] In the prior art, if there are multiple target objects in the surrounding environment, it is difficult to determine the historical motion state corresponding to a certain target object in each historical data in the collected historical data, and it is difficult to know the change of the motion state of the target object over time. In addition, the target objects in the surrounding environment will also affect each other during the movement process. The interaction between the target objects will affect the prediction result of the future motion state of the target object. The mutual influence between the target objects is the interaction between the target objects. The prediction of the motion state of the target object in the prior art cannot determine the interaction between the target objects, and it is even more impossible to reflect the interaction between the target objects in the prediction result when predicting the motion state of the target object. As a result, the future motion state of the target object determined on this basis is not accurate enough.

[0005] It can be seen that the accuracy of predicting the motion state of each target object in the surrounding environment by using the method in the prior art is low. Summary of the invention

[0006] The embodiments of this specification provide a method and device for predicting the motion state of a target object for order reassignment, so as to partially solve the above-mentioned problems existing in the prior art.

[0007] The embodiments of this specification adopt the following technical solutions:

[0008] The method for predicting the motion state of a target provided in this specification includes:

[0009] Get the motion state of each target object at the current moment;

[0010] For each target object, according to the motion state corresponding to the target object, determine the motion state characteristics corresponding to the target object at the current moment;

[0011] According to the motion state characteristics corresponding to each target object at the current moment, a spatial interaction characteristic for characterizing the interaction status of each target object in space at the current moment is determined;

[0012] Determine, according to the spatial interaction features at the current moment and the predetermined spatial interaction features of each target object at a historical moment, a temporal interaction feature for characterizing how the spatial interaction features of each target object change over time;

[0013] For each target object, the future motion state of the target object is predicted according to the motion state characteristics corresponding to the target object and the time interaction characteristics.

[0014] Optionally, the motion state corresponding to the target object is: at least one of a speed direction, a velocity, a position and a drivable route corresponding to the target object.

[0015] Optionally, the motion state corresponding to each target object at the current moment is obtained, specifically including:

[0016] Collect the actual image at the current moment;

[0017] According to the image of each target object contained in the actual image, the motion state corresponding to each target object is determined.

[0018] Optionally, determining the motion state feature corresponding to the target object at the current moment according to the motion state corresponding to the target object specifically includes:

[0019] For each motion state corresponding to the target object at the current moment, determining a virtual image used to represent the motion state;

[0020] A motion state feature corresponding to the motion state is determined according to the virtual image used to represent the motion state.

[0021] Optionally, determining a motion state feature corresponding to the motion state according to a virtual image used to represent the motion state specifically includes:

[0022] A virtual image used to represent the motion state is input into a pre-trained first convolutional neural network model to obtain a feature vector corresponding to the virtual image as a motion state feature corresponding to the motion state.

[0023] Optionally, according to the motion state feature corresponding to each target at the current moment, determining the spatial interaction feature used to characterize the interaction status of each target at the current moment in space specifically includes:

[0024] For each motion state, the motion state features corresponding to the motion state of each target object at the current moment are pooled to obtain a pooling vector corresponding to the motion state;

[0025] According to the pooling vectors corresponding to each motion state, a spatial interaction feature for characterizing the interaction status of each target object in space at the current moment is determined.

[0026] Optionally, according to the spatial interaction feature at the current moment and the predetermined spatial interaction features of each target object at a historical moment, determining a temporal interaction feature for characterizing how the spatial interaction features of each target object change over time specifically includes:

[0027] The spatial interaction features at the current moment are input into a pre-trained LSTM model to determine a feature vector corresponding to the spatial interaction features at the current moment as a temporal interaction feature for characterizing how the spatial interaction features of each target object change over time.

[0028] Optionally, for each target object, predicting the future motion state of the target object according to the motion state feature corresponding to the target object and the time interaction feature specifically includes:

[0029] For each target object, a feature matrix corresponding to the target object is obtained according to the motion state feature corresponding to the target object and the time interaction feature;

[0030] The feature matrix is ​​input into a pre-trained second convolutional neural network model to obtain an image for characterizing the future motion state of the target object.

[0031] The device for predicting the motion state of a target provided in this specification includes:

[0032] An acquisition module is used to obtain the motion state corresponding to each target object at the current moment;

[0033] A motion state feature determination module is used to determine, for each target object, the motion state feature corresponding to the target object at the current moment according to the motion state corresponding to the target object;

[0034] The spatial interaction feature determination module at the current moment is used to determine the spatial interaction feature used to characterize the interaction status of each target object in space at the current moment according to the motion state feature corresponding to each target object at the current moment;

[0035] A temporal interaction feature determination module, used to determine a temporal interaction feature for characterizing how the spatial interaction feature of each target object changes over time based on the spatial interaction feature at the current moment and the predetermined spatial interaction features of each target object at a historical moment;

[0036] The prediction module is used to predict the future motion state of each target object according to the motion state characteristics corresponding to the target object and the time interaction characteristics.

[0037] The computer-readable storage medium provided in this specification stores a computer program, and when the computer program is executed by a processor, the method for predicting the motion state of a target object is implemented.

[0038] The electronic device provided in this specification includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for predicting the motion state of a target object is implemented.

[0039] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0040] The method and device for predicting the motion state of a target object of the present specification are used to distinguish each target object in the surrounding environment when predicting the motion state of each target object, and determine the interaction between each target object. The determined interaction between each target object includes: the spatial interaction characteristics between each target object in the surrounding environment and the temporal interaction characteristics of the spatial interaction characteristics of each target object changing over time. Finally, the method in the present specification predicts the motion state of each target object based on the determined interaction between each target object, so that the prediction result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0042] Figure 1 The process of predicting the motion state of the target object provided in this manual;

[0043] Figure 2 An exemplary network architecture diagram provided for the embodiments of this specification;

[0044] Figure 3 A schematic diagram of the process of predicting the motion state of a target object provided in this manual;

[0045] Figure 4 The process of obtaining the motion state of the target object at the current moment provided in this manual;

[0046] Figure 5 The future motion state of the target object α predicted by the process of predicting the motion state of the target object provided in this specification;

[0047] Figure 6 The future comprehensive position image of the target object determined by the process of predicting the motion state of the target object provided in this specification;

[0048] Figure 7 A schematic diagram of the structure of a device for predicting the motion state of a target object provided in an embodiment of this specification;

[0049] Figure 8The embodiments of this specification provide corresponding to Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0050] In the prior art, when predicting the future motion state of a target object, firstly, the surrounding environment is imaged in chronological order to obtain a number of actual images, and then, for each actual image obtained by the acquisition, each target object in the actual image is determined.

[0051] Since it is difficult in the prior art to set a distinctive mark for each target object in an image, and it is also difficult to make the distinctive mark accompany the corresponding target object throughout the image recognition process for all actual images and the prediction process for the motion state of the target object, therefore, when the prior art processes each actual image, the various targets in the actual image cannot be distinguished by other means except for their different positions.

[0052] In actual application scenarios, most of the targets in the surrounding environment are in motion, so the position of the target in the actual image at the current moment has changed compared to the position of the target in the actual image at the previous moment. However, since it is impossible to set an identifier for the target in the prior art, it is impossible to know exactly which target in the actual image at the previous moment a target in the actual image at the current moment corresponds to, resulting in the inability to determine the correlation between the positions of the same target in each actual image corresponding to different moments, and thus it is difficult to know the change of the motion state of the target over time.

[0053] Furthermore, for a certain target object, if the target object cannot be identified from each actual image, it is also impossible to determine the interaction between each target object, and there is no basis for reflecting the interaction between each target object in the result of the motion state prediction for the target object, resulting in the method of predicting the motion state of the target object in the prior art being unable to predict the motion state corresponding to each target object in the surrounding environment separately. In addition, the results obtained by the existing prediction method are of low accuracy.

[0054] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0055] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0056] Figure 1 The process of predicting the motion state of a target object provided in the embodiments of this specification may specifically include the following steps:

[0057] S100: Obtain the motion state corresponding to each target object at the current moment.

[0058] Specifically, the actual image of the environment in which the current vehicle is located can be collected by a camera set on the current vehicle, and the image content of the actual image can include the target object. There can be multiple cameras, and the multiple cameras collect images of the environment in different directions. Then, the method in this specification can predict the target objects around the current vehicle based on the actual images collected.

[0059] The target object may be any object in the environment where the current vehicle is located that may affect the driving of the current vehicle. For example, other vehicles in the environment. Specifically, the objects in the environment can be identified in the actual image through existing image recognition technology.

[0060] In this description, the current vehicle may be a vehicle with an assisted driving function or an unmanned vehicle. The operation of obtaining the motion state corresponding to each target object at the current moment may be implemented with the aid of devices such as mobile phones, tablet computers, personal computers (PCs), drones, etc.

[0061] It can be seen that, through the method in this specification, when determining the motion state of each target object, the motion state of each target object is determined separately, thereby avoiding the problem of being unable to distinguish between the targets in subsequent steps.

[0062] Specifically, the motion state corresponding to the target object in this specification may be: one or more of the environment map, position, speed, motion trajectory and drivable route corresponding to the target object. In this specification, the motion state of the target object may be represented by numerical values ​​or by images. The following is an example of representing the motion state of the target object by images.

[0063] S102: For each target object, determine the motion state feature corresponding to the target object at the current moment according to the motion state corresponding to the target object.

[0064] In this specification, for each target object in the actual image, the motion state corresponding to the target object at the current moment is extracted to determine the motion state characteristics corresponding to the motion state. For example, when the motion state corresponding to the target object is represented in the form of an image, the image representing the motion state of the target object can be feature extracted by a two-dimensional convolution method to determine the motion state characteristics of the motion state corresponding to the target object. Specifically, the process of determining the motion state characteristics corresponding to the target object at the current moment can be as follows: Figure 3 CNN-1 shown in .

[0065] When the target object corresponds to two or more motion states, feature extraction may be performed separately for each motion state corresponding to the target object. Figure 3 The position features FP are determined for the position and velocity of the target object α, respectively. α and speed characteristic FV α .

[0066] S104: Determine, according to the motion state feature corresponding to each target object at the current moment, a spatial interaction feature for characterizing the interaction status of each target object in space at the current moment.

[0067] In this specification, the spatial interaction feature used to characterize the current moment of each target object can be determined based on at least one of the position interaction feature, speed interaction feature, and motion direction interaction feature of each target object. Among them, the position interaction feature is used to characterize the position interaction status between each target object. The position interaction feature can be determined based on the position feature of each target object, for example Figure 3 The position interaction feature X is obtained by Pooling as shown p ; The speed interaction feature is used to characterize the speed interaction status between each target object. The speed interaction feature can be determined based on the speed feature of each target object, for example Figure 3 The position interaction feature V is obtained by Pooling as shown p The motion direction interaction feature is used to characterize the motion trajectory interaction status between the targets, and the motion direction interaction feature can be determined based on the motion trajectory features of each target.

[0068] For example, the process of determining the spatial interaction characteristics at the current moment can be as follows Figure 3 Concat as shown. Several of the determined position interaction features, speed interaction features, and motion trajectory interaction features are concatenated to obtain the spatial interaction features at the current moment.

[0069] In an optional embodiment of the present description, the spatial interaction feature may be a vector, and the spatial interaction feature at the current moment obtained by concatenating the interaction features may also be a vector.

[0070] S106: Determine, based on the spatial interaction features at the current moment and the predetermined spatial interaction features of each target object at a historical moment, a temporal interaction feature for characterizing how the spatial interaction features of each target object change over time.

[0071] In this specification, the spatial interaction characteristics of each target object at a historical moment are used as one of the bases for determining the temporal interaction characteristics, so that the spatial interaction status of each target object at a historical moment can be reflected in the temporal interaction characteristics, so that the historical interaction status of each target object can be used as the basis for predicting the future motion state of the target object, thereby improving the accuracy of the prediction. The process of determining the temporal interaction characteristics can be as follows: Figure 3 The LSTM (Long Short Term Memory) shown.

[0072] S108: For each target object, predict the future motion state of the target object according to the motion state feature corresponding to the target object and the time interaction feature.

[0073] In this specification, when predicting the future motion state of the target object, based on the determined time interaction characteristics corresponding to each target object and the motion state characteristics of the target object, the spatial interaction status between the target objects at the current moment and the change of the interaction status over time can be comprehensively considered, thereby improving the accuracy of the prediction of the future motion state of the target object.

[0074] Furthermore, predicting the future motion state of each target object separately can improve the prediction effect of possible future changes in the surrounding environment. Figure 3 CNN-2 shown.

[0075] The method for sending data described in this specification is described in detail below.

[0076] like Figure 2 As shown, in this specification, the network architecture includes a collection device and a processing device, and the collection device is communicatively connected with the processing device. Among them, the collection device can be a camera for collecting actual images. The processing device is used to predict the motion state of the target object based on the actual image collected by the collection device. In an optional embodiment of this specification, the collection device and the processing device can both be set on the current vehicle.

[0077] In this specification, obtaining the motion state corresponding to each target object at the current moment may include: capturing images of the surrounding environment to obtain actual images. Then, according to the actual images captured, corresponding virtual images are obtained for each motion state corresponding to each target object.

[0078] Specifically, for the actual image captured, one target object in the actual image may be retained, while other targets in the actual image may be removed to obtain image content corresponding to the actual image that only includes the target object. Thereafter, virtual images corresponding to each motion state of the target object may be determined separately based on the image content of the actual image that only includes the target object.

[0079] For example, Figure 3 As shown, for the Input actual image, the image contents of the target object α and the target object β in the actual image corresponding to the actual image are extracted respectively by Extract. Thereafter, for the target object α and the target object β, virtual images for representing the motion state of the target object are determined according to their respective corresponding image contents.

[0080] The process of determining each virtual image for representing the motion state of the target object α according to the image content corresponding to the target object α in the actual image can be as follows: Figure 4 shown.

[0081] In this specification, the determined virtual image can represent at least one of the corresponding motion states of the target object, and each determined virtual image can represent at least one motion state of a target object. That is, an actual image can correspond to multiple targets, while a virtual image can only correspond to one target object. Optionally, a virtual image only corresponds to one motion state of a target object.

[0082] When determining the motion state corresponding to each target object, a coordinate system can be established for the environment in which each target object is located, and each virtual image corresponding to each target object can be unified in the same coordinate system. The coordinate system can be determined by the environment map in which each target object is located, and the origin of the coordinate system can be the current vehicle position.

[0083] The environment map may include preset facilities (e.g., lane lines, etc.) that regulate the drivable routes of each vehicle, such as lane structures within a preset range; the environment map may also include landmarks in the environment that may affect the current vehicle (e.g., walls, temporary roadblocks, etc.). The drivable route may be a route that the target object can travel on in the environment map. The preset range may be determined based on the shooting range of the current vehicle's camera.

[0084] Specifically, Figure 3 , Figure 4As shown, according to the environment where each target object is located, a coordinate system is established, and a virtual environment map image E and a position image P (for example, the position image corresponding to the target object α is P) can be determined according to the environment map, position, speed, motion trajectory and drivable route corresponding to the target object. α ), velocity image V (for example, the corresponding position image of the target object α is V α )、Motion trajectory image( Figure 3 , Figure 4 (not shown) and at least one of the drivable route images R.

[0085] At least one of the virtual images may be an image drawn according to the probability of the target object appearing at each coordinate point in the image. Some virtual images are described in detail below:

[0086] (1) Environment map image E

[0087] The environment map image E may be determined according to the environment map topology structure.

[0088] Specifically, the environment map image E can be obtained by the following steps: according to the actual image acquired, the image content belonging to the environment map in the actual image is determined, and the image content other than the image content belonging to the environment map in the actual image is removed to obtain a processed image. The image content belonging to the environment map in the actual image may include: preset facilities such as lane structures within a preset range that regulate the drivable routes of each vehicle, and / or temporary markers temporarily set on the lane structure that may affect the current vehicle.

[0089] Then, the image content belonging to the environment map in the processed image is rendered to determine the environment map image E. Specifically, the brightness and color of the pixels corresponding to each coordinate point in the processed image may be adjusted to identify the image content belonging to the environment map in the processed image.

[0090] For example, a plurality of white pixels with high brightness may be used to represent lane lines in the processed image, and a plurality of yellow pixels with uniform brightness may be used to represent the dividing line of the two-way lane. The dividing line of the two-way lane represented by the yellow pixel may not be used as a calculation basis in the subsequent calculation process because it is not displayed in grayscale. However, the dividing line of the two-way lane represented by the yellow pixel may be reflected in the prediction result of the motion state of the target object for the convenience of user observation.

[0091] In an optional embodiment of the present specification, a coordinate system may be established for the environment in which each target object is located, so as to unify each target object in the same coordinate system, and each target object may correspond to the same environment map image E. For example, Figure 3As shown, the environment map can be directly obtained from the actual image (corresponding to Input) collected; the environment map can also be obtained from the image content of the target object α and the target object β corresponding to the actual image (corresponding to Extract) (not shown in the figure); the environment map can also be obtained according to the collection position corresponding to the actual image (for example, Figure 2 The server determines the environment map image E corresponding to the collection location. The server can be Figure 2 The collection device and / or processing device shown in the figure is communicated with the server.

[0092] (2) Drivable route image R

[0093] The drivable route image R of the target object can be determined according to the determined environment map image and preset traffic rules.

[0094] Specifically, the drivable route image R can be obtained by the following steps: according to the determined environment map image, determine each road in the environment map image that the target object can pass through. According to the preset traffic rules, determine the feasibility of the target object passing through each road. According to the environment map image and the feasibility corresponding to each road in the environment map image, determine the drivable route image R.

[0095] The preset traffic rules may include traffic lights, speed bumps, etc. in the environment map image. For example, if the traffic light at an intersection in the environment map image is red at the moment, it is less feasible for the target object to pass through the corresponding coordinate point at the intersection; accordingly, the grayscale value of the pixel at this position in the drivable route image is also smaller.

[0096] In this specification, the drivable route image R can be used to represent the feasibility of the target object passing through each coordinate point in the environment map image E. For example, the feasibility of the target object passing through the coordinate point in the environment map image E can be represented by the grayscale of the pixel corresponding to any coordinate point in the drivable route image R. Figure 4 In the drivable route image R shown, the larger the grayscale value of a pixel, the greater the feasibility of the target object passing through the coordinate point corresponding to the pixel in the environmental map image; if a pixel is black, it indicates that the target object cannot pass through the coordinate point corresponding to the pixel (for example, the coordinate point corresponding to the black pixel corresponds to a wall in the environmental map image, and the target object cannot pass through the coordinate point corresponding to the pixel).

[0097] Optionally, a coordinate system may be established for the environment in which each target object is located, so as to unify each target object in the same coordinate system, and each target object may correspond to the same drivable route image.

[0098] (3) Position image P

[0099] The position image P corresponding to the target object can be determined according to the position of the target object in the actual image and the established coordinate system.

[0100] Specifically, the position image P can be obtained through the following steps: based on the actual image acquired, for each target object, remove other image content in the actual image except the image content of the target object (for example, the other image content may be: the image content corresponding to other target objects in the actual image and the image content corresponding to the environment map in the actual image) to determine the image content in the actual image corresponding to the target object, so that the image content corresponding to the target object only contains the target object; based on the image content in the actual image corresponding to the target object, determine the position image P corresponding to the target object.

[0101] Since the existing image acquisition technology and image processing algorithm cannot completely avoid the influence of noise, the accuracy of the determined position of the target object is low. In addition, most of the targets in the surrounding environment are moving targets, so it is difficult to accurately reflect the position of the target object through only one pixel in the image. In this specification, the position image P is used to characterize the probability of the coordinate point corresponding to each pixel of the target object in the position image P. Specifically, the position image P of the target object can be obtained according to Gaussian function fitting, so that in the position image P, the grayscale value of the pixel corresponding to the coordinate point with a higher probability of the target object appearing is higher; the grayscale value of the pixel corresponding to the coordinate point with a lower probability of the target object appearing is lower, so that the position of the target object can be expressed as a light spot with inconsistent grayscale in the position image P of the target object.

[0102] In an optional embodiment of the present invention, the position image P can be calculated by the following formula:

[0103]

[0104] Where:

[0105] (i, j) is a coordinate point in the position image, which may correspond to a pixel;

[0106] P(i,j) is the probability of the target object appearing at the coordinate point (i,j);

[0107] c1 is the coefficient related to the position of the target object;

[0108] α is the horizontal coordinate of the actual position of the target;

[0109] β is the ordinate of the actual position of the target;

[0110] is a matrix used to characterize the error. The matrix may be a 2×2 matrix used to characterize the error caused by the camera position.

[0111] For a target object in the actual image, the position image of the target object can be obtained by calculating each coordinate point in the coordinate system according to the image content corresponding to the target object in the actual image through formula (1). For example, the position image corresponding to the target object α determined by the method in this specification is as follows: Figure 4 P α shown.

[0112] (4) Velocity image V

[0113] The velocity image V corresponding to the target object can be determined according to the position of the target object in the actual image, the established coordinate system, and the speed and velocity direction of the target object obtained by measurement.

[0114] Specifically, the speed image V can be obtained through the following steps: based on the actual image acquired, for each target object, remove the image content other than the image content of the target object in the actual image to determine the image content corresponding to the target object in the actual image, so that the image content corresponding to the target object only contains the target object; based on the image content corresponding to the target object in the actual image, determine the position image P corresponding to the target object; based on the position image P corresponding to the target object and the speed and speed direction of the target object obtained by measurement, determine the speed image V corresponding to the target object.

[0115] Existing methods for characterizing speed are mostly in the form of textual representations through the combination of numerical values ​​and speed units. In this specification, the speed direction, rate, rate distribution of the target object within a preset range around its position, and the error generated during speed measurement are represented by an image, so that the representation of the target object's speed is more comprehensive. In addition, by representing the speed and other factors that affect the speed in the form of an image, it is possible to unify the representation methods of speed and other types of motion states (for example, the position corresponding to the aforementioned target object is represented by a position image), thereby reducing the complexity of data processing of the motion state.

[0116] In an optional embodiment of the present invention, the velocity image can be obtained by calculating the following formula:

[0117]

[0118] Σ2=ZΣ0Z T

[0119]

[0120] Where:

[0121] (i, j) is a coordinate point in the velocity image, which may correspond to a pixel;

[0122] V(i,j) is the coordinate of the target object after adjustment based on the speed and direction of the target object.

[0123] The probability of occurrence of (i,j);

[0124] c2 is the coefficient related to the speed of the target;

[0125] α is the horizontal coordinate corresponding to the coordinate point of the maximum velocity value of the target object in the coordinate system;

[0126] β is the ordinate corresponding to the coordinate point of the maximum velocity value of the target object in the coordinate system;

[0127] v is the velocity of the target object obtained by measurement;

[0128] v max is a constant, which may be the maximum speed limit of the lane where the target object is located;

[0129] ∑0 is a matrix used to characterize the error. The matrix can be a 2×2 matrix, which is used to characterize the error caused by speed measurement;

[0130] θ is the speed direction of the target object, which can be obtained through measurement.

[0131] For a target object in the actual image, the velocity image of the target object can be obtained by calculating each coordinate point in the coordinate system according to the image content corresponding to the target object in the actual image through formula (2). For example, the velocity image corresponding to the target object α determined by the method in this specification is as follows: Figure 4 V α shown.

[0132] It can be seen that Σ2 in formula (2) can stretch the graph composed of pixels with higher grayscale values ​​in the velocity image of a target object in one direction, so that the graph is approximately elliptical. The probability of the target object appearing in the stretching direction (the long axis direction of the ellipse) is relatively high, and this direction is the velocity direction of the target object.

[0133] In formula (2) The speed-related information expressed by the speed images corresponding to each target object can be normalized, and the speed limit of the lane where the target object is located can be used as a basis for determining the speed image, so that the speed limit of the lane can be reflected in the speed image.

[0134] (5) Motion trajectory image ( Figure 4 (not shown)

[0135] The motion trajectory image corresponding to the target object can be determined according to the position of the target object in each actual image collected within a preset time period and the established coordinate system.

[0136] Specifically, the motion trajectory image can be obtained through the following steps: based on the actual image obtained by acquisition, for each target object, other image content except the image content of the target object in the actual image is removed to determine the image content corresponding to the target object in the actual image, so that the image content corresponding to the target object only contains the target object; based on the image content corresponding to the target object in the actual image, a position image P corresponding to the target object is determined; based on each position image of the target object within a preset time period, and the length of time from the acquisition time of each position image within the time period to the current moment, the motion trajectory image corresponding to the target object is determined.

[0137] The brightness of each pixel at each coordinate point of the motion trajectory image is used to represent the change in the probability of the target object appearing at the coordinate point over time within a preset time period.

[0138] In an optional embodiment of the present invention, the motion trajectory image can be obtained by calculating the following formula:

[0139]

[0140] Where:

[0141] (i, j) is a coordinate point in the velocity image, which may correspond to a pixel;

[0142] T raj (i, j) is the probability of the target object appearing at the coordinate point (i, j) over time within the preset time period;

[0143] c3 is a coefficient related to the preset time period;

[0144] n is the time from any moment in the preset time period to the current moment, and n is an integer greater than or equal to 0;

[0145] t n is the moment in the preset time period corresponding to a time length n from the current moment;

[0146] γ is the preset parameter;

[0147] t n The probability of the target object appearing at the coordinate point (i, j) in the coordinate system at the moment is calculated by formula (1).

[0148] For a target object in the actual image, each coordinate point in the coordinate system can be calculated by formula (3) to obtain the motion trajectory image corresponding to the target object. raj The larger the value of (i, j), the greater the brightness of the coordinate point, indicating that within the preset time period, the probability of the target object appearing at the coordinate point is less affected by time.

[0149] When obtaining the motion trajectory image, the formula (3) can be used to calculate each point in the coordinate system, so as to distinguish the influence of each moment in the preset time period on the motion trajectory image according to the time length from any moment in the preset time period to the current moment.

[0150] For example, for the coordinate point (i k ,j k ), the preset time period is within 3 seconds from the current time, and the step length is 1 second, then n∈{0, 1, 2, 3}; among them, when n=0, the corresponding time t0 is the current time. The terms in the formula It can make n = 0 The calculation result T raj (i k ,j k ) is greater than when n = 3 Contribution to the calculation result. That is, within the preset time period, the influence of the moment closer to the current moment on the motion trajectory image will be greater than the influence of the moment farther from the current moment on the motion trajectory image, and then the change of the probability of the target object appearing at the coordinate point over time is reflected in the obtained motion trajectory image.

[0151] Through the above method steps, for each target object, each motion state corresponding to the target object can be represented separately in the form of virtual images. Optionally, each virtual image corresponding to the same moment has the same coordinate system.

[0152] In the method of this description, for each motion state corresponding to the target object at the current moment, after determining the virtual image used to represent the motion state, the motion state feature corresponding to the motion state can be determined based on the virtual image of the motion state. In an optional embodiment of this description, the motion state feature can be represented by a vector.

[0153] Specifically, for each target object, a virtual image of each motion state corresponding to the target object can be input into a pre-trained first convolutional neural network model to obtain a feature vector corresponding to the virtual image as a motion state feature corresponding to the motion state of the target object.

[0154] For example, Figure 3 The process of predicting the motion state of the target object shown in FIG. 1 can be based on the position image P of the target object α at the current moment. α , get the position feature FP corresponding to the target object α at the current moment α Other types of motion state features can be obtained by the same method, which will not be listed here one by one.

[0155] The first convolutional neural network model for obtaining the corresponding feature vector according to the virtual image (corresponding to Figure 3 CNN-1) in the figure may be a two-dimensional convolutional neural network model. In an optional embodiment of the present invention, the same motion states corresponding to different targets may correspond to the same first convolutional neural network model.

[0156] Then, for one or several motion states, the motion state features of at least part of the target objects corresponding to the motion state at the current moment can be pooled to obtain a pooling vector corresponding to the motion state. It can be seen that each obtained pooling vector corresponds to a motion state; and each pooling vector corresponds to at least one target object.

[0157] For example, Figure 3 As shown, the position feature FP in the motion state feature corresponding to the target α and the target β can be α and FP β Pooling is performed to obtain the position pooling vector X p , for the velocity characteristic FV α and FV β Pooling is performed to obtain the velocity pooling vector X p .

[0158] Specifically, the pooling method may be maximum pooling. For example, the position feature FP corresponding to the target object α α =(α1, α2, α3, α4, α5,), the position feature FP corresponding to the target β β =(β1,β2,β3,β4,β5),FP α and FP β The length of is 5. Among them, α1>β1, α2>β2, α3<β3, α4>β4, α5<β5. Then, for FP α and FP β For each dimension corresponding to α and FP β The maximum value in this dimension is used as the position pooling vector X p The value corresponding to this dimension. Then the position pooling vector X obtained by pooling through maximum pooling p=(α1, α2, β3, α4, β5). In addition, the pooling method is not limited to maximum pooling, and can also be general pooling, overlapping pooling, stochastic pooling, etc.

[0159] Other types of motion state features can be pooled using the same or similar methods, which are not listed here one by one.

[0160] like Figure 3 As shown, in an optional embodiment of the present specification, no pooling process is performed for the environmental features corresponding to the environmental map image, and subsequent steps are performed directly based on the pooling vectors corresponding to the environmental features and other state features.

[0161] According to the pooling vectors corresponding to each motion state, the spatial interaction features used to characterize the spatial interaction status of each target object at the current moment are determined. Specifically, the pooling vectors corresponding to the environmental features and other state features can be spliced ​​in a preset order to obtain the spatial interaction features at the current moment. The process of obtaining the spatial interaction features at the current moment can be as follows: Figure 3 The Concat shown in the figure, the spatial interaction features at the current moment obtained by this process can be expressed as X c .

[0162] The spatial interaction features obtained through the method in this specification are combined with the environmental map, position, speed, motion trajectory and other information corresponding to each target object at the current moment, which can comprehensively characterize the spatial interaction status of each target object at the current moment from multiple angles, so that the prediction of the future motion state of the target object based on this is also more accurate.

[0163] For example, Figure 3 As shown, the spatial interaction feature X at the current moment c , input the pre-trained LSTM model, the LSTM model stores at least some of the spatial interaction features of historical moments, and the LSTM model uses the spatial interaction features of historical moments stored in it and the spatial interaction features of the current moment input X c , determine the spatial interaction feature X at the current moment c The corresponding time interaction feature X t The time interaction feature may be used to characterize how the spatial interaction feature of each target object changes over time. The time interaction feature may be a feature vector that can characterize how the spatial interaction feature of each target object changes over time.

[0164] In an optional embodiment of the present invention, when predicting the future motion state of a target object, for each target object, the motion state feature corresponding to the target object and the time interaction feature can be concatenated to obtain a feature matrix corresponding to the target object. Then, the feature matrix is ​​input into a pre-trained second convolutional neural network model (corresponding to Figure 3 CNN-2 in the output) to obtain an image used to characterize the future motion state of the target object (for example, for target object α, CNN-2 outputs an image L used to characterize the future motion state of target object α α ). Specifically, the second convolutional neural network model may be a two-dimensional convolutional neural network model. Optionally, the image used to characterize the future motion state of the target object is a virtual image generated by predicting the future motion state of the target object.

[0165] The image of the future motion state of the target object obtained according to the feature matrix corresponding to the target object may include at least one of a position prediction map and a speed prediction map. Figure 5 As shown, the position prediction map predicted for the target object α is L α -P, speed prediction map is L α -V.

[0166] Among them, the position prediction map corresponding to the target object can represent the probability of the target object appearing at the coordinate point through the grayscale value of the pixel corresponding to each coordinate point in the position prediction map. Then, in the position prediction map, the grayscale value of the pixel corresponding to the coordinate point with a higher probability of the target object appearing is higher; the grayscale value of the pixel corresponding to the coordinate point with a lower probability of the target object appearing is lower. The predicted position of the target object can be reflected in the position prediction map as a light spot with inconsistent grayscale. The position prediction map can have the same representation method for the position of the target object as the aforementioned position image.

[0167] The speed prediction map can represent the predicted speed and speed direction of the target object by the probability of the target object appearing at each coordinate point in the speed prediction map. The speed prediction map can have the same speed representation method as the aforementioned speed image.

[0168] In an optional embodiment of the present description, for the target object, the position prediction map and the speed prediction map corresponding to the same future moment may have the same coordinate system. Further, for each target object, each future motion state image corresponding to the same future moment may have the same coordinate system.

[0169] In an optional embodiment of the present specification, after predicting the future motion state of each target object, the method further includes: determining the target object future comprehensive motion state image corresponding to each target object based on the target object future motion state image corresponding to each target object. The target object future comprehensive motion state image can be used to characterize: the probability of at least one of the targets appearing at each coordinate point in the image of the future motion state. For each motion state, the target object future comprehensive motion state image corresponding to the motion state can be determined respectively.

[0170] For example, based on a position prediction map of at least part of the target object, a future comprehensive position image of the target object can be determined, which is used to characterize the probability of appearance of each coordinate point of at least part of the target object in the position prediction map; based on a speed prediction map of at least part of the target object, a future comprehensive speed image of the target object can be determined, which is used to characterize the probability of appearance of each coordinate point of at least part of the target object in the speed prediction map.

[0171] Specifically, the future comprehensive position image of the target object can be calculated by the following formula.

[0172]

[0173] Where:

[0174] (i, j) is a coordinate point in the future integrated position image of the target object, which may correspond to a pixel;

[0175] M is the number of targets in the future comprehensive position image of the targets, M and m are both positive integers, m≤M;

[0176] q m (i, j) is the probability of the mth target object appearing at the coordinate point (i, j);

[0177] Q(i,j) is the probability that at least one target object appears at coordinate point (i,j).

[0178] It can be seen that in formula (4), [1-q m (i, j)] represents the probability that the mth target object does not appear at the coordinate point (i, j). For all coordinate points in the future integrated position image of the target object, the probability of each target object not appearing at the coordinate point is determined, and the target image is determined based on the determined superposition probability. The determined target image is reversed to obtain the following: Figure 6As shown, the future comprehensive position image of the target object can characterize the probability of each target object appearing at each coordinate point. The image obtained by this method can reduce the error and improve the accuracy of the obtained image. In the future comprehensive position image of the target object calculated by formula (4), the larger the gray value of a pixel, the greater the probability of at least one target object appearing at the coordinate point corresponding to the pixel; the larger the gray value of a pixel, the smaller the probability of at least one target object appearing at the coordinate point corresponding to the pixel.

[0179] The method in this specification can also determine the future integrated velocity image of the target object through the same process as determining the future integrated position image of the target object.

[0180] The method for predicting the motion state of a target object provided in this specification can be specifically applied to the field of delivery using unmanned vehicles, such as the use of unmanned vehicles for delivery of express delivery, take-out, etc. Specifically, in the above-mentioned scenarios, an autonomous driving fleet consisting of multiple unmanned vehicles can be used for delivery.

[0181] Based on the same idea, the embodiment of this specification also provides a corresponding device for sending data such as Figure 7 shown.

[0182] Figure 7 A schematic diagram of the structure of a first device for sending data provided in an embodiment of this specification includes:

[0183] An acquisition module 700 is used to acquire the motion state corresponding to each target object at the current moment;

[0184] The motion state feature determination module 702 is used to determine the motion state feature corresponding to each target object at the current moment according to the motion state corresponding to the target object;

[0185] The current moment spatial interaction feature determination module 704 is used to determine the spatial interaction feature used to characterize the spatial interaction status of each target object at the current moment according to the motion state feature corresponding to each target object at the current moment;

[0186] The time interaction feature determination module 706 is used to determine the time interaction feature used to characterize the change of the spatial interaction feature of each target object over time based on the spatial interaction feature of the current moment and the spatial interaction feature of each target object at a predetermined historical moment. Specifically, the time interaction feature determination module 706 is used to input the spatial interaction feature of the current moment into a pre-trained LSTM model, and determine the feature vector corresponding to the spatial interaction feature of the current moment as the time interaction feature used to characterize the change of the spatial interaction feature of each target object over time.

[0187] The prediction module 708 is used to predict the future motion state of each target object according to the motion state characteristics corresponding to the target object and the time interaction characteristics.

[0188] Among them, the acquisition module 700, the motion state feature determination module 702, the current moment spatial interaction feature determination module 704, the time interaction feature determination module 706 and the prediction module 708 are electrically connected in sequence, and the prediction module 708 is also electrically connected to the motion state feature determination module 702.

[0189] Optionally, the acquisition module 700 includes an actual image acquisition submodule 700 and a motion state determination submodule 7002 , and the actual image acquisition submodule 700 and the motion state determination submodule 7002 are electrically connected.

[0190] The actual image acquisition submodule 700 is used to acquire the actual image at the current moment.

[0191] The motion state determination submodule 7002 is used to determine the motion state corresponding to each target object according to the image of each target object contained in the actual image.

[0192] Optionally, the motion state determination submodule 7002 includes a virtual image determination unit and a motion state feature determination unit, and the virtual image determination unit and the motion state feature determination unit are electrically connected.

[0193] The virtual image determination unit is used to determine a virtual image used to characterize the motion state for each motion state corresponding to the target object at the current moment. The virtual image determination unit includes an environment map image determination subunit, a position image determination subunit, a speed image determination subunit, a motion trajectory image determination subunit and a drivable route image determination subunit. The environment map image determination subunit is used to determine the environment map image at the current moment based on the actual image at the current moment. The position image determination subunit is used to determine the position image at the current moment based on the actual image at the current moment. The speed image determination subunit is used to determine the speed image at the current moment based on the actual image at the current moment. The motion trajectory image determination subunit is used to determine the motion trajectory image based on the actual image of a preset time period. The drivable route image determination subunit is used to determine the speed image at the current moment based on the actual image at the current moment.

[0194] The motion state feature determination unit is used to determine the motion state feature corresponding to the motion state according to the virtual image used to represent the motion state. Specifically, the motion state feature determination unit can be used to input the virtual image representing the motion state into a pre-trained first convolutional neural network model to obtain a feature vector corresponding to the virtual image as the motion state feature corresponding to the motion state.

[0195] Optionally, the spatial interaction feature determination module 704 at the current moment includes a pooling vector determination submodule and a spatial interaction feature determination submodule. The pooling vector determination submodule and the spatial interaction feature determination submodule are electrically connected.

[0196] The pooling vector determination submodule is used to pool the motion state features corresponding to the motion state of each target object at the current moment for each motion state, and obtain the pooling vector corresponding to the motion state.

[0197] The spatial interaction feature determination submodule is used to determine the spatial interaction features used to characterize the interaction status of each target object in space at the current moment according to the pooling vectors corresponding to each motion state.

[0198] Optionally, the prediction module 708 includes a feature matrix determination submodule and a target object future motion state image determination submodule. The feature matrix determination submodule and the target object future motion state image determination submodule are electrically connected.

[0199] The feature matrix determination submodule is used to obtain, for each target object, a feature matrix corresponding to the target object according to the motion state characteristics corresponding to the target object and the time interaction characteristics.

[0200] The image determination submodule of the future motion state of the target object is used to input the feature matrix into a pre-trained second convolutional neural network model to obtain an image used to characterize the future motion state of the target object.

[0201] The present specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The process of predicting the motion state of a target object is provided.

[0202] The embodiments of this specification also propose Figure 8 The schematic structure diagram of the electronic device shown in FIG. Figure 8 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The process of predicting the motion state of the target object. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0203] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0204] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0205] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0206] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0207] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0209] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0211] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0212] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0213] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0214] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0215] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0216] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0217] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0218] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for predicting the motion state of a target object, characterized in that: The method comprises: Get the motion state of each target object at the current moment; For each target object, according to the motion state corresponding to the target object, determine the motion state characteristics corresponding to the target object at the current moment; For each motion state, the motion state features corresponding to the motion state of each target object at the current moment are pooled to obtain a pooling vector corresponding to the motion state; The pooling vectors corresponding to several features in each motion state feature are spliced ​​to obtain a spatial interaction feature for characterizing the interaction status of each target object in space at the current moment; Determine, according to the spatial interaction features at the current moment and the predetermined spatial interaction features of each target object at a historical moment, a temporal interaction feature for characterizing how the spatial interaction features of each target object change over time; For each target object, the future motion state of the target object is predicted according to the motion state characteristics corresponding to the target object and the time interaction characteristics.

2. The method according to claim 1, characterized in that The motion state corresponding to the target object is: at least one of an environment map, speed, position, motion trajectory and drivable route corresponding to the target object.

3. The method according to claim 2, characterized in that Get the motion state of each target object at the current moment, including: Collect the actual image at the current moment; According to the image of each target object contained in the actual image, the motion state corresponding to each target object is determined.

4. The method according to claim 3, characterized in that According to the motion state corresponding to the target object, the motion state characteristics corresponding to the target object at the current moment are determined, specifically including: For each motion state corresponding to the target object at the current moment, determining a virtual image used to represent the motion state; A virtual image used to represent the motion state is input into a pre-trained first convolutional neural network model to obtain a feature vector corresponding to the virtual image as a motion state feature corresponding to the motion state.

5. The method according to claim 3, characterized in that According to the spatial interaction features at the current moment and the predetermined spatial interaction features of each target at a historical moment, a temporal interaction feature for characterizing how the spatial interaction features of each target change over time is determined, specifically including: The spatial interaction features at the current moment are input into a pre-trained LSTM model to determine a feature vector corresponding to the spatial interaction features at the current moment as a temporal interaction feature for characterizing how the spatial interaction features of each target object change over time.

6. The method according to claim 5, characterized in that For each target object, based on the motion state characteristics corresponding to the target object and the time interaction characteristics, predicting the future motion state of the target object specifically includes: For each target object, a feature matrix corresponding to the target object is obtained according to the motion state feature corresponding to the target object and the time interaction feature; The feature matrix is ​​input into a pre-trained second convolutional neural network model to obtain an image for characterizing the future motion state of the target object.

7. A device for predicting the motion state of a target object, characterized in that: include: An acquisition module is used to obtain the motion state corresponding to each target object at the current moment; A motion state feature determination module is used to determine, for each target object, the motion state feature corresponding to the target object at the current moment according to the motion state corresponding to the target object; The spatial interaction feature determination module at the current moment is used to pool the motion state features corresponding to the motion state of each target object at the current moment for each motion state, and obtain the pooling vector corresponding to the motion state; the pooling vectors corresponding to several features in each motion state feature are spliced ​​to obtain the spatial interaction feature used to characterize the interaction status of each target object in space at the current moment; A temporal interaction feature determination module, used to determine a temporal interaction feature for characterizing how the spatial interaction feature of each target object changes over time based on the spatial interaction feature at the current moment and the predetermined spatial interaction features of each target object at a historical moment; The prediction module is used to predict the future motion state of each target object according to the motion state characteristics corresponding to the target object and the time interaction characteristics.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

Images