Trajectory prediction method, device and storage medium

Abstract

The present disclosure provides a trajectory prediction method, device, and storage medium. The method includes predicting future occupancy grid information, optical flow information, and trajectory information of multiple dynamic objects in the target scene based on static information of the target scene, occupancy grid information, dynamic information, and historical trajectory information of the dynamic objects in the target scene; optimizing the optical flow information using the occupancy grid information of the dynamic objects, taking into account the spatial topological relationship between the dynamic objects in the target scene and the target scene; and then verifying the trajectory information using the optimized optical flow information to obtain the target trajectory of the dynamic object, thereby preventing the predicted target trajectory from being infeasible in the actual environment and improving the prediction accuracy of the target trajectory.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle driving technology, and in particular to a trajectory prediction method, device, and storage medium. Background Art

[0002] As intelligent driving technology matures, intelligent driving cars are being used in more and more scenarios, such as urban roads, closed campuses, and highways. These scenarios have clear lane markings and comprehensive traffic rules. However, in scenarios without standard lane markings and strictly enforced traffic rules, the surrounding vehicles have more complex dynamic changes. For example, in parking lots, the distance between vehicles is close and parking operations are frequent. Therefore, existing trajectory prediction methods cannot meet the vehicle trajectory prediction task. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a trajectory prediction method, device and storage medium.

[0004] According to a first aspect of the present disclosure, a trajectory prediction method is provided, the method comprising:

[0005] Acquire grid map information of multiple frames of a target scene, the grid map information including static information of the target scene, occupancy grid information of multiple dynamic objects in the target scene, dynamic information, and historical trajectory information, wherein the dynamic information is obtained by mapping information of the multiple dynamic objects sensed by the target vehicle onto the grid map;

[0006] Predicting the occupancy grid information, optical flow information, and trajectory information of the dynamic object in multiple frames in the future based on the grid map information of the multiple frames of the target scene;

[0007] Optimizing the optical flow information using the occupancy grid information of the dynamic object;

[0008] The trajectory information is verified using the optimized optical flow information to obtain the target trajectory of the dynamic object.

[0009] In any embodiment, the static information includes semantic information of each pixel in the grid map; the method further includes:

[0010] Get the semantic information of each pixel in the raster map;

[0011] Obtaining the locations of multiple dynamic objects in the target scene and size information of the dynamic objects;

[0012] Mapping the position and size information of the dynamic object to the grid map to obtain grid occupancy information of the dynamic object;

[0013] The dynamic information of the dynamic object is acquired, and the dynamic information is marked on each pixel point within the range of the occupied grid of the dynamic object to obtain grid map information of the target scene.

[0014] In any embodiment, the dynamic information includes the velocity, acceleration and attitude angle of the dynamic object;

[0015] The step of marking the dynamic information on each pixel within the range of the occupied grid of the dynamic object includes:

[0016] Determining the displacement vector of each pixel point within the range of the occupied grid of the dynamic object based on the speed, acceleration and attitude angle of the dynamic object and a preset time interval;

[0017] The components of the displacement vector of the pixel point in the x-axis direction and the y-axis direction are used as the optical flow information of the pixel point.

[0018] In any embodiment, the predicting of the occupancy grid information, optical flow information, and trajectory information of the dynamic object in multiple frames in the future based on the grid map information of the multiple frames of the target scene includes:

[0019] Performing feature extraction on the grid map information of the target scene in multiple frames to obtain features of the grid map information;

[0020] Inputting the features into a pre-trained occupancy grid and optical flow prediction network for prediction, thereby obtaining occupancy grid information and optical flow information of the dynamic object in multiple frames in the future;

[0021] The features are input into a pre-trained trajectory prediction network for prediction to obtain the trajectory information of the dynamic object in multiple frames in the future.

[0022] In any embodiment, extracting features from the grid map information of the target scene in multiple frames to obtain features of the grid map information includes:

[0023] The grid map information of each frame of the target scene is downsampled multiple times to obtain sub-features of different scales;

[0024] The sub-features of different scales are fused to obtain the features of the raster map information.

[0025] In any embodiment, the occupancy grid and flow prediction network is trained based on sample images annotated with occupancy grid labels and optical flow labels, wherein the optical flow labels are used to indicate the displacement difference between pixels in two adjacent frames of sample images;

[0026] The inputting the features into a pre-trained occupancy grid and optical flow prediction network for prediction to obtain occupancy grid information and optical flow information of the dynamic object in multiple frames in the future includes:

[0027] Using the occupancy grid and optical flow prediction network to fuse features corresponding to multi-frame grid map information;

[0028] The occupancy grid information and optical flow information of the dynamic object in multiple frames in the future are predicted based on the fused features.

[0029] In any embodiment, the trajectory prediction network is trained based on sample images labeled with real trajectories;

[0030] Inputting the features into a pre-trained trajectory prediction network for prediction to obtain trajectory information of the dynamic object in multiple frames in the future includes:

[0031] Using the trajectory prediction network, semantic enhancement is performed on the features corresponding to each frame of raster map information;

[0032] The trajectory information of the dynamic object in multiple frames in the future is predicted based on the enhanced features.

[0033] In any embodiment, the dynamic object corresponds to a plurality of optical flow information, and each piece of optical flow information corresponds to occupancy grid information;

[0034] The optimizing the optical flow information by utilizing the occupancy grid information of the dynamic object includes:

[0035] Obtaining target occupancy grid information that overlaps with occupancy grids of other dynamic objects in the occupancy grid information of the dynamic object;

[0036] Delete the optical flow information corresponding to the target occupancy grid information.

[0037] According to a second aspect of the present disclosure, a trajectory prediction device is provided, the device comprising:

[0038] an acquisition unit, configured to acquire grid map information of a target scene in multiple frames, the grid map information including static information of the target scene, grid occupancy information of multiple dynamic objects in the target scene, dynamic information, and historical trajectory information, wherein the dynamic information is obtained by mapping information of the multiple dynamic objects sensed by the target vehicle onto the grid map;

[0039] A prediction unit, configured to predict, based on the grid map information of the target scene in multiple frames, the occupancy grid information, the optical flow information, and the trajectory information of the dynamic object in multiple frames in the future;

[0040] an optimization unit, configured to optimize the optical flow information using the occupancy grid information of the dynamic object;

[0041] The verification unit is used to verify the trajectory information using the optimized optical flow information to obtain the target trajectory of the dynamic object.

[0042] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor to execute the method described in any embodiment of the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in any embodiment of the present disclosure is implemented.

[0044] The technical solution provided by the present disclosure may include the following beneficial effects: predicting the future occupancy grid information, optical flow information, and trajectory information of multiple dynamic objects in the target scene based on static information of the target scene, occupancy grid information, dynamic information, and historical trajectory information of the dynamic objects in the target scene; optimizing the optical flow information using the occupancy grid information of the dynamic objects, taking into account the spatial topological relationship between the dynamic objects in the target scene and the target scene; and then verifying the trajectory information using the optimized optical flow information to obtain the target trajectory of the dynamic object, thereby avoiding the predicted target trajectory from being infeasible in the actual environment and improving the prediction accuracy of the target trajectory.

[0045] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, are used to explain the technical solutions of the present disclosure.

[0047] Figure 1 FIG. 4 is a flowchart of a trajectory prediction method according to an exemplary embodiment of the present disclosure.

[0048] Figure 2 FIG. 4 is a schematic diagram showing occupancy grid information in a parking lot according to an exemplary embodiment of the present disclosure.

[0049] Figure 3 FIG. 4 is a schematic diagram showing optical flow information in a parking lot according to an exemplary embodiment of the present disclosure.

[0050] Figure 4 The figure is a schematic diagram of a specific flow chart of a trajectory prediction method according to an exemplary embodiment of the present disclosure.

[0051] Figure 5 The figure is a schematic structural diagram of a trajectory prediction device according to an exemplary embodiment of the present disclosure.

[0052] Figure 6 The figure is a schematic diagram of the structure of an electronic device for trajectory prediction according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0054] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0055] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0056] Currently, when predicting the movement trajectory of dynamic objects, the method usually predicts the trajectory points based on map information and the historical trajectory information of the dynamic object, and then generates the movement trajectory of the dynamic object. However, this prediction method is less efficient in scenarios where the surrounding environment changes rapidly. For example, in parking lot scenarios, since the space in the parking lot is more crowded than the road, the surrounding obstacles are closer to the vehicle, and the vehicle parking operation is usually more complicated than road driving. As a result, the accuracy of predicting the target trajectory of dynamic objects in the parking lot using existing trajectory prediction methods is relatively low, increasing the risk of collision.

[0057] In view of this, the present disclosure provides a trajectory prediction method that can predict the trajectories of vehicles performing parking operations and other dynamic obstacles around a target vehicle, so as to predict risks and avoid accidents.

[0058] The following embodiments will illustrate the trajectory prediction method provided by the present disclosure with reference to the accompanying drawings.

[0059] Figure 1 is a flow chart of a trajectory prediction method according to an exemplary embodiment of the present disclosure, as shown in FIG. Figure 1 As shown, the trajectory prediction method includes the following steps 101 to 104.

[0060] In step 101, grid map information of a multi-frame target scene is obtained, wherein the grid map information includes static information of the target scene, occupancy grid information of multiple dynamic objects in the target scene, dynamic information, and historical trajectory information, wherein the dynamic information is obtained by mapping information of the multiple dynamic objects sensed by the target vehicle onto the grid map.

[0061] Usually, the target trajectory of a dynamic object in the next multiple frames can be predicted based on the historical trajectory information of the dynamic object in multiple frames. However, since the movement of dynamic objects in some target scenes is relatively frequent, the static information of the target scene, as well as the occupancy grid information and dynamic information of multiple dynamic objects in the target scene can be considered when presetting the trajectory.

[0062] In this embodiment, a map of the target scene can be obtained in advance, n pixels can be sampled in a manner of one pixel per m meters, semantic information of each pixel can be obtained, and pixels with semantic information can be merged to obtain a grid map.

[0063] The dynamic object is mapped onto the grid map according to the coordinates of the dynamic object, and the grid occupied by the dynamic object on the grid map is determined according to the size information of the dynamic object.

[0064] In this embodiment, the operation information of multiple dynamic objects sensed by the target vehicle can be obtained, and the motion information can be mapped onto a grid map to obtain the dynamic information of the dynamic objects.

[0065] In step 102, the grid occupancy information, optical flow information and trajectory information of the dynamic object in the future multiple frames are predicted based on the grid map information of the multiple frames of the target scene.

[0066] The optical flow information is used to represent the motion vector of the pixel point of the dynamic object.

[0067] In this embodiment, a second amount of future raster map information can be predicted based on the first amount of raster map information, where the first amount and the second amount can be determined according to actual needs. For example, the first amount can be 10, 11, etc., and the second amount can be 8, 9, etc. Usually, the first amount is greater than the second amount, and the last raster map information in the first amount is adjacent to the first raster map information in the second amount.

[0068] In an embodiment of the present disclosure, a pre-trained network model may be used to predict occupancy grid information, optical flow information, and trajectory information of future multi-frame dynamic objects.

[0069] Using the grid map information of the target scene in multiple frames, it is possible to predict multiple optical flow information and multiple trajectory information of the dynamic object in multiple frames in the future that conform to the spatial relationship. Each optical flow information and trajectory information can have a probability value for describing the dynamic object selecting the optical flow information and trajectory information.

[0070] In step 103, the optical flow information is optimized using the occupancy grid information of the dynamic object.

[0071] The occupancy grid information of the dynamic object can represent the position of the dynamic object in the current frame. The optical flow information can be optimized based on the occupancy grid information of the dynamic object to obtain optical flow information that conforms to the spatial topological relationship in the target scene.

[0072] In step 104, the trajectory information is verified using the optimized optical flow information to obtain the target trajectory of the dynamic object.

[0073] The trajectory information that is inconsistent with the optimized optical flow information can be deleted, and the trajectory information with the same motion trend as that represented by the optical flow information can be obtained based on the optimized optical flow information as the target trajectory of the dynamic object.

[0074] Based on the static information of the target scene, the occupancy grid information, dynamic information and historical trajectory information of multiple dynamic objects in the target scene, the future occupancy grid information, optical flow information and trajectory information of the dynamic objects are predicted; the occupancy grid information of the dynamic objects is used to optimize the optical flow information, taking into account the spatial topological relationship between the dynamic objects in the target scene and the target scene; the trajectory information is then verified using the optimized optical flow information to obtain the target trajectory of the dynamic object, thereby avoiding the predicted target trajectory from being unfeasible in the actual environment and improving the prediction accuracy of the target trajectory.

[0075] In some embodiments, the static information may include semantic information of each pixel in the grid map; in this case, the method further includes: obtaining semantic information of each pixel in the grid map; obtaining the positions of multiple dynamic objects in the target scene and size information of the dynamic objects; mapping the positions and size information of the dynamic objects to the grid map to obtain grid occupation information of the dynamic objects; obtaining dynamic information of the dynamic objects, marking the dynamic information on each pixel within the range of the dynamic object's occupied grid, and obtaining grid map information of the target scene.

[0076] In this embodiment, the pre-stored semantic information of pixels in the grid map can be updated based on the acquired semantic information of each pixel in the grid map. The semantic information may include drivable roads, open parking spaces, and static obstacles (e.g., vehicles parked in parking spaces). If the acquired semantic information of a pixel indicates an obstacle, such as a vehicle or pedestrian, while the pre-stored semantic information of the pixel indicates a drivable road, indicating that the obstacle blocks the drivable road, the semantic information of the pixel in the grid map is updated to indicate an obstacle.

[0077] By obtaining the target vehicle's location information (i.e., GPS coordinates) and the position information of other dynamic objects relative to the target vehicle collected by the target vehicle's sensing module, the target vehicle's position coordinates on the grid map can be determined based on the conversion relationship between the world coordinate system and the image coordinate system. Furthermore, the position coordinates of the dynamic objects on the grid map can be determined based on their positional relationships relative to the target vehicle. The occupancy grid information corresponding to the dynamic objects can be determined based on their position coordinates on the grid map and their size.

[0078] In some embodiments, the dynamic information includes the speed, acceleration and attitude angle of the dynamic object; in this case, marking the dynamic information on each pixel point within the range of the occupied grid of the dynamic object may include: determining the displacement vector of each pixel point within the range of the occupied grid of the dynamic object based on the speed, acceleration and attitude angle of the dynamic object, and a preset time interval; and using the components of the displacement vector of the pixel point in the x-axis direction and the y-axis direction as the optical flow information of the pixel point.

[0079] The target vehicle's sensing module acquires dynamic information of dynamic objects at a set frequency during the target vehicle's operation. This dynamic information may include, for example, velocity, acceleration, attitude angle, angular velocity, and angular acceleration. To facilitate calculations, the dynamic information of dynamic objects collected by the target vehicle's sensing module may be converted to obtain data centered on the target vehicle.

[0080] In one embodiment, a grid map of the target scene can be cached to reduce redundant computations and lower memory usage. During prediction, the acquired static information of the target scene, the dynamic information of multiple dynamic objects within the target scene, and the location information of the dynamic objects are mapped onto the pre-stored grid map to obtain the grid map information of the target scene.

[0081] In some embodiments, the predicting of the occupancy grid information, optical flow information and trajectory information of the dynamic object in multiple frames in the future based on the grid map information of the multiple frames of the target scene may include: extracting features from the grid map information of the multiple frames of the target scene to obtain features of the grid map information; inputting the features into a pre-trained occupancy grid and optical flow prediction network for prediction to obtain the occupancy grid information and optical flow information of the dynamic object in multiple frames in the future; and inputting the features into a pre-trained trajectory prediction network for prediction to obtain trajectory information of the dynamic object in multiple frames in the future.

[0082] In this embodiment, the grid map information of each frame of the target scene may be downsampled multiple times to obtain sub-features of different scales; and the sub-features of different scales may be fused to obtain the features of the grid map information.

[0083] In some embodiments, the occupancy grid and optical flow prediction network is trained based on sample images annotated with occupancy grid labels and optical flow labels, where the optical flow labels are used to indicate the displacement difference of pixels in two adjacent frames of sample images.

[0084] Among them, the feature map corresponding to the sample image used to train the occupancy grid and optical flow prediction network can include multiple dimensions, such as channel dimension, height dimension, width dimension, batch dimension, etc. The format of the feature map can be expressed as [BHW], where B represents the batch dimension, H represents the height dimension, and W represents the width dimension. The labels of the occupied grid can be divided into two types: one is the observable grid with a shape of [Batch, H, W, 1]; the second is the occluded grid with a shape of [Batch, H, W, 1]. "1" refers to the category value 0 or 1, and 1 represents the presence of a grid at each position on the grid map. The label of the optical flow is the real flow with a shape of [Batch, H, W, 2]. "2" represents dx, dy, the difference in displacement between the previous frame and the current frame.

[0085] In this case, the feature is input into a pre-trained occupancy grid and optical flow prediction network for prediction to obtain the occupancy grid information and optical flow information of the dynamic object in multiple frames in the future, including: using the occupancy grid and flow prediction network to fuse the features corresponding to the multiple frames of grid map information; and predicting the occupancy grid information and optical flow information of the dynamic object in multiple frames in the future based on the fused features.

[0086] Figure 2 is a schematic diagram of occupancy grid information in a parking lot according to an exemplary embodiment of the present disclosure, such as Figure 2 As shown, occupancy grid 21 is used to represent occupancy grid information corresponding to a vehicle parked in a parking space, and occupancy grid 22 and occupancy grid 23 are used to represent occupancy grid information corresponding to a vehicle in motion.

[0087] Figure 3 is a schematic diagram of optical flow information in a parking lot according to an exemplary embodiment of the present disclosure. Figure 3 As shown, the optical flow information of dynamic objects in the parking lot can be predicted.

[0088] In some embodiments, the trajectory prediction network is trained based on sample images labeled with real trajectories; for example, the trajectory label can be the true value of the trajectory of the next frame, with a shape of [Batch, 1, 60, 2], where "2" indicates whether a trajectory exists at the x, y position.

[0089] In this embodiment, the features are input into a pre-trained trajectory prediction network for prediction to obtain the trajectory information of the dynamic object in multiple frames in the future, including: using the trajectory prediction network to semantically enhance the features corresponding to each frame of raster map information; and predicting the trajectory information of the dynamic object in multiple frames in the future based on the enhanced features.

[0090] In this embodiment, the features corresponding to each frame of raster map information can be input into an encoder-decoder structure, where the encoder can be a Transformer encoder. The enhanced features are then input into a decoder, where the decoder can be a convolutional neural network, a recurrent neural network, a Transformer decoder, or other methods, to generate predicted trajectory information.

[0091] In some embodiments, the dynamic object corresponds to multiple optical flow information, and each piece of optical flow information corresponds to occupancy grid information; in this case, optimizing the optical flow information using the occupancy grid information of the dynamic object may include: obtaining target occupancy grid information that overlaps with the occupancy grids of other dynamic objects in the occupancy grid information of the dynamic object; and deleting the optical flow information corresponding to the target occupancy grid information.

[0092] Figure 4 FIG. 1 is a schematic diagram of a specific process of a trajectory prediction method according to an exemplary embodiment of the present disclosure. Figure 4 As shown, in one embodiment, trajectory prediction can be performed according to the following steps.

[0093] In step 401, information collected by a sensing module of a target vehicle is obtained, and the obtained information is preprocessed to obtain static information of a target scene, occupancy grid information of multiple dynamic objects in the target scene, dynamic information, and historical trajectory information.

[0094] In step 402, grid map information of a multi-frame target scene is acquired, and feature extraction is performed on the grid map information of the multi-frame target scene to obtain features of the grid map information.

[0095] In step 403, features of multiple frames of raster map information are aggregated to obtain aggregated features.

[0096] In step 404a, the aggregated features are input into the pre-trained occupancy grid and optical flow prediction network for prediction to obtain the occupancy grid information and optical flow information of the dynamic object in multiple frames in the future, and the optical flow information is optimized using the occupancy grid information of the dynamic object.

[0097] In step 404b, the aggregated features are input into a pre-trained trajectory prediction network for prediction to obtain trajectory information of the dynamic object in multiple frames in the future.

[0098] In step 405, the trajectory information is verified using the optimized optical flow information to obtain the target trajectory of the dynamic object.

[0099] Corresponding to the aforementioned method embodiments, the present disclosure also provides embodiments of a device and a terminal to which the device is applied.

[0100] Figure 5 FIG. 1 is a schematic diagram of a trajectory prediction device according to an exemplary embodiment of the present disclosure. Figure 5 As shown, the trajectory prediction device includes an acquisition unit 501 , a prediction unit 502 , an optimization unit 503 and a verification unit 504 .

[0101] An acquisition unit 501 is configured to acquire grid map information of a target scene in multiple frames, wherein the grid map information includes static information of the target scene, grid occupancy information of multiple dynamic objects in the target scene, dynamic information, and historical trajectory information. The dynamic information is obtained by mapping information of the multiple dynamic objects sensed by the target vehicle onto the grid map.

[0102] A prediction unit 502 is configured to predict, based on the grid map information of the target scene in multiple frames, the occupancy grid information, the optical flow information, and the trajectory information of the dynamic object in multiple frames in the future;

[0103] an optimization unit 503, configured to optimize the optical flow information using the occupancy grid information of the dynamic object;

[0104] The verification unit 504 is configured to verify the trajectory information using the optimized optical flow information to obtain the target trajectory of the dynamic object.

[0105] In some embodiments, the static information includes semantic information of each pixel in the grid map; the acquiring unit 501 is further configured to:

[0106] Get the semantic information of each pixel in the raster map;

[0107] Obtaining the locations of multiple dynamic objects in the target scene and size information of the dynamic objects;

[0108] Mapping the position and size information of the dynamic object to the grid map to obtain grid occupancy information of the dynamic object;

[0109] The dynamic information of the dynamic object is acquired, and the dynamic information is marked on each pixel point within the range of the occupied grid of the dynamic object to obtain grid map information of the target scene.

[0110] In some embodiments, the dynamic information includes the velocity, acceleration, and attitude angle of the dynamic object;

[0111] The acquisition unit 501 is specifically configured to:

[0112] Determining the displacement vector of each pixel point within the range of the occupied grid of the dynamic object based on the speed, acceleration and attitude angle of the dynamic object and a preset time interval;

[0113] The components of the displacement vector of the pixel point in the x-axis direction and the y-axis direction are used as the optical flow information of the pixel point.

[0114] In some embodiments, the prediction unit 502 is specifically configured to:

[0115] Performing feature extraction on the grid map information of the target scene in multiple frames to obtain features of the grid map information;

[0116] Inputting the features into a pre-trained occupancy grid and optical flow prediction network for prediction, thereby obtaining occupancy grid information and optical flow information of the dynamic object in multiple frames in the future;

[0117] The features are input into a pre-trained trajectory prediction network for prediction to obtain the trajectory information of the dynamic object in multiple frames in the future.

[0118] In some embodiments, the prediction unit 502 is specifically configured to:

[0119] The grid map information of each frame of the target scene is downsampled multiple times to obtain sub-features of different scales;

[0120] The sub-features of different scales are fused to obtain the features of the raster map information.

[0121] In some embodiments, the occupancy grid and flow prediction network is trained based on sample images labeled with occupancy grid labels and optical flow labels, where the optical flow labels are used to indicate the displacement difference between pixels in two adjacent frames of sample images; the prediction unit 502 is specifically configured to:

[0122] Using the occupancy grid and optical flow prediction network to fuse features corresponding to multi-frame grid map information;

[0123] The occupancy grid information and optical flow information of the dynamic object in multiple frames in the future are predicted based on the fused features.

[0124] In some embodiments, the trajectory prediction network is trained based on sample images labeled with real trajectories; the prediction unit 502 is specifically configured to:

[0125] Using the trajectory prediction network, semantic enhancement is performed on the features corresponding to each frame of raster map information;

[0126] The trajectory information of the dynamic object in multiple frames in the future is predicted based on the enhanced features.

[0127] In some embodiments, the dynamic object corresponds to multiple pieces of optical flow information, and each piece of optical flow information corresponds to occupancy grid information; the optimization unit 503 specifically includes:

[0128] Obtaining target occupancy grid information that overlaps with occupancy grids of other dynamic objects in the occupancy grid information of the dynamic object;

[0129] Delete the optical flow information corresponding to the target occupancy grid information.

[0130] Figure 6 A schematic diagram of the electronic device structure for trajectory prediction provided by at least one embodiment of the present disclosure. Figure 6 As shown, the electronic device includes a memory and a processor, the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the trajectory prediction method described in any embodiment of the present disclosure when executing the computer instructions.

[0131] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any trajectory prediction method described in the present disclosure.

[0132] It will be understood by those skilled in the art that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] In the present disclosure, “and / or” means at least one of the two. For example, “A and / or B” includes three solutions: A, B, and “A and B”.

[0134] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the data processing device embodiment is generally similar to the method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the method embodiment.

[0135] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0136] Embodiments of the subject matter and functional operations described in this disclosure may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this disclosure and their structural equivalents, or a combination of one or more thereof. Embodiments of the subject matter described in this disclosure may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

[0137] The processes and logic flows described in this disclosure can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0138] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such mass storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0139] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0140] Although the present disclosure includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of the claimed protection, but are primarily used to describe the features of the specific embodiments of a particular invention. Certain features described in multiple embodiments within the present disclosure may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may function in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0141] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0142] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0143] The above description is merely a preferred embodiment of one or more embodiments of the present disclosure and is not intended to limit one or more embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present disclosure shall be included in the scope of protection of one or more embodiments of the present disclosure.

Claims

1. A trajectory prediction method, characterized in that: The method comprises: Acquire grid map information of multiple frames of a target scene, the grid map information including static information of the target scene, occupancy grid information of multiple dynamic objects in the target scene, dynamic information, and historical trajectory information, wherein the dynamic information is obtained by mapping information of the multiple dynamic objects sensed by the target vehicle onto the grid map; Predicting the occupancy grid information, optical flow information, and trajectory information of the dynamic object in multiple frames in the future based on the grid map information of the multiple frames of the target scene; Optimizing the optical flow information using the occupancy grid information of the dynamic object; The trajectory information is verified using the optimized optical flow information to obtain the target trajectory of the dynamic object.

2. The method according to claim 1, characterized in that The static information includes semantic information of each pixel in the grid map; the method further includes: Get the semantic information of each pixel in the raster map; Obtaining the locations of multiple dynamic objects in the target scene and size information of the dynamic objects; Mapping the position and size information of the dynamic object to the grid map to obtain grid occupancy information of the dynamic object; The dynamic information of the dynamic object is acquired, and the dynamic information is marked on each pixel point within the range of the occupied grid of the dynamic object to obtain grid map information of the target scene.

3. The method according to claim 2, characterized in that The dynamic information includes the speed, acceleration and attitude angle of the dynamic object; The step of marking the dynamic information on each pixel within the range of the occupied grid of the dynamic object includes: Determining the displacement vector of each pixel point within the range of the occupied grid of the dynamic object based on the speed, acceleration and attitude angle of the dynamic object and a preset time interval; The components of the displacement vector of the pixel point in the x-axis direction and the y-axis direction are used as the optical flow information of the pixel point.

4. The method according to claim 1, wherein The method of predicting the occupancy grid information, optical flow information, and trajectory information of the dynamic object in multiple frames in the future based on the grid map information of the multiple frames of the target scene includes: Performing feature extraction on the grid map information of the target scene in multiple frames to obtain features of the grid map information; Inputting the features into a pre-trained occupancy grid and optical flow prediction network for prediction, thereby obtaining occupancy grid information and optical flow information of the dynamic object in multiple frames in the future; The features are input into a pre-trained trajectory prediction network for prediction to obtain the trajectory information of the dynamic object in multiple frames in the future.

5. The method according to claim 4, characterized in that The feature extraction of the grid map information of the multi-frame target scene to obtain the features of the grid map information includes: The grid map information of each frame of the target scene is downsampled multiple times to obtain sub-features of different scales; The sub-features of different scales are fused to obtain the features of the raster map information.

6. The method according to claim 4, characterized in that The occupancy grid and flow prediction network is trained based on sample images annotated with occupancy grid labels and optical flow labels, where the optical flow labels are used to indicate the displacement difference between pixels in two adjacent frames of sample images. The inputting the features into a pre-trained occupancy grid and optical flow prediction network for prediction to obtain occupancy grid information and optical flow information of the dynamic object in multiple frames in the future includes: Using the occupancy grid and optical flow prediction network to fuse features corresponding to multi-frame grid map information; The occupancy grid information and optical flow information of the dynamic object in multiple frames in the future are predicted based on the fused features.

7. The method according to claim 4, characterized in that The trajectory prediction network is trained based on sample images marked with real trajectories; Inputting the features into a pre-trained trajectory prediction network for prediction to obtain trajectory information of the dynamic object in multiple frames in the future includes: Using the trajectory prediction network, semantic enhancement is performed on the features corresponding to each frame of raster map information; The trajectory information of the dynamic object in multiple frames in the future is predicted based on the enhanced features.

8. The method according to any one of claims 1 to 7, characterized in that The dynamic object corresponds to multiple pieces of optical flow information, and each piece of optical flow information corresponds to occupancy grid information; The optimizing the optical flow information by utilizing the occupancy grid information of the dynamic object includes: Obtaining target occupancy grid information that overlaps with occupancy grids of other dynamic objects in the occupancy grid information of the dynamic object; Delete the optical flow information corresponding to the target occupancy grid information.

9. A trajectory prediction device, characterized in that: The device comprises: an acquisition unit, configured to acquire grid map information of a target scene in multiple frames, the grid map information including static information of the target scene, grid occupancy information of multiple dynamic objects in the target scene, dynamic information, and historical trajectory information, wherein the dynamic information is obtained by mapping information of the multiple dynamic objects sensed by the target vehicle onto the grid map; A prediction unit, configured to predict, based on the grid map information of the target scene in multiple frames, the occupancy grid information, the optical flow information, and the trajectory information of the dynamic object in multiple frames in the future; an optimization unit, configured to optimize the optical flow information using the occupancy grid information of the dynamic object; The verification unit is used to verify the trajectory information using the optimized optical flow information to obtain the target trajectory of the dynamic object.

10. An electronic device, characterized in that: The device comprises: processor; A memory for storing processor-executable instructions for executing the method according to any one of claims 1 to 8.

Images