Trajectory prediction method and device, readable storage medium and electronic equipment

Abstract

The embodiment of the present disclosure discloses a trajectory prediction method and device, a readable storage medium and an electronic device, wherein the method comprises: obtaining the perception result information of the dynamic obstacle in each frame of image in the plurality of images based on the plurality of images; then determining the state quantity information of the dynamic obstacle according to the perception result information of the dynamic obstacle in each frame of image; and then predicting the motion trajectory of the dynamic obstacle based on the map information, the perception result information of the dynamic obstacle in each frame of image and the state quantity information of the dynamic obstacle by using a trajectory prediction network. As can be seen, in the embodiment of the present disclosure, the image information of the dynamic obstacle is fully developed and utilized to predict the motion trajectory of the dynamic obstacle, which helps to improve the accuracy of predicting the motion trajectory of the dynamic obstacle.

Description

Technical Field

[0001] This disclosure relates to autonomous driving technology and computer vision technology, and in particular to a trajectory prediction method, apparatus, readable storage medium, and electronic device. Background Technology

[0002] In autonomous driving technology, to better plan driving routes, it is necessary to predict the future trajectories of various dynamic obstacles in the environment. Current methods for predicting the trajectory of dynamic obstacles typically involve inputting an image of the obstacle into a trajectory prediction model, which then predicts the obstacle's future trajectory. However, the accuracy of these trajectory prediction models is relatively poor. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a trajectory prediction method, apparatus, readable storage medium, and electronic device.

[0004] According to one aspect of the present disclosure, a trajectory prediction method is provided, comprising: acquiring multiple frames of images and map information, wherein the multiple frames of images are multiple frames of images acquired by an image acquisition device on a vehicle within a time period for the environment surrounding the vehicle; obtaining perception result information of dynamic obstacles in each frame of the multiple frames of images; determining state quantity information of the dynamic obstacles based on the perception result information of the dynamic obstacles in each frame of images; and predicting the motion trajectory of the dynamic obstacles using a trajectory prediction network based on the map information, the perception result information of the dynamic obstacles in each frame of images, and the state quantity information of the dynamic obstacles.

[0005] According to another aspect of the present disclosure, a trajectory prediction device is provided, comprising: an image acquisition module for acquiring multiple frames of images and map information, wherein the multiple frames of images are multiple frames of images acquired by an image acquisition device on a vehicle within a time period of the environment surrounding the vehicle; a perception result acquisition module for obtaining perception result information of dynamic obstacles in each frame of the multiple frames of images; a state quantity acquisition module for determining state quantity information of the dynamic obstacles based on the perception result information of the dynamic obstacles in each frame of images; and a trajectory prediction module for predicting the motion trajectory of the dynamic obstacles using a trajectory prediction network based on the map information, the perception result information of the dynamic obstacles in each frame of images, and the state quantity information of the dynamic obstacles.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the trajectory prediction method according to any embodiment of the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the trajectory prediction method according to any embodiment of the present disclosure.

[0008] Based on the trajectory prediction method, apparatus, readable storage medium, and electronic device provided in the above embodiments of this disclosure, based on multiple frames of images, perception result information of dynamic obstacles in each frame of the multiple images is obtained; then, based on the perception result information of dynamic obstacles in each frame of the images, the state quantity information of the dynamic obstacles is determined; then, using a trajectory prediction network, based on map information, the state quantity information of the dynamic obstacles, and the perception result information of dynamic obstacles in each frame of the images, the motion trajectory of the dynamic obstacles is predicted. Therefore, in the embodiments of this disclosure, the perception result information and state quantity information of dynamic obstacles in each frame of the multiple images are used as the basic prediction information for trajectory prediction to predict the motion trajectory of dynamic obstacles. This fully develops and utilizes the image information of dynamic obstacles to predict their motion trajectory, which helps to improve the accuracy of predicting the motion trajectory of dynamic obstacles. Furthermore, by combining map information, the embodiments of this disclosure enable the trajectory prediction network to better distinguish between dynamic obstacles and their surrounding environment, thereby further improving the accuracy of the motion trajectory of dynamic obstacles.

[0009] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 This is a scene diagram to which the trajectory prediction method of this disclosure is applicable.

[0012] Figure 2 This is a schematic flowchart of a trajectory prediction method provided in an exemplary embodiment of this disclosure.

[0013] Figure 3 This is a flowchart illustrating step S240 provided in an exemplary embodiment of this disclosure.

[0014] Figure 4This is a flowchart illustrating step S242 provided in an exemplary embodiment of this disclosure.

[0015] Figure 5 This is a schematic diagram of the feature extraction region of a dynamic obstacle A provided in an exemplary embodiment of this disclosure.

[0016] Figure 6 This is a flowchart illustrating step S241 provided in an exemplary embodiment of this disclosure.

[0017] Figure 7 This is a flowchart illustrating an application example provided by an exemplary embodiment of this disclosure.

[0018] Figure 8 This is a schematic diagram of the trajectory prediction device provided in an exemplary embodiment of this disclosure.

[0019] Figure 9 This is a schematic diagram of the trajectory prediction module provided in an exemplary embodiment of this disclosure.

[0020] Figure 10 This is a schematic diagram of the structure of the second feature extraction submodule provided in an exemplary embodiment of this disclosure.

[0021] Figure 11 This is a schematic diagram of the structure of the first feature extraction submodule provided in an exemplary embodiment of this disclosure.

[0022] Figure 12 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0023] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0024] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0025] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0026] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0027] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0028] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0029] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0030] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0031] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0032] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0034] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0035] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0036] Application Overview

[0037] In the process of realizing this disclosure, the inventors discovered that in existing methods for predicting the trajectory of dynamic obstacles, only the positional information of the dynamic obstacle is used to predict its movement trajectory, resulting in poor accuracy of trajectory prediction.

[0038] Exemplary System

[0039] The technical solution disclosed herein can be used to assist vehicle driving. For example, it can assist a vehicle in planning its driving route in an autonomous driving system.

[0040] Figure 1 This is an exemplary application scenario of the trajectory prediction method provided in this disclosure. For example... Figure 1 As shown, an image acquisition device is installed on the vehicle. This device acquires multiple frames of images of the environment surrounding the vehicle over a given time period and transmits them to a computing platform. The computing platform processes these frames to obtain perception information of dynamic obstacles in each frame. Based on this perception information, it determines the state information of the dynamic obstacles in each frame. Then, using a trajectory prediction network, based on pre-stored map information, the state information of the dynamic obstacles, and the perception information of the dynamic obstacles in each frame, it obtains the motion trajectory of the dynamic obstacles. The computing platform transmits the motion trajectory of the dynamic obstacles to the vehicle's control platform. The image acquisition device can be a monocular camera or a spherical camera, etc.; the computing platform can be a backend server, etc., and can be located on the vehicle or elsewhere; the dynamic obstacles are moving objects in the environment surrounding the vehicle equipped with the image acquisition device, such as cars, pedestrians, animals, bicycles, etc.; the vehicle can be a vehicle, aircraft, etc.

[0041] In this embodiment of the disclosure, the perception result information and state information of the dynamic obstacle in each frame of the multi-frame image are used as the basic prediction information for trajectory prediction to predict the motion trajectory of the dynamic obstacle. Fully developing and utilizing the image information of the dynamic obstacle to predict the motion trajectory of the dynamic obstacle helps to improve the accuracy of predicting the motion trajectory of the dynamic obstacle.

[0042] In addition, by incorporating map information, the trajectory prediction network can better distinguish between dynamic obstacles and their surrounding environment, thereby further improving the accuracy of the movement trajectory of dynamic obstacles.

[0043] Exemplary methods

[0044] Figure 2 This is a schematic flowchart illustrating a trajectory prediction method provided in an exemplary embodiment of this disclosure. This disclosure is applicable, for example but not limited to, to electronic devices or vehicles, such as... Figure 2 As shown, it includes the following steps:

[0045] Step S210: Obtain multiple frames of images and map information.

[0046] The multi-frame images are multiple frames of images of the surrounding environment of the vehicle captured by the image acquisition device on the vehicle within a time period. The time period can be set according to actual needs.

[0047] In one implementation, the multi-frame image includes at least one historical image arranged according to temporal sequence and the current image. The historical image is the image that precedes the current image in temporal sequence. The multi-frame image can be acquired by an image acquisition device mounted on a vehicle, such as a monocular camera or a spherical camera.

[0048] In one implementation, each frame in the multi-frame image set includes dynamic obstacles. Image recognition techniques can be used to detect and determine whether the images contain dynamic obstacles. For example, a pre-trained neural network for detecting dynamic obstacles can be used to detect dynamic obstacles in the images acquired by the image acquisition device, and the images containing dynamic obstacles can be selected to form a multi-frame image set. The dynamic obstacles can be moving objects in the environment surrounding the vehicle equipped with the image acquisition device; for example, dynamic obstacles can be cars, pedestrians, animals, and bicycles. The neural network used for detecting dynamic obstacles can be a CNN (Convolutional Neural Network), etc.

[0049] Map information can include semantic segmentation information, which includes classification information for each pixel in the map. For example, pixel classification information can include lane lines, turn arrows, and stop signs.

[0050] In one implementation, the required map can be obtained from an existing map database. A pre-trained neural network for semantic segmentation can be used to perform semantic segmentation on the map, yielding the semantic segmentation result. Using a pre-defined table of classification information and color correspondence, along with the classification information of each pixel in the semantic segmentation result, color rendering is applied to each pixel in the semantic segmentation result to obtain the map's semantic segmentation information. This semantic segmentation result information is then used as the map information. The neural network used for semantic segmentation can be an FCN (Fully Convolutional Network), etc.

[0051] Step S220: Based on each frame of the multi-frame image, obtain the perception result information of dynamic obstacles in each frame of the image.

[0052] The perception results information for dynamic obstacles may include: the location, orientation, and size information of the dynamic obstacle. The orientation information may include the yaw angle of the dynamic obstacle. The size information may include the length and width, or the length, width, and height. The location information may be the location of the center point of the dynamic obstacle or the location of a preset point within the dynamic obstacle.

[0053] For example, the size information of a dynamic obstacle can be its length and width, or its length, width, and height, in the Vehicle Coordinate System (VCS). The position information of a dynamic obstacle can be the position of its center point in the VCS, or the position of a preset point within the obstacle. The orientation information of a dynamic obstacle can be its yaw angle in the VCS.

[0054] The vehicle coordinate system is used to describe the relative positional relationships between the vehicle and objects around it. The vehicle coordinate system can be defined by ISO (International Organization for Standardization), SAE (Society of Automotive Engineers), or based on IMU (Inertial Measurement Unit) coordinates.

[0055] In one implementation, when the position, orientation, and size information of a dynamic obstacle in the perception result information are in different coordinate systems, the position, orientation, and size information of the dynamic obstacle can be converted to the same coordinate system through coordinate system transformation. For example, the coordinate systems of the position, orientation, and size information of the dynamic obstacle can all be unified into the vehicle coordinate system.

[0056] Visual recognition technology can be used to process each frame of a multi-frame image to obtain the perception information of dynamic obstacles in each frame. For example, multiple frames of images can be input into a pre-trained neural network for detecting the perception information of dynamic obstacles. The output of this neural network can then be the perception information of dynamic obstacles in each frame. The neural network used for detecting the perception information of dynamic obstacles can be CNN, SCNN (Spatial Convolutional Neural Networks), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), ResNet (Residual Network), DenseNet (Depth Network), EfficientNet (Time-bound Network), etc. Specifically, the neural network used for detecting the perception information of dynamic obstacles can be trained using multiple sample images labeled with the location, orientation, and size information of the dynamic obstacles.

[0057] In one implementation, the perception results of dynamic obstacles in multiple frames can be optimized to improve the accuracy of determining the state information and predicting the trajectory information of dynamic obstacles based on the perception results of dynamic obstacles in each frame. For example, the perception results of dynamic obstacles in each frame can be smoothed using functions such as smoothing, smoothts, or filters. Alternatively, methods such as MOT (Multi-Object Tracking) can be used to track dynamic obstacles in each frame.

[0058] Step S230: Determine the state information of the dynamic obstacle based on the perception result information of the dynamic obstacle in each frame image.

[0059] The state information of the dynamic obstacle may include: the speed information, angular velocity information and / or acceleration information of the dynamic obstacle.

[0060] In one implementation, the velocity information of the dynamic obstacle can be determined using the position information of the dynamic obstacle in each frame of the image, the angular velocity information of the dynamic obstacle can be determined using the orientation information of the dynamic obstacle in each frame of the image, and the acceleration information of the dynamic obstacle can be determined using the velocity information of the dynamic obstacle.

[0061] For example, by using the positional information of a dynamic obstacle in two adjacent frames of a multi-frame image, and the time interval between corresponding moments in those two frames, the velocity information of the dynamic obstacle in those two frames can be obtained using a velocity formula. Similarly, the velocity information of the dynamic obstacle in all adjacent frames of the multi-frame image can be obtained. Then, the average value of the velocity information of the dynamic obstacle in all adjacent frames of the multi-frame image is determined as the velocity information of the dynamic obstacle. As another example, by using the orientation information of a dynamic obstacle in two adjacent frames of a multi-frame image, and the time interval between corresponding moments in those two frames, the angular velocity information of the dynamic obstacle in those two frames can be obtained using an angular velocity formula. Similarly, the angular velocity information of the dynamic obstacle in all adjacent frames of the multi-frame image can be obtained. Then, the average value of the angular velocity information of the dynamic obstacle in all adjacent frames of the multi-frame image is determined as the angular velocity information of the dynamic obstacle. As yet another example, based on the velocity information of the dynamic obstacle in all adjacent frames of a multi-frame image, and the time interval between corresponding moments in each frame, the acceleration information of the dynamic obstacle can be obtained using an acceleration formula.

[0062] Step S240: Using a trajectory prediction network, based on map information, perception results of dynamic obstacles in each frame image, and state information of dynamic obstacles, predict the motion trajectory of dynamic obstacles.

[0063] The trajectory prediction network can be a pre-trained neural network for trajectory prediction, which can be a single neural network or a composite neural network composed of multiple neural networks. For example, the neural network used for trajectory prediction can be CNN, SCNN, RNN, LSTM, etc.

[0064] In one implementation, map information, state information of dynamic obstacles, and perception results of dynamic obstacles in each frame of image can be input into a trajectory prediction network, which then outputs the predicted motion trajectory of the dynamic obstacles.

[0065] The predicted trajectory of a dynamic obstacle can include the obstacle's trajectory and its corresponding confidence level. The obstacle can have one or multiple trajectories, and the confidence level for each trajectory represents the probability that the obstacle will follow that trajectory.

[0066] In this embodiment, the perception results information and state information of dynamic obstacles in each frame of a multi-frame image are used as the basic prediction information for trajectory prediction to predict the motion trajectory of the dynamic obstacles. Fully developing and utilizing the image information of the dynamic obstacles to predict their motion trajectory helps improve the accuracy of the predicted trajectory. Furthermore, by combining semantic segmentation information from the map information, this embodiment enables the trajectory prediction network to better distinguish between dynamic obstacles and their surrounding environment, thereby further improving the accuracy of the dynamic obstacle's motion trajectory.

[0067] In an optional embodiment, the perception result information of the dynamic obstacle in this embodiment includes: the position information, orientation information and size information of the dynamic obstacle; step 220 may further include: using a perception network, based on multiple frames of images, obtaining the position information, orientation information and size information of the dynamic obstacle in each frame of the multiple frames of images.

[0068] In one implementation, the perception network can be a neural network trained to detect perception results of dynamic obstacles. For example, the perception network can be CNN, SCNN, RNN, LSTM, ResNet, DenseNet, EfficientNet, etc. The perception network can be obtained by training the neural network under training from multiple sample images labeled with the position information, orientation information and size information of the dynamic obstacles.

[0069] In one implementation, multiple frames of images can be sequentially input into a perception network, and the perception network outputs the perception result information of dynamic obstacles in each frame of images.

[0070] In one implementation, the orientation information of dynamic obstacles in historical images can be optimized. Specifically, the orientation information of dynamic obstacles in historical images can be compared with the standard orientation information corresponding to the historical images. If the error between the orientation information of dynamic obstacles in historical images and the standard orientation information corresponding to the historical images is greater than or equal to a preset error, then the orientation information of the velocity of dynamic obstacles in the current image is used as the orientation information of dynamic obstacles in the historical images. The standard orientation information can be the orientation information of dynamic obstacles detected by LiDAR in the same scene at the same time as the historical images.

[0071] In this embodiment of the disclosure, the powerful learning capability of the perceptual network can be utilized to quickly and accurately obtain the perception result information of dynamic obstacles in each frame of the image. This provides reliable basic data for determining the state information of the dynamic obstacles and predicting the motion trajectory of the dynamic obstacles through the perception result information of the dynamic obstacles in each frame of the image, thereby effectively improving the accuracy of predicting the motion trajectory of the dynamic obstacles.

[0072] In an optional embodiment, the state information of the dynamic obstacle in this disclosure includes: the velocity information, angular velocity information and / or acceleration information of the dynamic obstacle; step 230 may further include: determining the velocity information, angular velocity information and / or acceleration information of the dynamic obstacle based on the position information and orientation information of the dynamic obstacle in each frame image.

[0073] In one implementation, the state information of the dynamic obstacle may include multiple velocity information, multiple angular velocity information, and multiple acceleration information of the dynamic obstacle. The velocity information of the dynamic obstacle is obtained by using the position information of the dynamic obstacle in each frame image and the corresponding time of each frame image; the angular velocity information of the dynamic obstacle is obtained by using the orientation information of the dynamic obstacle in each frame image and the corresponding time of each frame image; and the acceleration information of the dynamic obstacle is determined by using the velocity information of the dynamic obstacle and the corresponding time of each frame image.

[0074] For example, a multi-frame image may include four frames arranged in chronological order, namely image 1, image 2, image 3 and image 4.

[0075] Displacement information is determined by using the positional information of the dynamic obstacle in Image 1 and Image 2. Based on the displacement information and the time interval between Image 1 and Image 2, the first velocity information corresponding to the dynamic obstacle is determined using the velocity formula. Similarly, the second velocity information of the dynamic obstacle is determined based on the positional information of the dynamic obstacle in Image 2 and Image 3. The third velocity information of the dynamic obstacle is determined based on the positional information of the dynamic obstacle in Image 3 and Image 4. The first, second, and third velocity information are all velocity information included in the state quantity information of the dynamic obstacle. The displacement information can be Euclidean distance or Mahalanobis distance, etc.

[0076] Based on the first velocity information and the second velocity information, as well as the time interval between Image 1 and Image 3, the first acceleration information corresponding to the dynamic obstacle is obtained based on the acceleration formula; similarly, the second velocity information and the third velocity information are used to obtain the second acceleration information corresponding to the dynamic obstacle; wherein, the first acceleration information and the second acceleration information are both acceleration information included in the state quantity information of the dynamic obstacle.

[0077] Based on the orientation information of the dynamic obstacles in Images 1 and 2, and the time interval between Images 1 and 2, the first angular velocity information of the dynamic obstacles is determined based on the angular velocity formula. Similarly, the second angular velocity information of the dynamic obstacles can be determined based on the orientation information of the dynamic obstacles in Images 2 and 3, and the third angular velocity information of the dynamic obstacles can be determined based on the orientation information of the dynamic obstacles in Images 3 and 4. The first, second, and third angular velocity information are all angular velocity information included in the state information of the dynamic obstacles.

[0078] In this embodiment of the disclosure, the position and orientation information of the dynamic obstacle in each frame image are used to determine the state information of the dynamic obstacle, thereby improving the accuracy of the motion trajectory of the dynamic obstacle predicted by the state information of the dynamic obstacle.

[0079] In an optional embodiment, such as Figure 3 As shown, step S240 in this embodiment may further include the following steps:

[0080] Step S241: Using the feature extraction subnetwork in the trajectory prediction network, based on map information and the perception results of dynamic obstacles in each frame image, the first image features corresponding to the dynamic obstacles are obtained.

[0081] The trajectory prediction network can include a feature extraction subnetwork and a trajectory prediction subnetwork. The feature extraction subnetwork can be a pre-trained neural network for feature extraction, such as ResNet, DenseNet, EfficientNet, etc. The trajectory prediction subnetwork can be a pre-trained neural network for trajectory prediction, such as CNN, SCNN, RNN, LSTM, etc.

[0082] In one implementation, map information and the perception results of dynamic obstacles in each frame are input into a feature extraction subnetwork, which then outputs the first image features of the dynamic obstacles. The first image features of the dynamic obstacles are features that include the initial feature extraction region of the dynamic obstacles.

[0083] The feature extraction subnetwork can combine semantic segmentation information from the map information to better distinguish dynamic obstacles from their surrounding environment, thereby accurately determining the range of dynamic obstacles. Within this range, the feature extraction subnetwork extracts the features (first image features) of the dynamic obstacles, which can improve the accuracy of feature extraction.

[0084] Step S242: Extract features from the first image features corresponding to the dynamic obstacle to obtain the second image features corresponding to the dynamic obstacle.

[0085] The second image feature of the dynamic obstacle can be obtained by extracting features from the first image feature using a pre-trained neural network for second feature extraction. Alternatively, features can be extracted from the first image feature using methods such as ROI Align. The second image feature of the dynamic obstacle includes the feature extraction region of the dynamic obstacle, and the initial feature extraction region is larger than the initial feature extraction region.

[0086] Step S243: Using the trajectory prediction subnetwork in the trajectory prediction network, the motion trajectory of the dynamic obstacle is obtained based on the state information of the dynamic obstacle and the second image features.

[0087] In one implementation, the state information of the dynamic obstacle and its second image features can be stitched together using a channel overlay method to obtain a stitched result. This stitched result is then input into a trajectory prediction subnetwork, which outputs the motion trajectory of the dynamic obstacle.

[0088] In this embodiment, the powerful learning capabilities of the feature extraction subnetwork and trajectory prediction subnetwork are utilized, and the trajectory prediction of dynamic obstacles is performed by combining the state information of dynamic obstacles, the perception result information of dynamic obstacles in each frame image, and map information, which effectively improves the accuracy of the predicted motion trajectory of dynamic obstacles.

[0089] In an optional embodiment, such as Figure 4 As shown, step S242 in this embodiment further includes the following steps:

[0090] Step S2421: Based on the position and orientation information of the dynamic obstacle in the current image, determine the origin and coordinate direction information of the feature extraction region of the dynamic obstacle.

[0091] The current image is included in the multiple frames.

[0092] The location information of a dynamic obstacle in the current image can be determined as the origin information of the coordinates of the feature extraction region of the dynamic obstacle, or the location information of a point at a preset distance from the location information of the dynamic obstacle in the current image can be used as the origin information of the coordinates of the feature extraction region of the dynamic obstacle, wherein the origin information includes the location information of the origin.

[0093] The orientation information of a dynamic obstacle in the current image can be determined as the coordinate orientation information of the feature extraction region of that dynamic obstacle. Alternatively, the angle that differs from the orientation information of the dynamic obstacle by a preset angle can be determined as the coordinate orientation information of the feature extraction region of that dynamic obstacle. The coordinate orientation information includes the angle between the X-axis and the horizontal direction in the coordinate system of the feature extraction region, wherein the coordinate system of the feature extraction region can be the vehicle coordinate system.

[0094] Step S2422: Based on the origin information, coordinate direction information and preset size information of the feature extraction region of the dynamic obstacle, extract the second image features of the dynamic obstacle from the first image features of the dynamic obstacle.

[0095] Specifically, based on the origin, orientation, and preset size information of the feature extraction region of the dynamic obstacle, the feature extraction region of the dynamic obstacle is determined. Features in the feature extraction region are then extracted from the first image features to obtain the second image features of the dynamic obstacle. The preset size can be set according to actual conditions.

[0096] In one implementation, the origin of the feature extraction region of the dynamic obstacle can be determined as the midpoint of the feature extraction region. The orientation of the feature extraction region is set based on the orientation information of the dynamic obstacle. The orientation of the feature extraction region can include the angle between the horizontal centerline of the feature extraction region and the X-axis of the feature extraction region's coordinate system, or the angle between the vertical centerline of the dynamic obstacle and the Y-axis of the feature extraction region's coordinate system. The horizontal centerline is a straight line passing through the center of the feature extraction region and parallel to at least one side of the feature extraction region.

[0097] For example, such as Figure 5 As shown, taking a vehicle as an example of a moving obstacle area A, the feature extraction region of the dynamic obstacle A is set to a rectangle. The size information of the feature extraction region of the dynamic obstacle A is set to the preset size information. The position of the center point of the feature extraction region is set to the position information shown in the coordinate origin information of the feature extraction region. The angle between the horizontal center line of the feature extraction region and the X-axis of the feature extraction region is set to 0°, that is, the orientation information of the feature extraction region is set to the coordinate orientation information of the feature extraction region. The features in the feature extraction region are extracted to obtain the second image features of the dynamic obstacle A.

[0098] In this embodiment, the origin coordinates and orientation information of the feature extraction region are determined by the position and orientation information of the dynamic obstacle. Based on the origin coordinates, orientation information, and preset size information, the second image features of the dynamic obstacle are obtained. This achieves efficient extraction of the second image features of the dynamic obstacle, providing reliable data support for subsequent prediction of the dynamic obstacle's trajectory using these features, thereby improving the accuracy of predicting the dynamic obstacle's trajectory using its second image features.

[0099] In an optional embodiment, such as Figure 6 As shown, in this embodiment of the present disclosure, step S241 further includes the following steps:

[0100] Step S2411: Convert the coordinates of the perception result information of dynamic obstacles in at least one frame of historical image into the coordinate system where the perception result information of dynamic obstacles in the current image is located.

[0101] The multi-frame images also include at least one historical image.

[0102] For the perception results of dynamic obstacles in each of at least one historical image frame, the coordinate system containing the perception results of the dynamic obstacles in that historical image can be converted to a preset coordinate system. Then, the preset coordinate system containing the perception results of the dynamic obstacles in that historical image can be converted to the coordinate system containing the dynamic obstacles in the current image. This process transforms the coordinates of the perception results of the dynamic obstacles in the historical image into the coordinate system containing the perception results of the dynamic obstacles in the current image. The preset coordinate system can be a world coordinate system (GCS), a three-dimensional coordinate system, etc.

[0103] For example, the vehicle coordinate system containing the perception results of dynamic obstacles in historical images can be converted to the world coordinate system. Then, the world coordinate system containing the perception results of dynamic obstacles in historical images can be converted to the vehicle coordinate system containing the dynamic obstacles in the current image using the odometry (range measurement) conversion method.

[0104] Step S2412: Render the perception result information of dynamic obstacles in the current image and the perception result information of dynamic obstacles in at least one frame of historical image after coordinate transformation to obtain the rendering perception result information of dynamic obstacles in each frame of image.

[0105] The rendering perception results of dynamic obstacles include the location, size, and orientation information of the dynamic obstacle.

[0106] The perception results of dynamic obstacles in the current image and the perception results of dynamic obstacles in all historical images after coordinate transformation can be rendered into a blank image respectively to obtain the rendering perception results of dynamic obstacles in each frame image.

[0107] For example, to obtain the perception result information of dynamic obstacles in the current image, a blank image can be acquired. The background color of the blank image can be white or another single color, and the size of the blank image can be the same as the size of the current image. Based on the position, orientation, and size information of the dynamic obstacles in the current image, the dynamic obstacles in the current image are rendered into the blank image to obtain the rendered perception result information of the dynamic obstacles in the current image. Similarly, the perception result information of dynamic obstacles in each frame of historical images can be rendered into other blank images to obtain the rendered perception result information of dynamic obstacles in each historical image. In this case, the perception result information of dynamic obstacles in each frame of multiple images is rendered into its corresponding blank image.

[0108] Step S2413: The rendering perception result information and map information are stitched together to obtain stitched information.

[0109] In this process, the rendering perception results of dynamic obstacles in each frame of the image can be stitched together with the map information in a channel overlay manner to obtain the stitched information.

[0110] Step S2414: Using the feature extraction sub-network, the first image features of the dynamic obstacle are obtained based on the splicing information.

[0111] The spliced ​​information is input into the feature extraction subnetwork, which then outputs the first image features of the dynamic obstacle.

[0112] In this embodiment, the coordinates of the perception result information of dynamic obstacles in historical images are converted into the coordinate system of the perception result information of dynamic obstacles in the current image. The perception result information of dynamic obstacles in the current image and dynamic obstacles in the coordinate-transformed historical images are rendered separately. This enables efficient and accurate extraction of the first image features of dynamic obstacles, thereby effectively improving the reliability of the second image features of dynamic obstacles obtained through the first image features. Consequently, the accuracy of the motion trajectory of dynamic obstacles predicted using the second image features of dynamic obstacles is effectively improved.

[0113] In one alternative embodiment, Figure 7 A flowchart illustrating an application example of the trajectory prediction method in this disclosure is shown. Figure 7 The trajectory prediction network may include a feature extraction subnetwork and a trajectory prediction subnetwork.

[0114] The perceptual network inputs each frame of a multi-frame image, outputting the perception results of dynamic obstacles in each frame. Based on the perception results of dynamic obstacles in each frame, the state information of the dynamic obstacles is determined. The coordinates of the perception results of dynamic obstacles in each historical image are converted to the coordinate system of the perception results of dynamic obstacles in the current image. The perception results of dynamic obstacles in the current image and the coordinate-transformed perception results of dynamic obstacles in each historical image are rendered to obtain the rendered perception results of dynamic obstacles in each frame. The rendered perception results of dynamic obstacles in each frame are stitched together with map information to obtain stitched information. This stitched information is input into the feature extraction sub-network to obtain the first image features of the dynamic obstacles. Based on the first image features of the dynamic obstacles, the second image features of the dynamic obstacles are obtained. The second image features of the dynamic obstacles and the state information of the dynamic obstacles are input into the trajectory prediction sub-network to obtain the motion trajectory of the dynamic obstacles.

[0115] Any trajectory prediction method provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any trajectory prediction method provided in this disclosure can be executed by a processor, such as by a processor executing any trajectory prediction method mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0116] Exemplary device

[0117] Figure 8 This is a structural block diagram of a trajectory prediction device in one embodiment of this disclosure. Figure 8 As shown, the trajectory prediction device includes: an image acquisition module 310, a perception result acquisition module 320, a state quantity acquisition module 330, and a trajectory prediction module 340.

[0118] The image acquisition module 310 is used to acquire multiple frames of images and map information, wherein the multiple frames of images are multiple frames of images of the surrounding environment of the vehicle acquired by the image acquisition device on the vehicle within a time period;

[0119] The perception result acquisition module 320 is used to obtain the perception result information of dynamic obstacles in each frame of the multi-frame images based on each frame of the multi-frame images;

[0120] The state quantity acquisition module 330 is used to determine the state quantity information of the dynamic obstacle based on the perception result information of the dynamic obstacle in each frame image;

[0121] The trajectory prediction module 340 is used to predict the motion trajectory of the dynamic obstacle by using a trajectory prediction network based on the map information, the perception result information of the dynamic obstacle in each frame image, and the state quantity information of the dynamic obstacle.

[0122] In one embodiment, the perception result information of the dynamic obstacle in this disclosure includes: the position information, orientation information, and size information of the dynamic obstacle, and the perception result acquisition module 320 is further configured to:

[0123] Using a perception network, the position, orientation, and size information of dynamic obstacles in each frame of the multi-frame images are obtained.

[0124] In one embodiment, the state information of the dynamic obstacle in this disclosure includes: velocity information, angular velocity information, and / or acceleration information of the dynamic obstacle, and the state acquisition module 330 is further configured to:

[0125] Based on the position and orientation information of the dynamic obstacle in each frame of the image, the velocity, angular velocity and / or acceleration information of the dynamic obstacle are determined.

[0126] In one embodiment, such as Figure 9 As shown, the trajectory prediction module 340 in this embodiment includes:

[0127] The first feature extraction submodule 341 is used to obtain the first image features corresponding to the dynamic obstacle by utilizing the feature extraction subnetwork in the trajectory prediction network, based on the map information and the perception result information of the dynamic obstacle in each frame image;

[0128] The second feature extraction submodule 342 is used to extract features from the first image features of the dynamic obstacle to obtain the second image features of the dynamic obstacle;

[0129] The trajectory prediction submodule 343 is used to obtain the motion trajectory of the dynamic obstacle based on the state information of the dynamic obstacle and the second image features by utilizing the trajectory prediction subnetwork in the trajectory prediction network.

[0130] In one embodiment, such as Figure 10 As shown, the multi-frame images in this embodiment include the current image, and the second feature extraction submodule 342 includes:

[0131] The determining unit 3421 is used to determine the origin information and coordinate direction information of the feature extraction region of the dynamic obstacle based on the position information and orientation information of the dynamic obstacle in the current image;

[0132] The first extraction unit 3422 is used to extract the second image features of the dynamic obstacle from the first image features of the dynamic obstacle based on the origin information, coordinate direction information and preset size information of the feature extraction region of the dynamic obstacle.

[0133] In one embodiment, such as Figure 11 As shown, the multi-frame images in this embodiment of the present disclosure further include at least one historical image, and the first feature extraction submodule 341 includes:

[0134] The coordinate transformation unit 3411 is used to convert the coordinates of the perception result information of the dynamic obstacle in the at least one frame of historical image into the coordinate system where the perception result information of the dynamic obstacle in the current image is located.

[0135] The rendering unit 3412 is used to render the perception result information of dynamic obstacles in the current image and the perception result information of dynamic obstacles in the at least one frame of historical image after coordinate transformation, so as to obtain the rendering perception result information of dynamic obstacles in each frame of image.

[0136] The stitching unit 3413 is used to stitch together the rendering perception result information and the map information to obtain stitching information;

[0137] The second extraction unit 3414 is used to obtain the first image features of the dynamic obstacle based on the stitching information using the feature extraction sub-network.

[0138] Exemplary electronic devices

[0139] Below, for reference Figure 12 To describe an electronic device according to embodiments of the present disclosure. Figure 12 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0140] like Figure 12 As shown, the electronic device includes one or more processors 410 and memory 420.

[0141] The processor 410 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0142] The memory 420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 410 may execute the program instructions to implement the trajectory prediction methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0143] In one example, the electronic device may also include an input device 430 and an output device 440, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0144] For example, the input device 430 may be the microphone or microphone array described above, used to capture the input signal from the sound source. Furthermore, the input device 430 may also include, for example, a keyboard, a mouse, etc.

[0145] The output device 440 can output various information to the outside, including determined distance information, direction information, etc. The output device 440 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0146] Of course, for simplicity, only some of the components in the electronic device relevant to this disclosure are shown in the figures, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0147] Exemplary computer program products and computer-readable storage media

[0148] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the trajectory prediction methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0149] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0150] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the trajectory prediction methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0151] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0153] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0154] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0155] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0156] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0157] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0158] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A trajectory prediction method, comprising: Acquire multiple frames of images and map information, wherein the multiple frames of images are multiple frames of images of the surrounding environment of the vehicle acquired by the image acquisition device on the vehicle within a time period; Based on each frame of the multi-frame images, the perception result information of dynamic obstacles in each frame of the images is obtained; Based on the perception results of dynamic obstacles in each frame of the image, determine the state information of the dynamic obstacles; Using a trajectory prediction network, based on the map information, the perception results of dynamic obstacles in each frame of the image, and the state information of the dynamic obstacles, the motion trajectory of the dynamic obstacle is predicted, including: using the feature extraction subnetwork in the trajectory prediction network, based on the map information and the perception results of the dynamic obstacle in each frame of the image, to obtain a first image feature corresponding to the dynamic obstacle; performing feature extraction on the first image feature corresponding to the dynamic obstacle to obtain a second image feature corresponding to the dynamic obstacle; and using the trajectory prediction subnetwork in the trajectory prediction network, based on the state information of the dynamic obstacle and the second image feature, to obtain the motion trajectory of the dynamic obstacle.

2. The method according to claim 1, wherein, The perception result information of the dynamic obstacle includes: the position information, orientation information, and size information of the dynamic obstacle. The step of obtaining the perception result information of the dynamic obstacle in each frame of the multi-frame image includes: Using a perception network, the position, orientation, and size information of the dynamic obstacle in each of the multiple frames of images are obtained.

3. The method according to claim 2, wherein, The state information of the dynamic obstacle includes: velocity information, angular velocity information, and / or acceleration information of the dynamic obstacle. Determining the state information of the dynamic obstacle based on the perception result information of the dynamic obstacle in each frame image includes: Based on the position and orientation information of the dynamic obstacle in each frame of the image, determine the velocity, angular velocity, and / or acceleration information of the dynamic obstacle.

4. The method according to claim 1, wherein, The multi-frame images include the current image. The step of extracting features from the first image features of the dynamic obstacle to obtain the second image features of the dynamic obstacle includes: Based on the position and orientation information of the dynamic obstacle in the current image, determine the origin and coordinate direction information of the feature extraction region of the dynamic obstacle; Based on the origin information of the feature extraction region of the dynamic obstacle, the coordinate direction information, and the preset size information, the second image feature of the dynamic obstacle is extracted from the first image feature of the dynamic obstacle.

5. The method according to claim 4, wherein, The multi-frame image also includes at least one historical image. The first image feature of the dynamic obstacle is obtained using the feature extraction subnetwork in the trajectory prediction network, based on the map information and the perception result information of the dynamic obstacle in each frame image. This includes: The coordinates of the perception result information of the dynamic obstacle in the at least one frame of historical image are converted into the coordinate system where the perception result information of the dynamic obstacle in the current image is located; The perception result information of the dynamic obstacle in the current image and the perception result information of the dynamic obstacle in the at least one frame of the historical image after coordinate transformation are rendered to obtain the rendering perception result information of the dynamic obstacle in each frame of the image. The rendering perception result information and the map information are concatenated to obtain the concatenated information; Using the feature extraction subnetwork, the first image features of the dynamic obstacle are obtained based on the stitching information.

6. A trajectory prediction device, comprising: The image acquisition module acquires multiple frames of images and map information, wherein the multiple frames of images are multiple frames of images of the surrounding environment of the vehicle acquired by the image acquisition device on the vehicle within a time period; The perception result acquisition module is used to obtain the perception result information of dynamic obstacles in each frame of the multi-frame images based on each frame of the multi-frame images; The state quantity acquisition module is used to determine the state quantity information of the dynamic obstacle based on the perception result information of the dynamic obstacle in each frame image; The trajectory prediction module is used to predict the motion trajectory of the dynamic obstacle using a trajectory prediction network, based on the map information, the perception result information of the dynamic obstacle in each frame image, and the state quantity information of the dynamic obstacle. The trajectory prediction module includes: The first feature extraction submodule is used to obtain the first image features corresponding to the dynamic obstacle by utilizing the feature extraction subnetwork in the trajectory prediction network, based on the map information and the perception result information of the dynamic obstacle in each frame image; The second feature extraction submodule is used to extract features from the first image features of the dynamic obstacle to obtain the second image features of the dynamic obstacle; The trajectory prediction submodule is used to obtain the motion trajectory of the dynamic obstacle based on the state information of the dynamic obstacle and the second image features by utilizing the trajectory prediction subnetwork in the trajectory prediction network.

7. The apparatus according to claim 6, wherein, The perception result information of the dynamic obstacle includes: the position information, orientation information, and size information of the dynamic obstacle. The perception result acquisition module is also used for: Using a perception network, the position, orientation, and size information of the dynamic obstacle in each frame of the multi-frame images are obtained.

8. A computer-readable storage medium storing a computer program for performing the trajectory prediction method according to any one of claims 1-5.

9. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the trajectory prediction method according to any one of claims 1-5.

Images