Obstacle trajectory prediction method and device

Abstract

This specification discloses a method and device for predicting obstacle trajectories. Based on the determined environmental feature data, the unmanned driving device can predict the image data of the obstacle collected after a set time, and based on the position of the obstacle image in the predicted image, the future position of the obstacle can be predicted. As the number of predicted images obtained continues to increase, the movement trajectory of the obstacle in the future can be finally obtained based on the position of the obstacle image in each predicted image, so that the unmanned driving device can avoid the obstacle in time based on the predicted movement trajectory of the obstacle, further ensuring the travel safety of the unmanned driving device.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving equipment, and in particular to a method and device for predicting obstacle trajectories. Background Art

[0002] At present, unmanned vehicles, robots and other unmanned driving equipment have been widely used in many fields. With the continuous improvement of science and technology, unmanned driving equipment will be further developed in various fields, thus bringing more convenience to people's real life.

[0003] In order to ensure the smooth driving of unmanned driving equipment, it is necessary to predict the driving conditions of various obstacles in the surrounding environment of the unmanned driving equipment, so that the unmanned driving equipment can avoid obstacles in time according to the predicted driving conditions of various obstacles during driving.

[0004] Taking unmanned vehicles as an example, in the existing technology, unmanned vehicles can collect sensor data through sensors such as lidar and infrared, and predict the lanes where obstacles around the unmanned vehicle will be located in the future based on the collected sensor data.

[0005] However, although the unmanned vehicle can predict the lanes that the surrounding obstacles will be in in the future through existing technology, it cannot predict the movement trajectories of the surrounding obstacles in the future. This results in the unmanned vehicle still colliding with the surrounding obstacles after predicting the lanes that the surrounding obstacles will be in next. That is, the smooth driving of the unmanned vehicle cannot be guaranteed by existing technology.

[0006] Therefore, how to predict the movement trajectory of obstacles around unmanned driving equipment is an urgent problem to be solved. Summary of the invention

[0007] This specification provides a method and device for obstacle trajectory prediction to partially solve the above-mentioned problems existing in the prior art.

[0008] This manual adopts the following technical solutions:

[0009] This specification provides a method for obstacle trajectory prediction, including:

[0010] Acquire sensor data of the obstacle collected by the unmanned driving device, and determine the position information of the unmanned driving device when collecting the sensor data, wherein the sensor data includes: image data of the obstacle;

[0011] Inputting the sensing data and the location information into a feature analysis model to determine environmental feature data corresponding to the obstacle;

[0012] Inputting the environmental feature data into a prediction model to predict image data of obstacles collected after a set time as a prediction image;

[0013] The movement trajectory of the obstacle is predicted according to the predicted image.

[0014] Optionally, obtaining sensor data of obstacles collected by unmanned driving equipment includes:

[0015] Acquiring sensor data of the obstacle continuously collected by the unmanned driving device;

[0016] Inputting the sensing data and the location information into a feature analysis model to determine the environmental feature data corresponding to the obstacle specifically includes:

[0017] For each acquired sensor data, the sensor data and the location information based on which the unmanned driving device collects the sensor data are input into the feature analysis model to determine the environmental feature data corresponding to the sensor data.

[0018] Optionally, the environmental feature data is input into a prediction model to predict image data of obstacles collected after a set time as a predicted image, specifically including:

[0019] For each sensing data, the environmental feature data corresponding to the sensing data is input into the prediction model to predict the image data of the obstacle collected by the unmanned driving device after the set time as the predicted image corresponding to the sensing data.

[0020] Optionally, predicting the movement trajectory of the obstacle according to the predicted image specifically includes:

[0021] The movement trajectory of the obstacle is predicted according to the image position of the image of the obstacle in the predicted image corresponding to each sensing data.

[0022] Optionally, predicting the movement trajectory of the obstacle according to the image position of the image of the obstacle in the predicted image corresponding to each sensing data specifically includes:

[0023] The movement trajectory of the obstacle is predicted based on the image position of the obstacle image in the predicted image corresponding to each sensor data and the estimated actual position based on which the unmanned driving device collects the predicted image corresponding to each sensor data.

[0024] Optionally, predicting the movement trajectory of the obstacle according to the image position of the image of the obstacle in the predicted image corresponding to each sensor data and the estimated actual position based on which the unmanned driving device collects the predicted image corresponding to each sensor data specifically includes:

[0025] For each predicted image corresponding to the sensor data, according to the image position of the image of the obstacle in the predicted image, determine the relative position of the obstacle with respect to the unmanned driving device when the unmanned driving device collects the predicted image as the relative position of the obstacle corresponding to the predicted image;

[0026] According to the relative position of the obstacle corresponding to the predicted image and the estimated actual position based on which the unmanned driving device collects the predicted image, predict the actual position of the obstacle when the unmanned driving device collects the predicted image as the actual position of the obstacle corresponding to the predicted image;

[0027] According to the actual position of the obstacle corresponding to each predicted image, the movement trajectory of the obstacle is predicted.

[0028] Optionally, estimating the actual position based on which the predicted image corresponding to each sensor data collected by the unmanned driving device is based specifically includes:

[0029] For each piece of sensor data, the environmental feature data and the predicted image corresponding to the sensor data are input into the position estimation model to estimate the actual position based on which the unmanned driving device collects the predicted image.

[0030] Optionally, the feature analysis model is trained in the following manner:

[0031] Acquire first training data collected during several driving processes in history, and first training position information corresponding to the first training data, wherein the first training data includes first training images collected during the driving process;

[0032] For each first training data, the first training data and the first training position information corresponding to the first training data are input into the feature analysis model to obtain the environmental feature data corresponding to the first training data;

[0033] Inputting the environmental feature data corresponding to the first training data into the adversarial model corresponding to the feature analysis model to obtain a second training image;

[0034] Taking maximizing the similarity between the first training image and the second training image in the first training data as an optimization goal, the parameters in the feature analysis model are adjusted until a preset training goal is achieved.

[0035] Optionally, the prediction model is trained in the following way:

[0036] Acquire first training data collected during several driving processes in history and first training position information corresponding to the first training data, wherein the first training data includes first training images collected during the driving process;

[0037] For each first training data, the first training data and the first training position information corresponding to the first training data are input into the feature analysis model to obtain the environmental feature data corresponding to the first training data;

[0038] Inputting the environmental feature data corresponding to the first training data into the prediction model to obtain a prediction image;

[0039] Determine the image data collected after a set time from the time when the first training data is collected as the target training image;

[0040] Taking maximizing the similarity between the predicted image and the target training image as the optimization goal, the parameters in the prediction model are adjusted until a preset training goal is achieved.

[0041] Optionally, the position estimation model is trained in the following way:

[0042] Acquire first training data collected during several driving processes in history and first training position information corresponding to the first training data, wherein the first training data includes first training images collected during the driving process;

[0043] For each first training data, the first training data and the first training position information corresponding to the first training data are input into the feature analysis model to obtain the environmental feature data corresponding to the first training data;

[0044] Inputting the first training data and the environmental feature data corresponding to the first training data into the position estimation model to obtain estimated position information;

[0045] Taking minimizing the position deviation between the estimated position information and the first training position information corresponding to the first training data as an optimization goal, the parameters in the position estimation model are adjusted until a preset training goal is achieved.

[0046] This specification provides a device for obstacle trajectory prediction, including:

[0047] an acquisition module, used to acquire sensor data of the obstacle collected by the device and determine the position information of the device when collecting the sensor data, wherein the sensor data includes: image data of the obstacle;

[0048] A feature determination module, used for inputting the sensing data and the position information into a feature analysis model to determine environmental feature data corresponding to the obstacle;

[0049] An image determination module, used to input the environmental feature data into a prediction model to predict image data of obstacles collected after a set time as a prediction image;

[0050] The trajectory prediction module is used to predict the movement trajectory of the obstacle according to the predicted image.

[0051] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned obstacle trajectory prediction method is implemented.

[0052] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned obstacle trajectory prediction method when executing the program.

[0053] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0054] Acquire sensor data of the obstacle collected by the unmanned driving device, and determine the position information of the unmanned driving device when collecting the sensor data, the sensor data including the image data of the obstacle collected by the unmanned driving device, input the sensor data and its position information into the feature analysis model to determine the environmental feature data corresponding to the obstacle, input the obtained environmental feature data into the prediction model to predict the image data of the obstacle collected after a set time as a prediction image, and then predict the movement trajectory of the obstacle based on the prediction image.

[0055] It can be seen from the above method that the unmanned driving device can predict the image data of the obstacle collected after a set time based on the determined environmental feature data, and can predict the future position of the obstacle based on the position of the obstacle image in the predicted image. As the number of predicted images obtained continues to increase, the unmanned driving device can finally obtain the movement trajectory of the obstacle in the future based on the position of the obstacle image in each predicted image, so that the unmanned driving device can avoid the obstacle in time based on the predicted movement trajectory of the obstacle, further ensuring the travel safety of the unmanned driving device. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0057] Figure 1 A schematic diagram of a process for obstacle trajectory prediction provided in this specification;

[0058] Figure 2 A schematic diagram of the sensor data collected by the unmanned driving device during driving provided in this manual;

[0059] Figure 3 A schematic diagram of training a feature analysis model provided in this specification;

[0060] Figure 4 A schematic diagram of obtaining a predicted image through collected sensor data provided in this specification;

[0061] Figure 5 A schematic diagram of a device for predicting obstacle trajectories provided in this specification;

[0062] Figure 6 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this specification clearer, the technical solutions in this specification will be clearly and completely described in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0064] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0065] Figure 1 A flowchart of obstacle trajectory prediction provided in this specification specifically includes the following steps:

[0066] S101: Acquire sensor data of an obstacle collected by an unmanned driving device, and determine the position information of the unmanned driving device when collecting the sensor data, wherein the sensor data includes: image data of the obstacle.

[0067] During the driving process, the unmanned driving equipment can collect sensor data of surrounding obstacles so that the unmanned driving equipment itself can obtain these sensor data. Among them, the unmanned driving equipment mentioned here refers to unmanned vehicles, robots, automatic delivery equipment and other equipment that can realize automatic driving. Based on this, the obstacle trajectory prediction method provided in this specification can be specifically applied to the field of delivery using unmanned driving equipment, such as the business scenarios of using unmanned driving equipment for express delivery, logistics, take-out and other delivery. In order to ensure that the unmanned driving equipment can move smoothly in these business scenarios, it is necessary to predict the movement trajectory of the surrounding obstacles in the unmanned driving equipment.

[0068] Obstacles around unmanned driving equipment refer to objects such as motor vehicles, pedestrians, guardrails, buildings, etc. that can hinder the movement of unmanned driving equipment. The sensor data of obstacles does not refer to the data used to measure the state of the obstacles themselves, but refers to the image data of the surrounding obstacles collected by the image sensors (such as cameras, depth cameras, etc.) installed on the unmanned driving equipment, and the distance between the surrounding obstacles collected by the distance measuring sensors (such as laser radar, ultrasonic sensors, infrared distance measuring sensors) installed on the unmanned driving equipment. Figure 2 shown.

[0069] Figure 2 This is a schematic diagram of the sensor data collected by the unmanned driving device during driving.

[0070] Assume that the unmanned driving device is an unmanned vehicle. Figure 2 In the figure, there are several motor vehicles driving around the unmanned vehicle A, and there is a pedestrian crossing the road at the crosswalk in front of the unmanned vehicle A. Since the motor vehicles around the unmanned vehicle A and the pedestrians in front of the unmanned vehicle A may hinder the driving of the unmanned vehicle A, they can all be regarded as obstacles to the unmanned vehicle A. While the unmanned vehicle A is driving, the image sensor and the distance sensor set in front of the vehicle can collect the sensor data in the cone range in front of the unmanned vehicle A, that is, collect the image data of the motor vehicles and pedestrians in the cone range in front, and collect the distance data between the motor vehicles and pedestrians in the cone range in front, etc.

[0071] It should be noted that in practical applications, unmanned driving equipment can not only collect Figure 2The sensor data of obstacles within the front cone range shown in the figure can also be collected for obstacles on both sides or the rear. This mainly depends on the position of each sensor on the unmanned driving device. In other words, sensors can be arranged on all sides of the unmanned driving device, and accordingly, the sensor data of obstacles on all sides can be collected during the driving process of the unmanned driving device. This not only enables the unmanned driving device to predict the movement trajectory of obstacles in front of the vehicle based on the collected sensor data, but also can predict the movement trajectory of obstacles on the left and right sides and the rear of the unmanned driving device based on the collected sensor data, thereby further ensuring the driving safety of the unmanned driving device during driving.

[0072] In this specification, the unmanned driving device can continuously collect sensor data of surrounding obstacles during driving. Therefore, the sensor data of obstacles collected by the unmanned driving device is actually temporally continuous sensor data. The unmanned driving device can subsequently predict the movement trajectory of the obstacle in the future based on the sensor data of the obstacle obtained through continuous collection.

[0073] While collecting sensor data of surrounding obstacles, the unmanned driving device also needs to determine its own location information when collecting sensor data. Specifically, the unmanned driving device can locate its own location when collecting sensor data through the Global Positioning System (GPS), thereby obtaining corresponding location information. The determined location information can be specifically expressed in the form of longitude and latitude.

[0074] Since the unmanned driving device continuously collects sensor data of obstacles during driving, each time the unmanned driving device collects a sensor data, it needs to determine the position information of itself when collecting the sensor data. That is, each sensor data corresponds to a position information.

[0075] S102: Input the sensing data and the location information into a feature analysis model to determine environmental feature data corresponding to the obstacle.

[0076] After the unmanned driving device collects the sensor data, the collected sensor data and the location information when the sensor data is collected can be input into a pre-trained feature analysis model to obtain environmental feature data corresponding to the surrounding obstacles.

[0077] Since the unmanned driving device continuously collects sensor data of obstacles during driving, for each sensor data collected, the sensor data and the position information of the unmanned driving device when collecting the sensor data are input into the feature analysis model, and the environmental feature data corresponding to the surrounding obstacles when the unmanned driving device collects the sensor data will be obtained. In other words, one sensor data corresponds to one environmental feature data.

[0078] Furthermore, for each sensor data, the unmanned driving device can extract feature data of feature dimensions required by the feature analysis model from the sensor data and the position information when the sensor data is collected, and then input the extracted feature data into the feature analysis model, thereby obtaining the environmental feature data corresponding to the surrounding obstacles when the unmanned driving device collects the sensor data. The environmental feature data mentioned here is mainly used to characterize the image features of the surrounding obstacles and the surrounding environment in the image when the unmanned driving device collects the sensor data. The feature analysis model can be specifically constructed using a long short-term memory network (Long Short-Term Memory, LSTM).

[0079] From the above, it can be seen that although the environmental feature data obtained by the feature analysis model is used to characterize the image features of the surrounding obstacles on the image, the data input into the feature analysis model is not only the image data of the surrounding obstacles. In other words, in addition to inputting the image data of the surrounding obstacles into the feature analysis model, it is also necessary to input other sensor data of the surrounding obstacles collected and the location information based on which the unmanned driving equipment collects the sensor data into the feature analysis model, so as to further strengthen and accurately characterize the image features of the surrounding obstacles on the image through the other sensor data and location information.

[0080] In order to obtain accurate environmental feature data, in this specification, it is necessary to pre-train the feature analysis model. Among them, the training of the feature analysis model can be implemented in an adversarial network manner. Specifically, the first training data collected by a designated device (the driving device mentioned here can be an unmanned driving device or a manually driven motor vehicle) during several historical driving processes, and the location information based on which the first training data was collected can be obtained as the first training location information. The first training data mentioned here can be data collected by multiple designated devices during historical driving processes. Each collected first training data corresponds to a first training location information.

[0081] In the process of training the feature analysis model, for each first training data, the first training data and the first training location information corresponding to the first training data can be input into the feature analysis model to obtain the environmental feature data corresponding to the first training data. Then, the environmental feature data corresponding to the first training data is input into the adversarial model corresponding to the feature analysis model to obtain the image data corresponding to the environmental feature data as the second training image. Furthermore, the similarity between the first training image in the first training data and the second training image can be maximized as the optimization goal, and the parameters in the feature analysis model can be adjusted until the preset training goal is reached, such as Figure 3 shown.

[0082] Figure 3 A schematic diagram of training a feature analysis model provided for this specification.

[0083] Figure 3 The first training image shown in is a frame of image captured by an image sensor disposed on the left side of the designated device during the historical driving process of the designated device. In the process of training the feature analysis model, the first training data including the first training image and the position information corresponding to the first training data can be input into the feature analysis model to obtain environmental feature data, and the environmental feature data can be input into the adversarial model corresponding to the feature analysis model to obtain the second training image.

[0084] from Figure 3 It can be seen that when training 80,000 times, the adversarial model cannot output a second training image close to the first training image based on the environmental feature data output by the feature analysis model. When the number of training times reaches 160,000 times, the adversarial model can obtain a second training image close to the first training image to a certain extent based on the environmental feature data output by the feature analysis model. However, there are still large differences between the image and the first training image in many places. With the continuous increase in the number of training times, after reaching 480,000 training times, the adversarial model can basically obtain a second training image very close to the first training image based on the environmental feature data output by the feature analysis model, thereby achieving the preset training goal.

[0085] From the above examples, it can be seen that by continuously increasing the number of training times of the adversarial network, the feature analysis model can "recognize" the obstacle image contained in the first training image more comprehensively. Correspondingly, the environmental feature data output by the feature analysis model can effectively characterize the image features of each obstacle in the first training image. Furthermore, since the environmental feature data output by the feature analysis model is becoming more and more accurate, the second training image output by the adversarial model in the adversarial network based on the environmental feature data will also become closer to the first training image. In other words, the adversarial model can restore the first training image to a certain extent based on the environmental feature data.

[0086] In this specification, there are many ways to determine the similarity between the first training image and the second training image, for example, Euclidean distance, cosine angle, Jaccard distance, etc. Of course, the evidence lower bound (ELBO) between the first training image and the second training image can also be determined first, and the similarity between the first training image and the second training image is determined based on the obtained ELBO.

[0087] Since the above-mentioned methods for determining similarity and ELBO are relatively mature technologies, the specific implementation process will not be described in detail in this specification. However, it should be noted that since there are many ways to determine the similarity between images, the similarity determination method used is not limited in this specification. The execution subject for training the feature analysis model can be a terminal device such as a computer, or a server, and this specification does not make specific restrictions on this.

[0088] S103: Inputting the environmental feature data into a prediction model to predict image data of obstacles collected after a set period of time as a prediction image.

[0089] In practical applications, after the unmanned driving device obtains environmental feature data through the above-mentioned trained feature analysis model, the environmental feature data can be input into the pre-trained prediction model to predict the image data of the obstacle collected after a set time as a predicted image. Specifically, since the unmanned driving device continuously collects multiple sensor data of surrounding obstacles, the unmanned driving device can obtain the environmental feature data corresponding to each sensor data through the above-mentioned feature analysis model. After each environmental feature data is input into the prediction model respectively, the predicted image corresponding to each sensor data will be obtained. The predicted image mentioned here refers to the image data of surrounding obstacles that the unmanned driving device may collect in the future, as predicted by the prediction model based on the environmental feature data, such as Figure 4 shown.

[0090] Figure 4 A schematic diagram of obtaining a predicted image through collected sensor data provided in this specification.

[0091] exist Figure 4 In the figure, Figure A is a frame of image of surrounding obstacles (i.e., the motor vehicle in Figure A) collected by the unmanned driving device at a moment. The unmanned driving device inputs the sensor data of the surrounding obstacles collected when collecting this frame of image (including this frame of image) and the position information when collecting the sensor data into the feature analysis model to obtain the corresponding environmental feature data. After the unmanned driving device inputs the environmental feature data into the prediction model, it will obtain the predicted image shown in Figure B. It can be seen from Figure B that the position of the surrounding obstacles relative to the unmanned driving device has changed, and this change is actually predicted by the unmanned driving device through the prediction model.

[0092] As can be seen from the above examples, the unmanned driving device can obtain the corresponding predicted image by inputting each environmental feature data into the prediction model. Since the predicted image can reflect the position change of the surrounding obstacles relative to the unmanned driving device in the future, the unmanned driving device can subsequently predict the movement trajectory of the surrounding obstacles based on the obtained predicted images.

[0093] Before using the prediction model to obtain the prediction image, the prediction model needs to be trained in advance. Specifically, in the process of training the prediction model, the first training data collected during several driving processes in the history of each designated device and the location information based on which the first training data was collected can be first obtained as the first training location information. The first training data mentioned here includes the first training image collected during the driving process.

[0094] After acquiring each first training data, for each first training data, the first training data and the first training location information corresponding to the first training data can be input into the feature analysis model to obtain the environmental feature data corresponding to the first training data, and the environmental feature data corresponding to the first training data can be input into the prediction model to be trained to obtain the predicted image. Then, since the acquired first training data are all historical data, it is possible to further determine the image data collected after a set time from the time when the first training data is collected as the target training image, and then adjust the parameters in the prediction model with maximizing the similarity between the predicted image and the target training image as the optimization goal until the prediction training goal is achieved.

[0095] For example, assuming that the first training data obtained is collected by a designated device at the historical moment t0, the environmental feature data corresponding to the first training data can be determined through the feature analysis model, and the environmental feature data can be input into the prediction model to be trained to obtain a predicted image. At the same time, the image data collected by the designated device at the historical moment t0+Δt can be determined from the many collected image data as the target training image, where Δt is the set time mentioned above. Finally, the prediction model can be trained with the prediction image gradually approaching the target training image, which is the actual collected image data, as the optimization goal.

[0096] It should be noted that the methods used to determine the similarity between the predicted image and the target training image are all conventional methods, and this specification does not make specific limitations on this. The feature analysis model used in training the prediction model can be a trained model. The prediction model can be specifically constructed using LSTM. The execution subject for training the prediction model can be a terminal device such as a computer, or a server, and this specification does not make specific limitations on this.

[0097] S104: Predicting a movement trajectory of the obstacle according to the predicted image.

[0098] Through the above prediction model, the prediction image corresponding to each environmental feature data (that is, the prediction image corresponding to each sensor data) can be obtained. The unmanned driving device can predict the movement trajectory of the obstacle according to the image position of the obstacle in each prediction image. Specifically, during the actual driving process of the unmanned driving device, there are often multiple obstacles around it. Correspondingly, there may also be multiple obstacles in the prediction image obtained by the unmanned driving device through the prediction model. Therefore, when obtaining each prediction image, the unmanned driving device needs to first determine which obstacle images are included in these prediction images. Among them, the method used to determine the image of the obstacle from the prediction image is a conventional target recognition method, which is not specifically limited here.

[0099] For each identified obstacle, since the obstacle may move, the image of the obstacle may not be present in every predicted image. Therefore, the unmanned driving device needs to select the predicted images containing the image of the obstacle, and determine the image position of the image of the obstacle in the selected predicted images.

[0100] Based on the determined image position of the obstacle in the predicted image, only the relative position between the unmanned driving device and the obstacle can be determined. Therefore, in this specification, the unmanned driving device must also estimate the actual position of the unmanned driving device when the predicted image is collected based on each predicted image.

[0101] Specifically, the unmanned driving device can input the environmental feature data and the predicted image corresponding to each sensor data collected into a pre-trained position estimation model to estimate the actual position based on which the unmanned driving device collected the predicted image. The position estimation model mentioned here can be constructed using LSTM.

[0102] In the process of training the above-mentioned position estimation model, the first training data collected during several driving processes in the history of each designated device and the position information based on which the first training data was collected can also be obtained as the first training position information corresponding to the first training data. For each first training data, the first training data and the first training position information corresponding to the first training data can be input into the feature analysis model to obtain the environmental feature data corresponding to the first training data. The feature analysis model mentioned here can be regarded as a trained feature analysis model.

[0103] Furthermore, the first training data and the environmental feature data corresponding to the first training data can be input into a position estimation model that needs to be trained to obtain estimated position information. The estimated position information is the specific position of the designated device estimated by the position estimation model when the first training data is collected. Then, the position deviation between the estimated position information and the first training position information corresponding to the first training data can be minimized as an optimization goal, and the parameters in the position estimation model can be adjusted until the preset training goal is reached. In other words, the smaller the deviation between the position corresponding to the estimated position information output by the position estimation model and the actual position corresponding to the first training position information, the better.

[0104] In this specification, for each predicted image, the unmanned driving device determines the relative position of the obstacle relative to the unmanned driving device when the unmanned driving device collects the predicted image based on the image position of the obstacle in the predicted image, as the relative position of the obstacle corresponding to the predicted image. After estimating the actual position based on which the unmanned driving device collects each predicted image, the unmanned driving device can further predict the actual positions of the obstacles in the future based on the determined relative positions of the obstacles, and then predict the movement trajectory of the obstacles based on these predicted actual positions.

[0105] For example, assuming that the unmanned driving device continuously collects sensor data at five moments in the time period from t1 to t5, the predicted images A1 to A5 corresponding to the five sensor data can be obtained through the above-mentioned feature analysis model and prediction model. These five predicted images refer to the image data of surrounding obstacles that may be collected at the next five moments predicted by the unmanned driving device based on the sensor data at the five moments from t1 to t5. The corresponding relationship between these five predicted images and the five moments from t1 to t5 is shown in the following table.

[0106]

[0107]

[0108] Table 1

[0109] Among them, the time length of Δt can be greater than the time length of t5-t1. That is to say, for any predicted image, the predicted image refers to the image data collected by the unmanned driving device after the five moments t1 to t5. After obtaining the above five predicted images, the actual positions p1 to p5 based on which the unmanned driving device collects these five predicted images can be estimated respectively. At the same time, for an obstacle contained in these five predicted images, the unmanned driving device can determine the relative position p′1 to p′5 of the obstacle relative to the unmanned driving device when the unmanned driving device collects these five predicted images based on the image position of the obstacle in these five predicted images. Furthermore, the unmanned driving device can determine the actual positions of the five obstacles in the future based on the relative positions of the five obstacles and the predicted actual positions based on which the above five predicted images are collected, as shown in the following table.

[0110]

[0111] Table 2

[0112] After obtaining the actual positions of the five obstacles, the movement trajectory of the obstacles can be obtained according to the preset electronic map. In this way, the unmanned driving device can predict the movement trajectory of each obstacle in the future, and then avoid obstacles based on these movement trajectories, ensuring the driving safety of the unmanned driving device during driving.

[0113] It should be noted that determining the relative position of an obstacle to the unmanned driving device by the image position of the obstacle in the predicted image is a conventional technique, which will not be described in detail here. Correspondingly, determining the actual position of the obstacle by the relative position of the obstacle and the actual position of the unmanned driving device is also a conventional technique, which will not be described in detail here.

[0114] It can be seen from the above method that the unmanned driving device can predict the image data of the obstacle collected after a set time based on the determined environmental feature data, and based on the position of the obstacle image in the predicted image, the future position of the obstacle can be predicted. As the number of predicted images obtained continues to increase, the movement trajectory of the obstacle in the future can be finally obtained based on the position of the obstacle image in each predicted image, so that the unmanned driving device can avoid the obstacle in time based on the predicted movement trajectory of the obstacle, further ensuring the travel safety of the unmanned driving device.

[0115] In this specification, obstacles are not limited to dynamic obstacles such as pedestrians and motor vehicles, but also include static obstacles such as telephone poles and roadblocks. Accordingly, when the obstacle is a dynamic obstacle, the obstacle's motion trajectory predicted by the obstacle trajectory prediction method provided in this specification should be a linear trajectory, and when the obstacle is a static obstacle, the obstacle's motion trajectory predicted by the unmanned driving device should be a point. Therefore, in addition to being able to predict the motion trajectories of the surrounding obstacles through the above method, the unmanned driving device can also determine the category of the obstacle through the specific form of the motion trajectory of each obstacle, that is, which are static obstacles and which are dynamic obstacles, and then make more accurate behavioral decisions by combining the predicted obstacle's motion trajectory and the obstacle's category.

[0116] It should also be noted that the execution subject of the obstacle trajectory prediction method provided in this specification can be not only the unmanned driving device mentioned above, but also a server. That is, the unmanned driving device uploads the continuously collected sensor data and the location information based on which the sensor data is collected to the server. The server predicts the movement trajectory of obstacles around the unmanned driving device based on the data uploaded by the unmanned driving device, and returns the predicted movement trajectory to the unmanned driving device. In this case, the above-mentioned feature analysis model, prediction model and position estimation model can be set in the server.

[0117] The above is a method for predicting obstacle trajectories provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding device for predicting obstacle trajectories, such as Figure 5 shown.

[0118] Figure 5 A schematic diagram of a device for predicting obstacle trajectories provided in this specification specifically includes:

[0119] The acquisition module 501 is used to acquire the sensor data of the obstacle collected by the device and determine the position information of the device when collecting the sensor data, wherein the sensor data includes: image data of the obstacle;

[0120] A feature determination module 502, used to input the sensing data and the location information into a feature analysis model to determine environmental feature data corresponding to the obstacle;

[0121] The image determination module 503 is used to input the environmental feature data into the prediction model to predict the image data of the obstacle collected after a set time as a prediction image;

[0122] The trajectory prediction module 504 is used to predict the movement trajectory of the obstacle according to the predicted image.

[0123] Optionally, the acquisition module 501 is specifically used to acquire the sensor data of the obstacle continuously collected by the device;

[0124] The feature determination module 502 is specifically used to input each acquired sensor data and the location information based on which the device collects the sensor data into the feature analysis model to determine the environmental feature data corresponding to the sensor data.

[0125] Optionally, the image determination module 503 is specifically used to, for each sensor data, input the environmental feature data corresponding to the sensor data into the prediction model to predict the image data of the obstacle collected by the device after the set time as the predicted image corresponding to the sensor data.

[0126] Optionally, the trajectory prediction module 504 is specifically configured to predict the movement trajectory of the obstacle according to an image position of the image of the obstacle in a predicted image corresponding to each sensing data.

[0127] Optionally, the trajectory prediction module 504 is specifically used to predict the movement trajectory of the obstacle based on the image position of the image of the obstacle in the predicted image corresponding to each sensor data, and the estimated actual position based on which the device collects the predicted image corresponding to each sensor data.

[0128] Optionally, the trajectory prediction module 504 is specifically used to determine, for each predicted image corresponding to the sensor data, the relative position of the obstacle with respect to the device when the device collects the predicted image according to the image position of the image of the obstacle in the predicted image, as the relative position of the obstacle corresponding to the predicted image; predict the actual position of the obstacle when the device collects the predicted image according to the relative position of the obstacle corresponding to the predicted image and the estimated actual position based on which the device collects the predicted image, as the actual position of the obstacle corresponding to the predicted image; and predict the movement trajectory of the obstacle according to the actual position of the obstacle corresponding to each predicted image.

[0129] Optionally, the trajectory prediction module 504 is specifically used to, for each sensor data, input the environmental feature data corresponding to the sensor data and the predicted image into a position estimation model to estimate the actual position based on which the device collects the predicted image.

[0130] Optionally, the device further comprises:

[0131] The training module 505 is used to obtain first training data collected during several driving processes in history, and first training position information corresponding to the first training data, wherein the first training data includes a first training image collected during the driving process; for each first training data, the first training data and the first training position information corresponding to the first training data are input into the feature analysis model to obtain environmental feature data corresponding to the first training data; the environmental feature data corresponding to the first training data is input into an adversarial model corresponding to the feature analysis model to obtain a second training image; and the parameters in the feature analysis model are adjusted with the optimization goal of maximizing the similarity between the first training image in the first training data and the second training image until the preset training goal is reached.

[0132] Optionally, the training module 505 is also used to obtain first training data collected during several historical driving processes and first training position information corresponding to the first training data, the first training data including first training images collected during the driving process; for each first training data, input the first training data and the first training position information corresponding to the first training data into the feature analysis model to obtain environmental feature data corresponding to the first training data; input the environmental feature data corresponding to the first training data into the prediction model to obtain a predicted image; determine the image data collected after a set time from the time the first training data is collected as the target training image; and adjust the parameters in the prediction model with the optimization goal of maximizing the similarity between the predicted image and the target training image until the preset training goal is reached.

[0133] Optionally, the training module 505 is also used to obtain first training data collected during several historical driving processes and first training position information corresponding to the first training data, wherein the first training data includes a first training image collected during the driving process; for each first training data, input the first training data and the first training position information corresponding to the first training data into the feature analysis model to obtain environmental feature data corresponding to the first training data; input the first training data and the environmental feature data corresponding to the first training data into the position estimation model to obtain estimated position information; and adjust the parameters in the position estimation model with the optimization goal of minimizing the position deviation between the estimated position information and the first training position information corresponding to the first training data until the preset training goal is reached.

[0134] This specification also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the above-mentioned obstacle trajectory prediction method.

[0135] This manual also provides Figure 6 The schematic structure diagram of the electronic device shown in FIG. Figure 6 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned obstacle trajectory prediction method. Of course, in addition to the software implementation method, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0136] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0137] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0138] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0139] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0140] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0145] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0146] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0148] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0150] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0151] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for predicting obstacle trajectories, characterized in that: include: Acquire sensor data of the obstacle collected by the unmanned driving device, and determine the position information of the unmanned driving device when collecting the sensor data, wherein the sensor data includes: image data of the obstacle; Inputting the sensing data and the location information into a feature analysis model to determine environmental feature data corresponding to the obstacle; Inputting the environmental feature data into a prediction model to predict image data of obstacles after a set time as a prediction image; Predicting a movement trajectory of the obstacle according to the predicted image; The obtaining of sensor data of obstacles collected by the unmanned driving equipment specifically includes: Acquire sensor data of the obstacle continuously collected by the unmanned driving device; Inputting the sensing data and the location information into a feature analysis model to determine the environmental feature data corresponding to the obstacle specifically includes: For each acquired sensor data, the sensor data and the location information based on which the unmanned driving device collects the sensor data are input into the feature analysis model to determine the environmental feature data corresponding to the sensor data; The step of inputting the environmental feature data into a prediction model to predict the image data of the obstacle collected after a set time as a prediction image specifically includes: For each sensing data, inputting the environmental feature data corresponding to the sensing data into the prediction model to predict the image data of the obstacle collected by the unmanned driving device after the set time as the predicted image corresponding to the sensing data; Predicting the movement trajectory of the obstacle according to the predicted image specifically includes: Predicting a movement trajectory of the obstacle according to an image position of the image of the obstacle in the predicted image corresponding to each sensing data; Predicting the movement trajectory of the obstacle according to the image position of the image of the obstacle in the predicted image corresponding to each sensing data specifically includes: The movement trajectory of the obstacle is predicted based on the image position of the obstacle image in the predicted image corresponding to each sensor data and the estimated actual position based on which the unmanned driving device collects the predicted image corresponding to each sensor data.

2. The method according to claim 1, characterized in that Predicting the movement trajectory of the obstacle according to the image position of the image of the obstacle in the predicted image corresponding to each sensor data and the estimated actual position based on which the unmanned driving device collects the predicted image corresponding to each sensor data specifically includes: For each predicted image corresponding to the sensor data, according to the image position of the image of the obstacle in the predicted image, determine the relative position of the obstacle with respect to the unmanned driving device when the unmanned driving device collects the predicted image as the relative position of the obstacle corresponding to the predicted image; According to the relative position of the obstacle corresponding to the predicted image and the estimated actual position based on which the unmanned driving device collects the predicted image, predict the actual position of the obstacle when the unmanned driving device collects the predicted image as the actual position of the obstacle corresponding to the predicted image; According to the actual position of the obstacle corresponding to each predicted image, the movement trajectory of the obstacle is predicted.

3. The method according to claim 2, characterized in that Estimating the actual position based on which the predicted image corresponding to each sensor data collected by the unmanned driving device is based, specifically including: For each piece of sensor data, the environmental feature data and the predicted image corresponding to the sensor data are input into the position estimation model to estimate the actual position based on which the unmanned driving device collects the predicted image.

4. The method according to claim 1, characterized in that The feature analysis model is trained by: acquiring first training data collected during several driving processes in history, and first training position information corresponding to the first training data, wherein the first training data includes first training images collected during the driving process; For each first training data, the first training data and the first training position information corresponding to the first training data are input into the feature analysis model to obtain the environmental feature data corresponding to the first training data; Inputting the environmental feature data corresponding to the first training data into the adversarial model corresponding to the feature analysis model to obtain a second training image; Taking maximizing the similarity between the first training image and the second training image in the first training data as an optimization goal, the parameters in the feature analysis model are adjusted until a preset training goal is achieved.

5. The method according to claim 1, characterized in that The prediction model is trained in the following way: Acquire first training data collected during several driving processes in history and first training position information corresponding to the first training data, wherein the first training data includes first training images collected during the driving process; For each first training data, the first training data and the first training position information corresponding to the first training data are input into the feature analysis model to obtain the environmental feature data corresponding to the first training data; Inputting the environmental feature data corresponding to the first training data into the prediction model to obtain a prediction image; Determine the image data collected after a set time from the time when the first training data is collected as the target training image; Taking maximizing the similarity between the predicted image and the target training image as the optimization goal, the parameters in the prediction model are adjusted until a preset training goal is achieved.

6. The method according to claim 3, characterized in that The position estimation model is trained by: acquiring first training data collected during several driving processes in history and first training position information corresponding to the first training data, wherein the first training data includes first training images collected during the driving process; For each first training data, the first training data and the first training position information corresponding to the first training data are input into the feature analysis model to obtain the environmental feature data corresponding to the first training data; Inputting the first training data and the environmental feature data corresponding to the first training data into the position estimation model to obtain estimated position information; Taking minimizing the position deviation between the estimated position information and the first training position information corresponding to the first training data as an optimization goal, the parameters in the position estimation model are adjusted until a preset training goal is achieved.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

8. An unmanned driving device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images