Obstacle trajectory prediction method, vehicle-mounted device, vehicle and computer program product
Two-stage training through pre-trained encoder and decoder, combining non-parametric and parameterized prediction, the uncertainty and planning difficulty of obstacle prediction in the prior art are solved, and the safety of autonomous vehicles is improved. Accuracy of obstacle trajectory prediction.
Patent Information
- Application Number
- CN202411455956.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the existing obstacle prediction methods, parameterized prediction ignores the uncertainty and randomness of obstacle movement, and non-parametric prediction cannot provide specific future location information, resulting in increased difficulty in planning for autonomous driving vehicles and insufficient safety.
Using pre-trained encoder and decoder, the future grid occupation distribution and trajectory prediction of obstacles are generated through two-stage training, combining non-parametric and parameterized predictions, and fusing to generate accurate obstacle trajectories.
The accuracy and safety of obstacle trajectory prediction are improved, and the parameterized trajectory prediction is guided through the non-parametric predicted grid occupation distribution, generating higher-precision future trajectories of obstacles.
Smart Images

Figure CN119007163A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of autonomous driving prediction technology, and in particular to an obstacle trajectory prediction method, vehicle-mounted equipment, vehicle, and computer program product. Background Art
[0002] The obstacle trajectory prediction module plays a vital role in autonomous driving technology. This module mainly receives information such as obstacle type, location distribution, speed, driving direction, etc. output by the upstream perception module, analyzes and processes these input information, and finally predicts the location, speed, heading angle, etc. of each obstacle in a certain period of time in the future. Ultimately, these obstacle prediction information will be passed to the planning link to guide the autonomous driving vehicle to generate a reasonable planned route. Therefore, the accuracy of obstacle prediction is directly related to the collision risk, driving safety and comfort of the autonomous driving vehicle itself. Summary of the invention
[0003] The embodiments of the present disclosure at least provide a method for predicting obstacle trajectories, vehicle-mounted equipment, a vehicle, and a computer program product.
[0004] In a first aspect, an embodiment of the present disclosure provides a method for predicting an obstacle trajectory, comprising: Acquire images of the target vehicle during its driving process; Encoding the image based on a pre-trained encoder to obtain encoded data corresponding to the image; fusing the coded data with historical trajectory information of the target obstacle in the image to obtain fused data; Decoding the fused data based on a pre-trained decoder to obtain a predicted trajectory of the target obstacle in a future preset period of time; The encoder and the decoder are obtained by performing second-stage training on a pre-trained encoder and a decoder to be trained; the pre-trained encoder is obtained by performing first-stage training on an encoder to be trained; in the first-stage training, the encoder to be trained and the auxiliary decoder are used to predict the future grid occupancy distribution of obstacles, and in the second-stage training, the pre-trained encoder and the decoder to be trained are used to predict the future trajectory of obstacles.
[0005] In a possible implementation manner, the fusing the coded data with the historical trajectory information of the target obstacle in the image to obtain the fused data includes: From the encoded data, cutting target feature data corresponding to the target obstacle; Encoding the historical trajectory information of the target obstacle to obtain an encoding vector of the historical trajectory information; The target feature data and the encoding vector are concatenated to obtain the fused data.
[0006] In a possible implementation manner, the step of clipping target feature data corresponding to the target obstacle from the encoded data includes: Determining a candidate area to be cropped from the coded data according to a position of the target obstacle in the image and a position mapping relationship between each pixel in the image and each element in the coded data; and, determining a clipping direction of the target feature data according to a moving direction of the target obstacle; Based on the clipping direction, the candidate area to be clipped is clipped to obtain feature data corresponding to the target obstacle.
[0007] In a possible implementation manner, the clipping process is performed on the candidate area to be clipped based on the clipping direction to obtain feature data corresponding to the target obstacle, including: Based on the cropping direction, determining the tilt direction of the rectangular frame used for cropping, and determining the target position of the rectangular frame in the candidate region according to the center position of the candidate region and the tilt direction; Elements falling into the rectangular frame are intercepted from the candidate area to obtain feature data corresponding to the target obstacle.
[0008] In a possible implementation manner, the encoder to be trained and the auxiliary decoder are trained in the first phase in the following manner: Based on the encoder to be trained, encoding processing is performed on a first sample image generated by the vehicle during driving to obtain first sample encoding data corresponding to the first sample image; the first sample encoding data is used to describe features of a first obstacle contained in the first sample image; Decoding the first sample coded data based on the auxiliary decoder to obtain a predicted grid occupancy image of the first obstacle in a first future time period; Based on the predicted grid occupancy image and the actual grid occupancy image of the first obstacle in the first future time period, a first loss is determined, and the encoder to be trained and the auxiliary decoder are trained using the first loss to obtain the pre-trained encoder.
[0009] In a possible implementation manner, the second stage training is performed on the pre-trained encoder and the decoder to be trained in the following manner: Performing encoding processing on a second sample image acquired during the driving of the vehicle based on the pre-trained encoder to obtain second sample encoding data corresponding to the second sample image; fusing the second sample coded data and the historical trajectory information of the second obstacle in the second sample image to obtain sample fusion data, and decoding the sample fusion data based on the decoder to be trained to obtain a predicted trajectory of the second obstacle in the second future time period; A second loss is determined according to difference information between the predicted trajectory and the actual trajectory of the second obstacle in the second future time period, and based on the second loss, the pre-trained encoder and the decoder to be trained are trained to obtain the encoder and the decoder.
[0010] In a possible implementation manner, the method further includes: Determining a target driving path from a plurality of candidate driving paths according to the predicted trajectory; Control the target vehicle to travel along the target driving path.
[0011] In a second aspect, an embodiment of the present disclosure further provides a vehicle-mounted device, comprising a processor; the processor is used to execute the obstacle trajectory prediction method as described in the first aspect, or any one of the first aspect.
[0012] In a third aspect, an embodiment of the present disclosure further provides a vehicle, comprising the vehicle-mounted device as described in the second aspect.
[0013] In a fourth aspect, an optional implementation of the present disclosure further provides a computer program product, which carries a program code, and the instructions included in the program code can be used to execute the steps of the obstacle trajectory prediction method as described in the first aspect or any one of the first aspect.
[0014] The obstacle trajectory prediction method provided by the embodiment of the present disclosure utilizes a pre-trained encoder and a decoder to predict the trajectory of obstacles encountered by a vehicle during driving, wherein the encoder and the decoder are obtained by performing a second-stage training on the pre-trained encoder and the decoder to be trained; the pre-trained encoder is obtained by performing a first-stage training on the encoder to be trained; in the first-stage training, the encoder to be trained and the auxiliary decoder are used to predict the future grid occupancy distribution of the obstacle, and in the second-stage training, the pre-trained encoder and the decoder to be trained are used to predict the future trajectory of the obstacle, so that the possible future grid occupancy distribution of the obstacle generated by the non-parametric prediction is used as a priori knowledge to guide the output of the parameterized trajectory prediction of the obstacle, thereby integrating the non-parametric prediction and the parameterized prediction of the obstacle, so that the obstacle trajectory prediction model can learn the characteristics of the non-parametric prediction, thereby generating a more accurate obstacle trajectory and improving the safety during driving.
[0015] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0017] Figure 1 A flowchart of a method for predicting obstacle trajectories provided in some embodiments of the present disclosure is shown; Figure 2 An example of performing the first stage training of an encoder to be trained provided by some embodiments of the present disclosure is shown; Figure 3 An example of clipping sample feature data from second sample encoded data provided by some embodiments of the present disclosure is shown; Figure 4 A schematic diagram showing an obstacle trajectory prediction device provided by some embodiments of the present disclosure; Figure 5 A schematic diagram of a computer device provided by some embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0019] Among the existing obstacle prediction methods, there are mainly two forms: parametric obstacle prediction and non-parametric obstacle prediction. Parametric prediction directly predicts and outputs parameterized information such as the position coordinates and speed of the obstacle through a neural network model, while non-parametric prediction predicts and outputs the possible grid occupancy distribution of the obstacle in the future and maps it to a rasterized image.
[0020] Although parametric prediction can directly provide information such as the position and speed of obstacles in the future for use by downstream planning modules, it ignores the uncertainty and randomness of obstacle movement. Therefore, the future trajectory information given by parametric prediction cannot truly reflect the actual movement state of obstacles in the future under some working conditions; non-parametric prediction gives the possible distribution area and grid occupancy of obstacles in a certain period of time in the future. Although it can better reflect the uncertainty of obstacle movement, it cannot provide specific and definite future position information of obstacles to downstream planning modules, which increases the difficulty of autonomous driving vehicle planning.
[0021] At present, most of the industry uses one of these methods to predict obstacles, or uses multi-task training to simultaneously generate non-parametric prediction information and parametric information. However, in this way, non-parametric prediction and parametric prediction are relatively independent, resulting in room for further optimization of obstacle prediction and the possibility of further improvement in driving safety.
[0022] Based on the above research, the present disclosure provides a method for predicting obstacle trajectories, which utilizes a pre-trained encoder and a decoder to predict the trajectories of obstacles encountered during vehicle driving, wherein the encoder and the decoder are obtained by performing a second-stage training on the pre-trained encoder and the decoder to be trained; the pre-trained encoder is obtained by performing a first-stage training on the encoder to be trained; in the first-stage training, the encoder to be trained and the auxiliary decoder are used to predict the future grid occupancy distribution of the obstacle, and in the second-stage training, the pre-trained encoder and the decoder to be trained are used to predict the future trajectory of the obstacle, so that the possible future grid occupancy distribution of the obstacle generated by the non-parametric prediction is used as a priori knowledge to guide the output of the parameterized trajectory prediction of the obstacle, thereby integrating the non-parametric prediction and the parameterized prediction of the obstacle, so that the obstacle trajectory prediction model can learn the characteristics of the parameterized prediction, thereby generating a more accurate obstacle trajectory and improving the safety during driving.
[0023] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.
[0024] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0025] To facilitate understanding of this embodiment, a method for predicting obstacle trajectories disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the method for predicting obstacle trajectories provided in an embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device may be a user equipment (UE), a mobile device, a vehicle-mounted device, etc. In some possible implementations, the method for predicting obstacle trajectories may be implemented by a processor calling a computer-readable instruction stored in a memory.
[0026] The following is an explanation of the obstacle trajectory prediction method provided by the embodiment of the present disclosure.
[0027] See also Figure 1 FIG. 1 is a flowchart of a method for predicting obstacle trajectories provided in an embodiment of the present disclosure, wherein the method includes steps S101 to S104, wherein: S101: Acquire an image of the target vehicle during its driving process; wherein the image may be a BEV (Bird's-Eye-View) image. S102: encoding the image based on a pre-trained encoder to obtain encoded data corresponding to the image; S103: Fusing the coded data with the historical trajectory information of the target obstacle in the image to obtain fused data; S104: decoding the fused data based on a pre-trained decoder to obtain a predicted trajectory of the target obstacle in a future preset period of time; The encoder and the decoder are obtained by performing second-stage training on a pre-trained encoder and a decoder to be trained; the pre-trained encoder is obtained by performing first-stage training on an encoder to be trained; in the first-stage training, the encoder to be trained and the auxiliary decoder are used to predict the future grid occupancy distribution of obstacles, and in the second-stage training, the pre-trained encoder and the decoder to be trained are used to predict the future trajectory of obstacles.
[0028] The above S101~S104 are described in detail below: Regarding S101 above: The target vehicle may be, for example, any vehicle that deploys the obstacle trajectory prediction method provided by the embodiment of the present disclosure. A visual perception system may be deployed in the target vehicle, and the visual perception system includes a camera, etc. The visual perception system can obtain raw image data within a certain range around the target vehicle. The raw image data is usually data under a perspective view (Perspective View, PV); in the embodiment of the present disclosure, the raw image data of multiple perspectives can be obtained from the perspective view by a plane projection method to obtain a BEV image under a bird's-eye view (Bird Eye View, BEV) perspective centered on the visual perception system, which is the image described in the embodiment of the present disclosure. Compared with the collected raw image, the BEV image can provide a wider perspective. In the image, according to the driving environment of the target vehicle, it may cover one or more of the lane lines, navigation lines, road boundaries, grid occupancy images of obstacles at the current time and historical time, and grid occupancy images of the vehicle at the current time and historical time.
[0029] In the embodiment of the present disclosure, before executing the obstacle trajectory prediction method, in order to save computing power, it can be determined whether there are dynamic obstacles that need to be tracked in the image, such as other vehicles, pedestrians, etc., based on various information contained in the image. If there are no dynamic obstacles that need to be tracked in the image, the obstacle trajectory prediction method provided by the embodiment of the present disclosure may not be executed.
[0030] Regarding the above S102~S103: The encoder and decoder described in the embodiments of the present disclosure can be deployed in a target vehicle as part of an obstacle trajectory prediction model, for example. The obstacle trajectory prediction model is used to predict the obstacle trajectory using images acquired during the vehicle's driving process to obtain the movement trajectory of a specific obstacle near the vehicle within a certain period of time in the future. Specifically, the encoder is used to encode images acquired by the vehicle during driving to obtain coded data that can be used to describe the characteristics of the target obstacle in the image; if the image includes a target obstacle, the encoder can map the relevant features of the target obstacle into a high-dimensional feature space, so that the coded data carries the characteristic data of the target obstacle.
[0031] Then, the encoded data and the historical trajectory information of the target obstacle are fused to obtain fused data, and the decoder is used to decode the fused data to predict the predicted trajectory of the obstacle in the image within a certain period of time in the future.
[0032] Here, obstacles can also be divided into different categories according to their differences, such as vehicles, pedestrians, and non-motor vehicles. When there are multiple types of obstacles, when generating fusion data, for example, obstacle type information can also be included. In this way, the obstacle trajectory prediction model can perform trajectory prediction for more types of obstacles, thereby improving the generalization of the obstacle trajectory prediction model.
[0033] Here, the historical trajectory information and a certain period in the future are determined based on the time when the image is acquired as a reference; for example, assuming that the time when the vehicle acquires the image is T1, T0 is earlier than T1, and T2 is later than T1; the historical trajectory information includes: the trajectory information of the obstacle in the image during the time period from T0 to T1; a certain period in the future, for example, a period with T1 and T2 as the starting point and end point respectively.
[0034] When training the above encoder and decoder, the encoder to be trained is first trained in the first stage to obtain a pre-trained encoder, and then the pre-trained encoder and the decoder to be trained are trained in the second stage to obtain the encoder and decoder used in the above trajectory prediction process. In this way, the model trained in the first stage is loaded as a pre-trained model into the parameterized prediction training in the second stage. The parameterized prediction in the second stage reuses the CNN encoder in the first stage to perform convolution processing on the input image, and the initial neural network parameters of the encoder network are the neural network parameters after the final training convergence in the first stage.
[0035] Specifically: For example, the following method can be used to perform the first stage training on the encoder to be trained and the auxiliary decoder to obtain the pre-trained encoder: Based on the encoder to be trained, encoding processing is performed on a first sample image generated by the vehicle during driving to obtain first sample encoding data corresponding to the first sample image; the first sample encoding data is used to describe features of a first obstacle contained in the first sample image; Decoding the first sample coded data based on the auxiliary decoder to obtain a predicted grid occupancy image of the first obstacle in a first future time period; Based on the predicted grid occupancy image and the actual grid occupancy image of the first obstacle in the first future time period, a first loss is determined, and the encoder to be trained and the auxiliary decoder are trained using the first loss to obtain the pre-trained encoder.
[0036] Specifically, in the above embodiment, the obstacle grid occupancy image is used to describe the possible future grid occupancy distribution of obstacles during the driving process of the vehicle. Furthermore, the obstacle grid occupancy image includes multiple grid areas, and the pixel value corresponding to each grid area is used to represent the probability of the obstacle appearing in the corresponding grid area.
[0037] The pre-trained encoder generated by the first stage of training of the encoder to be trained can learn the characteristics of the obstacle grid occupancy image, thereby realizing the use of the possible future grid occupancy distribution of the obstacle generated by non-parametric prediction as a priori knowledge to guide the obstacle parametric trajectory prediction output, so as to achieve the fusion of non-parametric prediction and parametric prediction, so that the obstacle trajectory prediction model can output a more accurate obstacle trajectory image.
[0038] The above-mentioned first sample image, for example, is an image obtained during the driving process of the vehicle; the first sample image usually contains the first obstacle; at the same time, after obtaining the first sample image, it is also possible to obtain an actual grid occupancy image of the first obstacle within a certain future period after obtaining the first sample image, and the actual grid occupancy image is used as the true value label of the first sample image to supervise the training process of the first stage training.
[0039] The encoder may be, for example, a convolutional neural network (CNN) encoder, which may perform convolution, pooling, normalization, and other processing on the images acquired by the vehicle during driving, map the images to a high-dimensional feature space, and obtain encoded data of the images. The encoded data includes feature information of various objects in the image, for example, if there are obstacles in the image, the encoded data will include feature information of the obstacles; if there are lane lines in the image, the encoded data will also include feature information of the lane lines.
[0040] During the first stage of training the encoder, in order to enable the encoder to learn the relevant features of the obstacle grid occupancy image, an auxiliary decoder is set for it; the auxiliary decoder usually includes a CNN decoder; the decoder can decode the encoded data generated by the encoder to obtain a predicted grid occupancy image of the obstacle in a certain period of time in the future; the certain period of time in the future is a future period corresponding to the timestamp when the first sample image is obtained.
[0041] The size of the grid occupation image of the first obstacle in the first future time period is the same as the size of the first sample image.
[0042] Afterwards, a cross entropy loss is determined based on the predicted grid occupancy image and the actual grid occupancy image of the first obstacle pair in the first future time period, and the cross entropy loss is used to adjust the parameters of the auxiliary decoder and the encoder to be trained to achieve back propagation training of the encoder to be trained and the auxiliary decoder to obtain a pre-trained encoder.
[0043] like Figure 2 In the example shown, an example of the first stage training of the encoder to be trained is shown. In this example, in the first sample image, there are three obstacles, namely, obstacle 1, obstacle 2 and obstacle 3; the first sample image is input into the encoder to be trained to obtain the first sample encoding data, and then the first sample encoding data is input into the auxiliary decoder to be trained for decoding processing to obtain the predicted grid occupation image of the above three obstacles in the first future period. Among them, the predicted grid occupation image includes multiple grid areas, and the higher the gray value of the grid area, the higher the probability that the obstacle appears at the position corresponding to the grid area in the first future period.
[0044] After the encoder to be trained is trained in the first stage to obtain the pre-trained encoder, the pre-trained encoder and the decoder to be trained can be trained in the second stage in the following manner: Performing encoding processing on a second sample image acquired during the driving of the vehicle based on the pre-trained encoder to obtain second sample encoding data corresponding to the second sample image; fusing the second sample coded data and the historical trajectory information of the second obstacle in the second sample image to obtain sample fusion data, and decoding the sample fusion data based on the decoder to be trained to obtain a predicted trajectory of the second obstacle in the second future time period; A second loss is determined according to difference information between the predicted trajectory and the actual trajectory of the second obstacle in the second future time period, and based on the second loss, the pre-trained encoder and the decoder to be trained are trained to obtain the encoder and the decoder.
[0045] In a specific implementation, the second sample image and the first sample image may be the same image or different images.
[0046] After completing the first stage of training the encoder to be trained, the obtained pre-trained encoder is transplanted to the obstacle trajectory prediction model to be trained; in this process, the parameters of the pre-trained encoder can be assigned to the encoder in the obstacle trajectory prediction model to be trained. Alternatively, the pre-trained encoder and the decoder to be trained are directly associated to form the obstacle trajectory prediction model to be trained.
[0047] When the second sample coded data and the historical trajectory information of the second obstacle are fused to obtain sample fused data, for example, the following method may be used: Cutting sample feature data corresponding to the second obstacle from the sample coded data; Encoding the historical trajectory information of the second obstacle to obtain a sample encoding vector of the historical trajectory information; The sample feature data corresponding to the second obstacle and the sample encoding vector are concatenated to obtain the sample fusion data.
[0048] Specifically, when cutting out the sample feature data corresponding to the second obstacle from the second sample encoded data, for example, the following method may be used: Determine a sample candidate area to be cropped from the second sample coded data according to the position of the second obstacle in the second sample image and the position mapping relationship between each pixel in the second sample image and each element in the second sample coded data; and, determining a sample clipping direction of the sample feature data according to a moving direction of the second obstacle; Based on the sample clipping direction, the sample candidate area to be clipped is clipped to obtain sample feature data corresponding to the second obstacle.
[0049] In the specific implementation, The second sample coded data is obtained by encoding the second sample image, and each element in the second sample coded data is a mapping of pixel values of multiple pixel points in a specific area of the second sample image; this mapping constitutes a position mapping relationship between each pixel point in the second sample image and each element in the second sample coded data.
[0050] The position information of the second obstacle in the second sample image is carried in the second sample image in the form of a bounding box or pixel semantics. When determining the sample candidate area to be detected, for example, based on the position information of the second obstacle in the second sample image and the above-mentioned position mapping relationship, the target elements mapped to the pixels corresponding to the second obstacle are determined from the second sample encoded data, and the area where these target elements are located in the second sample encoded data is determined as the sample candidate area to be cropped.
[0051] In addition, a certain area expansion process may be performed on the position of the target element in the second sample coded data to determine a candidate area centered on the target element and including other corresponding elements of the environment near the second obstacle.
[0052] At the same time, the sample cropping direction of the sample feature data is determined according to the moving direction of the second obstacle, and then the candidate area is cropped according to a certain cropping size based on the sample cropping direction to obtain the second feature data corresponding to the second obstacle.
[0053] Specifically, during cropping, for example, based on the sample cropping direction, the tilt direction of the rectangular frame used for cropping is determined, and according to the center position of the sample candidate region and the tilt direction, the target position of the rectangular frame in the sample candidate region is determined; Elements falling into the rectangular frame are cropped from the sample candidate area to obtain sample feature data corresponding to the second obstacle.
[0054] In the above process, in order to allow the neural network to pay more attention to the obstacle information, the coding vector is cropped. Since the position of the obstacle in the original image and the position of the corresponding feature after convolution are in a one-to-one linear relationship, the feature area centered on the obstacle is found in the coding vector, and the feature is cropped in the area. The direction of the feature cropping is the vehicle orientation of the obstacle vehicle at the current moment. It is worth noting that the cropped area not only contains the obstacle vehicle information, but also the surrounding environment information centered on the obstacle. Compared with cropping the original image, cropping the high-dimensional feature coding vector can obtain more feature areas and feature information.
[0055] like Figure 3In the example shown, a specific example of performing cropping in the sample candidate region of the second sample coding vector according to different cropping directions is shown. In this example, the cropping directions of the two obstacles a1 and b1 are as follows: Figure 3 As shown by the arrows in , according to different clipping directions, the sample feature data corresponding to the two obstacles are obtained as follows Figure 3 As shown in a2 and b2.
[0056] After obtaining the sample feature data corresponding to the second obstacle, the sample feature data and the historical trajectory information corresponding to the second obstacle are spliced to obtain fused data.
[0057] Similarly, the sample feature data, the historical trajectory information corresponding to the second obstacle, and the type information of the second obstacle may be concatenated to obtain fused data. By adding the type information of the obstacle, the prediction accuracy of the movement trajectory of the obstacle can be improved.
[0058] Here, the historical trajectory information corresponding to the second obstacle includes, for example, the position information of the second obstacle in the second sample image at multiple historical time points during the process of acquiring the second sample image. Before splicing, the historical trajectory information can be encoded, and then the encoding result and the sample feature data are spliced.
[0059] The types of obstacles may include, for example, pedestrians, motor vehicles, non-motor vehicles, etc., and may be determined according to actual scene requirements.
[0060] After obtaining the sample fusion data, the sample fusion data is decoded by using the decoder to be trained to obtain the predicted trajectory of the second obstacle in the second future period. The second future period is, for example, a future period relative to the timestamp of the second sample image.
[0061] The decoder to be trained can be, for example, a recurrent neural network (RNN) decoder, such as a gated recurrent unit (GRU), a long short-term memory network (LSTM), etc., or other types of recurrent neural networks. Taking LSTM as an example, the LSTM decoder can use the input fusion data to predict the position of obstacles at multiple time points in the future; the obstacle positions corresponding to multiple time points in the future constitute the movement trajectory of the obstacle in a certain period of time in the future.
[0062] Taking the decoder as LSTM as an example, its input may include the obstacle's historical trajectory point information encoding vector and the image information encoding vector just cut out (i.e., target feature data), and the two are spliced and sent to the LSTM network structure to generate the position coordinate information of the obstacle within a certain period of time in the future, and then the coordinate information of these obstacles is sent to the downstream planning module. Optionally, the decoder input may also include the obstacle type encoding vector, which is spliced and input into the LSTM decoder by splicing the obstacle's historical trajectory point information encoding vector, the obstacle type encoding vector, and the image information encoding vector just cut out (i.e., target feature data).
[0063] After obtaining the predicted trajectory of the second obstacle in the second future time period using the decoder to be trained, the predicted trajectory and the L2 distance of the second obstacle in the second future time period can be calculated, the L2 distance can be used as the second loss, and back-propagated to achieve training of the pre-trained encoder and the decoder to be trained, thereby obtaining a pre-trained encoder and decoder.
[0064] Afterwards, the pre-trained encoder and decoder can be deployed on the target vehicle as an obstacle trajectory prediction model to implement the above steps S102 and S104.
[0065] In addition, in the above S103, the process of fusing the coded data and the historical trajectory information of the target obstacle in the image to obtain the fused data is similar to the process of fusing the sample coded data and the historical trajectory of the second obstacle in the second sample image to obtain the sample fused data in the above embodiment, which specifically includes: From the encoded data, cutting target feature data corresponding to the target obstacle; Encoding the historical trajectory information of the target obstacle to obtain an encoding vector of the historical trajectory information; The target feature data and the encoding vector are concatenated to obtain the fused data.
[0066] Here, the historical trajectory information may include, for example, trajectory coordinates, speed, acceleration, heading angle, and other information of the target vehicle.
[0067] The process of cutting out target feature data corresponding to the target obstacle from the encoded data may include, for example: Determining a candidate area to be cropped from the encoded data according to a position of the target obstacle in the first image and a position mapping relationship between each pixel in the image and each element in the encoded data; and, determining a clipping direction of the target feature data according to a moving direction of the target obstacle; Based on the clipping direction, the candidate area to be clipped is clipped to obtain feature data corresponding to the target obstacle.
[0068] When performing cropping processing on the candidate region to be cropped according to the cropping direction, for example, the inclination direction of the rectangular frame used for cropping can be determined based on the cropping direction, and the target position of the rectangular frame in the candidate region can be determined according to the center position of the candidate region and the inclination direction; Elements falling into the rectangular frame are intercepted from the candidate area to obtain feature data corresponding to the target obstacle.
[0069] The detailed process can be found in the above-mentioned process of clipping sample coded data, which will not be described in detail here.
[0070] In another embodiment of the present disclosure, the obstacle trajectory prediction method provided by the embodiment of the present disclosure further includes: Determining a target driving path from a plurality of candidate driving paths according to the predicted trajectory; Control the target vehicle to travel along the target driving path.
[0071] Here, the multiple candidate driving paths are, for example, multiple optional paths determined by using the current position of the target vehicle and the road conditions of the current driving section, such as the lane line position, road surface area, etc. Then, the cost of each driving path can be determined based on the predicted trajectory of the target obstacle and each candidate path; the cost is used to describe the risk level of each candidate driving path, such as the probability of collision with an obstacle. Then, based on the cost of each candidate driving path, the target driving path is determined from each candidate driving path, and the target vehicle is controlled to drive along the target driving path.
[0072] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0073] Based on the same inventive concept, an obstacle trajectory prediction device corresponding to the obstacle trajectory prediction method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned obstacle trajectory prediction method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0074] Reference Figure 4 FIG. 1 is a schematic diagram of a device for predicting obstacle trajectories provided by an embodiment of the present disclosure, wherein the device comprises: An acquisition module 41 is used to acquire images of the target vehicle during its driving process; The control module 42 is used to encode the image based on a pre-trained encoder to obtain encoded data corresponding to the image; fuse the encoded data with historical trajectory information of the target obstacle in the image to obtain fused data; and decode the fused data based on a pre-trained decoder to obtain a predicted trajectory of the target obstacle in a preset future period of time; The encoder and the decoder are obtained by performing second-stage training on a pre-trained encoder and a decoder to be trained; the pre-trained encoder is obtained by performing first-stage training on an encoder to be trained; in the first-stage training, the encoder to be trained and the auxiliary decoder are used to predict the future grid occupancy distribution of obstacles, and in the second-stage training, the pre-trained encoder and the decoder to be trained are used to predict the future trajectory of obstacles.
[0075] In a possible implementation manner, the control module 42, when fusing the coded data with the historical trajectory information of the target obstacle in the image to obtain the fused data, is configured to: From the encoded data, cutting target feature data corresponding to the target obstacle; Encoding the historical trajectory information of the target obstacle to obtain an encoding vector of the historical trajectory information; The target feature data and the encoding vector are concatenated to obtain the fused data.
[0076] In a possible implementation manner, the control module 42, when cutting the target feature data corresponding to the target obstacle from the coded data, is configured to: Determining a candidate area to be cropped from the encoded data according to a position of the target obstacle in the first image and a position mapping relationship between each pixel in the image and each element in the encoded data; and, determining a clipping direction of the target feature data according to a moving direction of the target obstacle; Based on the clipping direction, the candidate area to be clipped is clipped to obtain feature data corresponding to the target obstacle.
[0077] In a possible implementation manner, the control module 42, when performing a clipping process on the candidate area to be clipped based on the clipping direction to obtain feature data corresponding to the target obstacle, is configured to: Based on the cropping direction, determining the tilt direction of the rectangular frame used for cropping, and determining the target position of the rectangular frame in the candidate region according to the center position of the candidate region and the tilt direction; Elements falling into the rectangular frame are intercepted from the candidate area to obtain feature data corresponding to the target obstacle.
[0078] In a possible implementation manner, the present invention further includes: a first training module 43, configured to perform a first-stage training on the encoder to be trained in the following manner: Based on the encoder to be trained, encoding processing is performed on a first sample image generated by the vehicle during driving to obtain first sample encoding data corresponding to the first sample image; the first sample encoding data is used to describe features of a first obstacle contained in the first sample image; Decoding the first sample coded data based on the auxiliary decoder to obtain a predicted grid occupancy image of the first obstacle in a first future time period; Based on the predicted grid occupancy image and the actual grid occupancy image of the first obstacle in the first future time period, a first loss is determined, and the encoder to be trained and the auxiliary decoder are trained using the first loss to obtain the pre-trained encoder.
[0079] In a possible implementation manner, the present invention further includes: a second training module 44, configured to perform a second phase of training on the pre-trained encoder and the decoder to be trained in the following manner: Performing encoding processing on a second sample image acquired during the driving of the vehicle based on the pre-trained encoder to obtain second sample encoding data corresponding to the second sample image; fusing the second sample coded data and the historical trajectory information of the second obstacle in the second sample image to obtain sample fusion data, and decoding the sample fusion data based on the decoder to be trained to obtain a predicted trajectory of the second obstacle in the second future time period; A second loss is determined according to difference information between the predicted trajectory and the actual trajectory of the second obstacle in the second future time period, and based on the second loss, the pre-trained encoder and the decoder to be trained are trained to obtain the encoder and the decoder.
[0080] In a possible implementation manner, the control module 42 is further configured to: Determining a target driving path from a plurality of candidate driving paths according to the predicted trajectory; Control the target vehicle to travel along the target driving path.
[0081] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0082] The present disclosure also provides a vehicle-mounted device, such as Figure 5 FIG. 1 is a schematic diagram of a vehicle-mounted device provided by an embodiment of the present disclosure, including: A processor 51 and a memory 52; the memory 52 stores machine-readable instructions executable by the processor 51, and the processor 51 is used to execute the machine-readable instructions stored in the memory 52. When the machine-readable instructions are executed by the processor 51, the processor 51 performs the following steps: Acquire images of the target vehicle during its driving process; Encoding the image based on a pre-trained encoder to obtain encoded data corresponding to the image; fusing the coded data with historical trajectory information of the target obstacle in the image to obtain fused data; Decoding the fused data based on a pre-trained decoder to obtain a predicted trajectory of the target obstacle in a future preset period of time; The encoder and the decoder are obtained by performing second-stage training on a pre-trained encoder and a decoder to be trained; the pre-trained encoder is obtained by performing first-stage training on an encoder to be trained; in the first-stage training, the encoder to be trained is used to predict the future grid occupancy distribution of obstacles, and in the second-stage training, the pre-trained encoder and the encoder to be trained are used to predict the future trajectory of obstacles.
[0083] The above-mentioned memory 52 includes an internal memory 521 and an external memory 522; the internal memory 521 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 51, as well as the data exchanged with the external memory 522 such as a hard disk. The processor 51 exchanges data with the external memory 522 through the internal memory 521.
[0084] The specific execution process of the above instructions can refer to the steps of the obstacle trajectory prediction method described in the embodiment of the present disclosure, which will not be repeated here.
[0085] The embodiment of the present disclosure also provides a vehicle, including a vehicle body and the vehicle-mounted device as described in the embodiment of the present disclosure.
[0086] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the obstacle trajectory prediction method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0087] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the obstacle trajectory prediction method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.
[0088] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0089] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0091] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0092] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0093] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A method for predicting obstacle trajectories, characterized in that: include: Acquire images of the target vehicle during its driving process; Encoding the image based on a pre-trained encoder to obtain encoded data corresponding to the image; fusing the coded data with historical trajectory information of the target obstacle in the image to obtain fused data; Decoding the fused data based on a pre-trained decoder to obtain a predicted trajectory of the target obstacle in a future preset period of time; The encoder and the decoder are obtained by performing second-stage training on a pre-trained encoder and a decoder to be trained; the pre-trained encoder is obtained by performing first-stage training on an encoder to be trained; in the first-stage training, the encoder to be trained and the auxiliary decoder are used to predict the future grid occupancy distribution of obstacles, and in the second-stage training, the pre-trained encoder and the decoder to be trained are used to predict the future trajectory of obstacles.
2. The method according to claim 1, characterized in that The fusing the coded data and the historical trajectory information of the target obstacle in the image to obtain fused data includes: From the encoded data, cutting target feature data corresponding to the target obstacle; Encoding the historical trajectory information of the target obstacle to obtain an encoding vector of the historical trajectory information; The target feature data and the encoding vector are concatenated to obtain the fused data.
3. The method according to claim 2, characterized in that The step of cutting target feature data corresponding to the target obstacle from the encoded data includes: Determining a candidate area to be cropped from the coded data according to a position of the target obstacle in the image and a position mapping relationship between each pixel in the image and each element in the coded data; and, determining a clipping direction of the target feature data according to a moving direction of the target obstacle; Based on the clipping direction, the candidate area to be clipped is clipped to obtain feature data corresponding to the target obstacle.
4. The method according to claim 3, characterized in that The step of performing clipping processing on the candidate area to be clipped based on the clipping direction to obtain feature data corresponding to the target obstacle includes: Based on the cropping direction, determining the tilt direction of the rectangular frame used for cropping, and determining the target position of the rectangular frame in the candidate region according to the center position of the candidate region and the tilt direction; Elements falling into the rectangular frame are intercepted from the candidate area to obtain feature data corresponding to the target obstacle.
5. The method according to any one of claims 1 to 4, characterized in that: The first stage training of the encoder to be trained and the auxiliary decoder is performed in the following manner: Based on the encoder to be trained, encoding a first sample image generated by the vehicle during driving to obtain first sample encoding data corresponding to the first sample image; The first sample encoding data is used to describe the characteristics of the first obstacle contained in the first sample image; Decoding the first sample coded data based on the auxiliary decoder to obtain a predicted grid occupancy image of the first obstacle in a first future time period; Based on the predicted grid occupancy image and the actual grid occupancy image of the first obstacle in the first future time period, a first loss is determined, and the encoder to be trained and the auxiliary decoder are trained using the first loss to obtain the pre-trained encoder.
6. The method according to any one of claims 1 to 4, characterized in that: The pre-trained encoder and the decoder to be trained are trained in the second stage in the following manner: Performing encoding processing on a second sample image acquired during the driving of the vehicle based on the pre-trained encoder to obtain second sample encoding data corresponding to the second sample image; fusing the second sample coded data and the historical trajectory information of the second obstacle in the second sample image to obtain sample fusion data, and decoding the sample fusion data based on the decoder to be trained to obtain a predicted trajectory of the second obstacle in a second future time period; A second loss is determined according to difference information between the predicted trajectory and the actual trajectory of the second obstacle in the second future time period, and based on the second loss, the pre-trained encoder and the decoder to be trained are trained to obtain the encoder and the decoder.
7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determining a target driving path from a plurality of candidate driving paths according to the predicted trajectory; Control the target vehicle to travel along the target driving path.
8. A vehicle-mounted device, characterized in that: include: processor; The processor is used to execute the obstacle trajectory prediction method as described in any one of claims 1-7.
9. A vehicle, characterized in that: include: A vehicle body, and the vehicle-mounted device as claimed in claim 8.
10. A computer program product, characterized in that The computer program product carries a program code, and the program code includes instructions that can be used to execute the steps of the obstacle trajectory prediction method as described in any one of claims 1-7.
Citation Information
Patent Citations
Model training method and device
CN112677993A
Obstacle trajectory prediction method and device, equipment and medium
CN116309689A
Vehicle trajectory prediction method, neural network prediction model training method and device
CN117079235A
Trajectory prediction model training method and device, trajectory prediction method and device and vehicle
CN117935208A
Autonomous driving map enhanced trajectory prediction method integrating semantic information
CN118470672A
