Trajectory data processing method and device, model training method and device, and autonomous vehicle

Abstract

The present disclosure provides a trajectory data processing method, relates to the technical field of artificial intelligence, and particularly relates to the fields of automatic driving, deep learning and computer vision. The specific implementation scheme is: determining intention information and semantic relationship information of a target obstacle according to current trajectory information of the target obstacle, wherein the semantic relationship information is used to represent the relationship between the target obstacle and at least one object; and determining target trajectory information of the target obstacle according to the current trajectory information, the intention information and the semantic relationship information. The present disclosure also provides a deep learning model training method and device, an electronic device, a storage medium and an autonomous vehicle.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the fields of autonomous driving, deep learning and computer vision. More specifically, the present disclosure provides a trajectory data processing method, a deep learning model training method, an apparatus, an electronic device, a storage medium and an autonomous vehicle. BACKGROUND

[0002] An autonomous vehicle can perceive the surrounding environment through a perception component (e.g., a sensor) to obtain surrounding environment data. The surrounding environment data is combined with map navigation data, and data processing is performed based on artificial intelligence technology to make a driving decision. Finally, the autonomous driving of the autonomous vehicle is completed according to the driving decision through a control and execution system. SUMMARY

[0003] The present disclosure provides a trajectory data processing method, a deep learning model training method, an apparatus, an electronic device, a storage medium and an autonomous vehicle.

[0004] According to an aspect of the present disclosure, a trajectory data processing method is provided, which includes: determining intention information and semantic relationship information of a target obstacle according to current trajectory information of the target obstacle, wherein the semantic relationship information is used to represent a relationship between the target obstacle and at least one object; and determining target trajectory information of the target obstacle according to the current trajectory information, the intention information and the semantic relationship information.

[0005] According to an aspect of the present disclosure, a deep learning model training method is provided, the deep learning model including a first sub-model and a second sub-model, which includes: determining intention information and semantic relationship information of a target obstacle according to sample trajectory information of the target obstacle and the first sub-model, wherein the semantic relationship information is used to represent a relationship between the target obstacle and at least one object; inputting the sample trajectory information, the intention information and the semantic relationship information into the second sub-model to determine output trajectory information of the target obstacle; and training the deep learning model according to the output trajectory information and a trajectory label of the sample trajectory information.

[0006] According to another aspect of the present disclosure, there is provided a trajectory data processing apparatus, comprising: a first determining module configured to determine, according to current trajectory information of a target obstacle, intention information of the target obstacle and semantic relationship information, wherein the semantic relationship information is used to represent a relationship between the target obstacle and at least one object; and a second determining module configured to determine, according to the current trajectory information, the intention information and the semantic relationship information, target trajectory information of the target obstacle.

[0007] According to another aspect of the present disclosure, there is provided a training apparatus of a deep learning model, the deep learning model comprising a first sub-model and a second sub-model, the apparatus comprising: a third determining module configured to determine, according to sample trajectory information of a target obstacle and the first sub-model, intention information of the target obstacle and semantic relationship information, wherein the semantic relationship information is used to represent a relationship between the target obstacle and at least one object; a fourth determining module configured to input the sample trajectory information, the intention information and the semantic relationship information into the second sub-model to determine output trajectory information of the target obstacle; and a training module configured to train the deep learning model according to the output trajectory information and a trajectory label of the sample trajectory information.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided by the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method provided by the present disclosure.

[0010] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, the computer program implementing the method provided by the present disclosure when executed by a processor.

[0011] According to another aspect of the present disclosure, there is provided an autonomous vehicle comprising the electronic device provided by the present disclosure.

[0012] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0014] Figure 1 is an exemplary system architecture schematic diagram of an example system to which the trajectory data processing method and device can be applied according to an embodiment of the present disclosure;

[0015] Figure 2 is a flowchart of a trajectory data processing method according to an embodiment of the present disclosure;

[0016] Figure 3 is a flowchart of a trajectory data processing method according to another embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram of a trajectory data processing method according to an embodiment of the present disclosure;

[0018] Figure 5 is a flowchart of a training method of a deep learning model according to an embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram of a training method of a deep learning model according to an embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram of a second sub-model according to an embodiment of the present disclosure;

[0021] Figure 8 is a schematic diagram of a first sub-model according to an embodiment of the present disclosure;

[0022] Figure 9 is a block diagram of a trajectory data processing device according to an embodiment of the present disclosure;

[0023] Figure 10 is a block diagram of a training device of a deep learning model according to an embodiment of the present disclosure; and

[0024] Figure 11 is a block diagram of an electronic device to which a trajectory data processing method and / or a training method of a deep learning model can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of known functions and constructions are omitted in the following description for clarity and conciseness.

[0026] A large-scale deep learning model can be used to determine the driving trajectory of the obstacle. After training the deep learning model based on the training data set of scenario A, the trained deep learning model can accurately determine the driving trajectory of the obstacle under the scenario A. However, in scenario B, the trained deep learning model needs to be retrained to improve the accuracy of the model in scenario B.

[0027] Figure 1 is an exemplary system architecture schematic diagram to which the trajectory data processing method and apparatus according to an embodiment of the present disclosure can be applied. It should be noted that, Figure 1 The system architecture shown is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0028] As Figure 1 shown, the system architecture 100 according to the embodiment can include sensors 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the sensors 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired and / or wireless communication links, etc.

[0029] The sensors 101, 102, 103 can interact with the server 105 through the network 104 to receive or send messages, etc.

[0030] The sensors 101, 102, 103 can be functional elements integrated on the autonomous vehicle 106, such as infrared sensors, ultrasonic sensors, millimeter wave radars, information collection devices, etc. The sensors 101, 102, 103 can be used to collect state data of obstacles around the autonomous vehicle 106 and surrounding road data.

[0031] The server 105 can also be integrated on the autonomous vehicle 106, but is not limited thereto, and can also be set at a remote end capable of establishing communication with the vehicle terminal, and can be implemented as a distributed server cluster composed of multiple servers, or as a single server.

[0032] The server 105 can be a server that provides various services. For example, a map application, a data processing application, and the like can be installed on the server 105. Taking an example in which the server 105 runs the data processing application, the server 105 receives state data of an obstacle and road data transmitted by the sensors 101, 102, and 103 through the network 104. One or more of the state data of the obstacle and the road data can be treated as to-be-processed data. The to-be-processed data is processed to obtain target data.

[0033] It should be noted that the trajectory data processing method provided in the embodiments of the present disclosure can be executed by the server 105. Correspondingly, the trajectory data processing apparatus provided in the embodiments of the present disclosure can also be arranged in the server 105. However, this is not limited thereto. The trajectory data processing method provided in the embodiments of the present disclosure can also be executed by the sensors 101, 102, or 103. Correspondingly, the trajectory data processing apparatus provided in the embodiments of the present disclosure can also be arranged in the sensors 101, 102, or 103.

[0034] It should be understood that Figure 1 The number of sensors, networks, and servers in the above-mentioned system is merely illustrative. According to the needs of implementation, there can be any number of sensors, networks, and servers.

[0035] It should be noted that the serial numbers of the operations in the following method are merely used for representing the operations and for description, and should not be regarded as representing the execution sequence of the operations. Unless explicitly indicated, the method does not need to be executed in the order shown.

[0036] Figure 2 FIG. 2 is a flowchart of a trajectory data processing method according to one embodiment of the present disclosure.

[0037] As shown in FIG. 2, the method 200 can include operations S210 to S220. Figure 2

[0038] In operation S210, intention information and semantic relationship information of a target obstacle are determined according to current trajectory information of the target obstacle.

[0039] For example, the semantic relationship information is used to represent a relationship between the target obstacle and at least one object.

[0040] For example, the target obstacle can be an obstacle perceived by any sensor on the autonomous vehicle 106. In one example, the obstacle can include a vehicle, a pedestrian, and the like.

[0041] For example, the object can include an obstacle. In one example, as described above, the obstacle can include a vehicle, a pedestrian, and the like. ​

[0042] For another example, the object can also include a ground feature. In one example, the ground feature can include a lane line, a signboard, and the like.

[0043] For example, the current trajectory information can include a geographic coordinate of the target obstacle at a current time. For another example, the current trajectory information can also correspond to a plurality of geographic coordinates of the target obstacle within a preset time period, each geographic coordinate can correspond to a time within the preset time period.

[0044] For example, the intention information and the semantic relationship information of the target obstacle can be determined according to various manners. In one example, taking that the target obstacle is a vehicle C_1 perceived by the autonomous vehicle 106 as an example, the intention information Inten_1 of the vehicle C_1 can be overtaking, for example. In one example, the semantic relationship information Sr_1 of the vehicle C_1 can be that the vehicle C_1 will overtake from the left side of a vehicle C_2, for example.

[0045] In operation S220, according to the current trajectory information, the intention information, and the semantic relationship information, the target trajectory information of the target obstacle is determined.

[0046] For example, the target trajectory information can include a geographic coordinate of the target obstacle at a next time. In one example, according to the current trajectory information, the intention information Inten_1, and the semantic relationship information Sr_1 of the vehicle C_1, the geographic coordinate of the vehicle C_1 at the next time can be determined, so that the autonomous vehicle 106 perceiving the vehicle C_1 determines whether to adjust the driving route according to the geographic coordinate of the vehicle C_1 at the next time.

[0047] Through the embodiments of the present disclosure, the trajectory information of the target obstacle can be accurately determined based on the current trajectory information, the intention information, and the semantic relationship information.

[0048] In some embodiments, the target obstacle described above can also be the autonomous vehicle 106 itself.

[0049] In some embodiments, determining the intention information and the semantic relationship information of the target obstacle according to the current trajectory information of the target obstacle includes: performing feature extraction on the current trajectory information to obtain semantic feature information; and determining the intention information and the semantic relationship information of the target obstacle according to the semantic feature information.

[0050] For example, various deep learning models can be used to perform feature extraction on the current trajectory information. For another example, the various deep learning models can be a semantic segmentation model, a target detection model, and the like, which are not limited in the present disclosure.

[0051] For example, a GNN (Graph Neural Network) model can be used to determine the intention information and semantic relationship information of the target obstacle according to the semantic feature information. For another example, a graph Transformer model or a GCN (Graph Convolutional Network) model can be used to determine the intention information and semantic relationship information of the target obstacle according to the semantic feature information.

[0052] One way of determining the adjacency matrix of the GNN model, the GCN model or the graph Transformer model will be described below in combination with Figure 3

[0053] Figure 3 is a flowchart of a trajectory data processing method according to another embodiment of the present disclosure.

[0054] As shown in Figure 3 , the method 310 can determine the intention information and semantic relationship information of the target obstacle according to the current trajectory information of the target obstacle, which will be described in detail below in combination with operation S311 to operation S314.

[0055] In operation S311, at least one target object with a distance less than or equal to a preset distance threshold from the target obstacle is determined according to the position information of the target obstacle.

[0056] For example, the position information of the target obstacle can be the geographic coordinates of the target obstacle at the current time as described above.

[0057] For another example, the preset distance threshold can be 20 meters, for example. In one example, objects within 20 meters around the target obstacle can be regarded as target objects.

[0058] In operation S312, at least one edge relationship information is obtained according to the target obstacle and the at least one target object.

[0059] For example, an edge connecting the target obstacle and each target object can be established to obtain each edge relationship information.

[0060] In operation S313, an adjacency matrix is obtained according to the at least one edge relationship information.

[0061] For example, an adjacency matrix can be determined according to the target obstacle, the at least one target object and the at least one edge relationship information.

[0062] In operation S314, the intention information and the semantic relationship information are determined according to the current trajectory information and the adjacency matrix.

[0063] ​For example, the current trajectory information is input into a GNN model that performs data processing based on the adjacency matrix, to determine the intention information and the semantic relationship information.

[0064] In some embodiments, determining the target trajectory information according to the current trajectory information, the intention information, and the semantic relationship information includes: determining hidden layer feature information according to the current trajectory information; determining input feature information according to the intention information and the semantic relationship information; and determining the target trajectory information according to the hidden layer feature information and the input feature information. Details will be described below in conjunction with Figure 4

[0065] Figure 4 is a schematic diagram of a trajectory data processing method according to another embodiment of the present disclosure.

[0066] As shown in Figure 4 , the semantic feature information 402 can be input into the first sub-model 410 to obtain the intention information and the semantic relationship information of the target obstacle. In one example, the semantic feature information 402 is obtained by performing feature extraction on the current trajectory information 401.

[0067] As shown in Figure 4 , the current trajectory information 401, the intention information, and the semantic relationship information can be input into the second sub-model 420 to determine the target trajectory information 403 of the target obstacle.

[0068] For example, the first sub-model 410 can be, for example, the GNN model described above.

[0069] For example, the second sub-model 420 can be, for example, an LSTM (Long Short-Term Memory) model.

[0070] In one example, the LSTM model includes a plurality of LSTM units. The current trajectory information 401 can be input as the input X_i of the i-th LSTM unit. The i-th LSTM unit can determine the i-th hidden layer feature information h_i according to the current trajectory information 401. i is an integer greater than or equal to 1. In this embodiment, i can be 1.

[0071] Next, the intention information and the semantic relationship can be fused to determine the input feature information X_i+1. The i+1-th LSTM unit can determine the i+1-th hidden layer feature information according to the input feature information X_i+1 and the i-th hidden layer feature information h_i. In one example, the target trajectory information 403 can be determined based on the i+1-th hidden layer feature information.

[0072] Figure 5 is a flowchart of a training method of a deep learning model according to another embodiment of the present disclosure.​

[0073] In the embodiments of the present disclosure, the deep learning model can include a first sub-model and a second sub-model.

[0074] For example, the first sub-model can be one of a GNN model, a GCN model or a graph Transformer model.

[0075] For another example, the second sub-model can be an LSTM model.

[0076] As shown in the method 500 includes operations S510 to S530. Figure 5

[0077] At operation S510, intention information and semantic relationship information of the target obstacle are determined according to sample trajectory information of the target obstacle and the first sub-model.

[0078] For example, the semantic relationship information is used to represent the relationship between the target obstacle and at least one object.

[0079] At operation S520, the sample trajectory information, the intention information and the semantic relationship information are input into the second sub-model to determine output trajectory information of the target obstacle.

[0080] It can be understood that operations S510 and S520 in the method 500 are the same or similar to operations S210 and S220 in the method 200, and the present disclosure will not be repeated here.

[0081] At operation S530, the deep learning model is trained according to trajectory labels of the output trajectory information and the sample trajectory information.

[0082] For example, the difference between the output trajectory information and the trajectory labels can be calculated by using an MSE (Mean Square Error) loss function. The parameters of the first sub-model and the second sub-model are adjusted according to the difference to train the deep learning model.

[0083] Through the embodiments of the present disclosure, the deep learning model trained can accurately determine the trajectory information of the target obstacle.

[0084] Through the embodiments of the present disclosure, the deep learning model includes two sub-models, which can quickly and accurately determine the trajectory information of the target obstacle in different scenes. For example, after the deep learning model is trained based on a training data set of a scene A, the trained deep learning model can accurately determine the driving trajectory of the obstacle in the scene A.

[0085] ​In the scenario B, the parameters of the first sub-model in the trained deep learning model can be kept unchanged, and the second sub-model is trained based on the training data of the scenario B to improve the accuracy of the deep learning model in the scenario B, and the time cost for training the model can be reduced.

[0086] In some embodiments, determining the intention information and the semantic relationship information of the target obstacle according to the sample trajectory information of the target obstacle and the first sub-model comprises: performing feature extraction on the sample trajectory information to obtain semantic feature information; and inputting the semantic feature information into the first sub-model to determine the intention information and the semantic relationship information.

[0087] For example, various deep learning models can be used to perform feature extraction on the current trajectory information. For another example, the various deep learning models can be semantic segmentation models, target detection models, etc., which are not limited in the present disclosure.

[0088] For example, the GNN model can be used to determine the intention information and the semantic relationship information of the target obstacle according to the semantic feature information. For another example, the graph Transformer model or the GCN model can be used to determine the intention information and the semantic relationship information of the target obstacle according to the semantic feature information.

[0089] In some embodiments, determining the intention information and the semantic relationship information of the target obstacle according to the sample trajectory information of the target obstacle and the first sub-model comprises: determining at least one target object with a distance less than or equal to a preset distance threshold from the target obstacle according to the position information of the target obstacle; obtaining at least one edge relationship information according to the target obstacle and the at least one target object; obtaining an adjacency matrix according to the at least one edge relationship information; and determining the intention information and the semantic relationship information according to the sample trajectory information and the first sub-model based on the data processing of the adjacency matrix. It can be understood that the manner of determining the intention information and the semantic relationship information in this embodiment is the same as or similar to the method 310, and the present disclosure will not be repeated here.

[0090] In some embodiments, the second sub-model comprises a plurality of deep learning units, and inputting the sample trajectory information, the intention information and the semantic relationship information into the second sub-model to determine the output trajectory information comprises: inputting the sample trajectory information into an i-th deep learning unit of the second sub-model to determine hidden layer feature information, where i is an integer greater than or equal to 1; determining input feature information according to the intention information and the semantic relationship information; and inputting the hidden layer feature information and the input feature information into an i+1-th deep learning unit to determine the output trajectory information.

[0091] In some embodiments, training the deep learning model according to the trajectory label of the output trajectory information and the sample trajectory information comprises: adjusting parameters of the first sub-model and the second sub-model according to the output trajectory information and the trajectory label, so as to train the deep learning model. Details will be described below in combination with Figure 6

[0092] Figure 6 is a schematic diagram of a trajectory data processing method according to another embodiment of the present disclosure.

[0093] As shown in Figure 6 , the deep learning model 600 can include a first sub-model 610 and a second sub-model 620.

[0094] The semantic feature information 602 can be input into the first sub-model 610 to obtain the intention information and the semantic relationship information of the target obstacle. In one example, the semantic feature information 602 can be obtained by performing feature extraction on the sample trajectory information 601.

[0095] As shown in Figure 6 , the sample trajectory information 601, the intention information and the semantic relationship information can be input into the second sub-model 620 to determine the output trajectory information 603 of the target obstacle.

[0096] For example, the first sub-model 610 can be a GNN model.

[0097] For example, the second sub-model 620 can be an LSTM model.

[0098] In one example, the difference value 605 between the output trajectory information 603 and the trajectory label 604 can be output by using an MSE loss function. The parameters of the first sub-model 610 and the second sub-model 620 are adjusted according to the difference value 605, so as to train the deep learning model 600.

[0099] For example, the second sub-model 620 includes a plurality of deep learning units. In one example, as described above, the second sub-model 620 can be an LSTM model, and accordingly, the deep learning units can be LSTM units.

[0100] In one example, the sample trajectory information 601 can be input as X_i into the i-th LSTM unit. The i-th LSTM unit determines the i-th hidden layer feature information h_i according to the sample trajectory information 601. i is an integer greater than or equal to 1. In this embodiment, i can be 1.

[0101] ​Next, the intent information and the semantic relation can be fused to determine the input feature information X_i+1. The (i+1)th LSTM unit determines the (i+1)th hidden layer feature information according to the input feature information X_i+1 and the ith hidden layer feature information h_i. In one example, the output trajectory information 603 can be determined based on the (i+1)th hidden layer feature information.

[0102] Figure 7 schematic diagram of a second sub-model according to another embodiment of the present disclosure

[0103] As shown in FIG. 7, the second sub-model 720 can include I LSTM units. In one example, the LSTM units are the deep learning units described above. I is an integer greater than or equal to i. In the present embodiment, I can be 4. Figure 7 For example, the sample trajectory information can include a plurality of geographic coordinates within a preset sample period. The sample trajectory information can include I sample trajectory sub-information, each corresponding to a plurality of geographic coordinates within a sample sub-period. Accordingly, for each sample sub-period of the target obstacle, the following operations can be performed: determining at least one target object having a distance from the target obstacle less than or equal to a preset distance threshold according to the position information of the target obstacle. Obtaining at least one edge relation information according to the target obstacle and the at least one target object. Obtaining an adjacency matrix corresponding to each sample sub-period according to the at least one edge relation information. In one example, for each sample sub-period, the position information of a time point in each sample sub-period is selected as the position information of the target obstacle in each sample sub-period.

[0104] In addition, feature extraction can be performed on each sample sub-information to obtain each semantic feature information. In one example, feature extraction can be performed on the first sample sub-information to obtain the first semantic feature information. Similarly, the second semantic feature information to the Ith semantic feature information can be obtained.

[0105] The first sub-model can process each semantic feature information based on the adjacency matrix corresponding to each sample sub-period to determine each intent information and each semantic relation information. In one example, the first sub-model can process the first semantic feature information based on the adjacency matrix corresponding to the first sample sub-period to determine the first intent information and the first semantic relation information. Similarly, the second intent information to the Ith intent information can be determined. Similarly, the second semantic relation information to the Ith semantic relation information can also be determined.

[0106]

[0107] ​For example, the sample trajectory information can be taken as the input feature information X_1 of the LSTM unit 721 to determine the first hidden layer feature information h_1. The first intention information and the first semantic relationship information are fused to obtain the first fusion information, which can be taken as the input feature information X_2 of the LSTM unit 722. The input feature information X_2 and the first hidden layer feature information h_1 are input into the LSTM unit 722 to determine the second hidden layer feature information h_2.

[0108] Next, the second intention information and the second semantic relationship information are fused to obtain the second fusion information, which can be taken as the input feature information X_3 of the LSTM unit 723. The input feature information X_3 and the second hidden layer feature information h_2 are input into the LSTM unit 723 to determine the third hidden layer feature information h_3. Similarly, the (I-1)th intention information and the (I-1)th semantic relationship information can be fused to obtain the (I-1)th fusion information, which can be taken as the input feature information X_I of the LSTM unit 724. The input feature information X_I and the (I-1)th hidden layer feature information are input into the LSTM unit 724 to determine the Ith hidden layer feature information h_I.

[0109] The output trajectory information can be determined according to the Ith hidden layer feature information h_I.

[0110] In another example, the output trajectory information can be determined according to the first hidden layer feature information h_1 to the Ith hidden layer feature information h_I.

[0111] In some embodiments, inputting the semantic feature information into the first sub-model to determine the intention information and the semantic relationship information further includes: inputting the semantic feature information into the first sub-model to determine sample intention information and sample semantic relationship information, wherein the sample trajectory information has an intention label and a semantic relationship label; determining a first loss value according to the sample intention information and the intention label; determining a second loss value according to the sample semantic relationship information and the semantic relationship label; and training the first sub-model according to the first loss value and the second loss value to obtain the pre-trained first sub-model. Details will be described below in combination with Figure 8 .

[0112] Figure 8 is a schematic diagram of a first sub-model according to an embodiment of the present disclosure.

[0113] As Figure 8 shown, the semantic feature information 802 is input into the first sub-model 810 to determine sample intention information 811 and sample semantic relationship information 812. For example, the semantic feature information 802 is obtained by feature extraction on sample trajectory information. The sample trajectory information has an intention label 813 and a semantic relationship label 814.

[0114] According to the sample intention information 811 and the intention label 813, a first loss value 815 is determined. According to the sample semantic relationship information 812 and the semantic relationship label 814, a second loss value 816 is determined. According to the first loss value 815 and the second loss value 816, the first sub-model 810 is trained to obtain a pre-trained first sub-model. In one example, the first loss value 815 and the second loss value 816 can be added to obtain a total loss value, and the first sub-model 810 is pre-trained according to the total loss value.

[0115] Figure 9 is a block diagram of a trajectory data processing apparatus according to one embodiment of the present disclosure.

[0116] As shown in Figure 9 the apparatus 900 can include a first determination module 910 and a second determination module 920.

[0117] The first determination module 910 is configured to determine intention information and semantic relationship information of a target obstacle according to current trajectory information of the target obstacle. For example, the semantic relationship information is used to represent a relationship between the target obstacle and at least one object.

[0118] The second determination module 920 is configured to determine target trajectory information of the target obstacle according to the current trajectory information, the intention information, and the semantic relationship information.

[0119] In some embodiments, the first determination module includes a first feature extraction sub-module configured to perform feature extraction on the current trajectory information to obtain semantic feature information, and a first determination sub-module configured to determine the intention information and the semantic relationship information of the target obstacle according to the semantic feature information.

[0120] In some embodiments, the first determination module includes a second determination sub-module configured to determine at least one target object having a distance less than or equal to a preset distance threshold from the target obstacle according to position information of the target obstacle, a first obtaining sub-module configured to obtain at least one edge relationship information according to the target obstacle and the at least one target object, a second obtaining sub-module configured to obtain an adjacency matrix according to the at least one edge relationship information, and a third determination sub-module configured to determine the intention information and the semantic relationship information according to the current trajectory information and the adjacency matrix.

[0121] In some embodiments, the second determining module comprises: a fourth determining submodule configured to determine hidden layer feature information according to the current trajectory information; a fifth determining submodule configured to determine input feature information according to the intention information and the semantic relationship information; and a sixth determining submodule configured to determine target trajectory information according to the hidden layer feature information and the input feature information.

[0122] Figure 10 is a block diagram of a training apparatus of a deep learning model according to another embodiment of the present disclosure.

[0123] As shown in Figure 10 the apparatus 1000 can include a third determining module 1010, a fourth determining module 1020, and a training module 1030.

[0124] The deep learning model comprises a first sub-model and a second sub-model.

[0125] The third determining module 1010 is configured to determine intention information and semantic relationship information of a target obstacle according to sample trajectory information of the target obstacle and the first sub-model. For example, the semantic relationship information is used to represent a relationship between the target obstacle and at least one object.

[0126] The fourth determining module 1020 is configured to input the sample trajectory information, the intention information, and the semantic relationship information into the second sub-model to determine output trajectory information of the target obstacle.

[0127] The training module 1030 is configured to train the deep learning model according to trajectory labels of the output trajectory information and the sample trajectory information.

[0128] In some embodiments, the third determining module comprises: a second feature extraction submodule configured to perform feature extraction on the sample trajectory information to obtain semantic feature information; and a seventh determining submodule configured to input the semantic feature information into the first sub-model to determine the intention information and the semantic relationship information.

[0129] In some embodiments, the third determining module comprises: an eighth determining submodule configured to determine at least one target object having a distance less than or equal to a preset distance threshold from the target obstacle according to position information of the target obstacle; a third obtaining submodule configured to obtain at least one edge relationship information according to the target obstacle and the at least one target object; a fourth obtaining submodule configured to obtain an adjacency matrix according to the at least one edge relationship information; and a ninth determining submodule configured to determine the intention information and the semantic relationship information according to the sample trajectory information and the first sub-model performing data processing based on the adjacency matrix.

[0130] In some embodiments, the second sub-model comprises a plurality of deep learning units, the fourth determining module comprises: a tenth determining sub-module, configured to input the sample trajectory information into an i-th deep learning unit of the second sub-model, and determine hidden layer feature information, where i is an integer greater than or equal to 1; an eleventh determining sub-module, configured to determine input feature information according to the intention information and the semantic relationship information; and a twelfth determining sub-module, configured to input the hidden layer feature information and the input feature information into an i+1-th deep learning unit, and determine output trajectory information.

[0131] In some embodiments, the training module comprises an adjusting sub-module, configured to adjust parameters of the first sub-model and the second sub-model according to the output trajectory information and the trajectory label, so as to train the deep learning model.

[0132] In some embodiments, the seventh determining sub-module further comprises: a first determining unit, configured to input the semantic feature information into the first sub-model, and determine sample intention information and sample semantic relationship information, where the sample trajectory information has an intention label and a semantic relationship label; a second determining unit, configured to determine a first loss value according to the sample intention information and the intention label; a third determining unit, configured to determine a second loss value according to the sample semantic relationship information and the semantic relationship label; and a training unit, configured to train the first sub-model according to the first loss value and the second loss value, to obtain a pre-trained first sub-model.

[0133] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution comply with relevant laws and regulations and do not violate public order and good customs.

[0134] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0135] Figure 11 An illustrative block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0136] like Figure 11 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded from storage unit 1108 into random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0137] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as trajectory data processing methods and / or deep learning model training methods. For example, in some embodiments, the trajectory data processing methods and / or deep learning model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the trajectory data processing methods and / or deep learning model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform trajectory data processing methods and / or deep learning model training methods.

[0139] Various implementations of the systems and techniques described above can be realized in a digital electronic circuit system, an integrated circuit system, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip system (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0140] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a function / operation specified in the flowchart and / or block diagram. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a remote machine or entirely on a remote machine or server.

[0141] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more of: an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0143] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0144] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0145] In some embodiments, the present disclosure also provides an autonomous vehicle, which includes the electronic device provided by the present disclosure. For example, the autonomous vehicle may, for example, include the electronic device 1100.

[0146] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from, without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

[0147] The above detailed description does not limit the scope of the disclosure. Various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments disclosed herein without departing from the spirit of the disclosure. Any modification, equivalent replacement or improvement made within the spirit and principle of the disclosure should be included within the scope of the disclosure.

Claims

1. A trajectory data processing method, comprising: Based on the current trajectory information of the target obstacle, the intention information and semantic relationship information of the target obstacle are determined, wherein the semantic relationship information is used to characterize the relationship between the target obstacle and at least one object; and Based on the current trajectory information, the intent information, and the semantic relationship information, the target trajectory information of the target obstacle is determined; The step of determining the target trajectory information based on the current trajectory information, the intent information, and the semantic relationship information includes: Based on the current trajectory information, determine the hidden layer feature information; Based on the intent information and the semantic relationship information, the input feature information is determined; and The target trajectory information is determined based on the hidden layer feature information and the input feature information.

2. The method according to claim 1, wherein, The step of determining the intent information and semantic relationship information of the target obstacle based on its current trajectory information includes: Feature extraction is performed on the current trajectory information to obtain semantic feature information; and Based on the semantic feature information, the intent information and semantic relationship information of the target obstacle are determined.

3. The method according to claim 1, wherein, The step of determining the intent information and semantic relationship information of the target obstacle based on its current trajectory information includes: Based on the location information of the target obstacle, determine at least one target object whose distance from the target obstacle is less than or equal to a preset distance threshold; Based on the target obstacle and the at least one target object, at least one edge relationship information is obtained; Based on the at least one edge relationship information, an adjacency matrix is ​​obtained; and Based on the current trajectory information and the adjacency matrix, the intent information and semantic relationship information are determined.

4. A method for training a deep learning model, the deep learning model comprising a first sub-model and a second sub-model, the method comprising: Based on the sample trajectory information of the target obstacle and the first sub-model, the intention information and semantic relationship information of the target obstacle are determined, wherein the semantic relationship information is used to characterize the relationship between the target obstacle and at least one object; The sample trajectory information, the intent information, and the semantic relationship information are input into the second sub-model to determine the output trajectory information of the target obstacle; and The deep learning model is trained based on the trajectory labels of the output trajectory information and the sample trajectory information. The second sub-model includes multiple deep learning units. The step of inputting the sample trajectory information, the intent information, and the semantic relationship information into the second sub-model to determine the output trajectory information includes: The sample trajectory information is input into the i-th deep learning unit of the second sub-model to determine the hidden layer feature information, where i is an integer greater than or equal to 1; Based on the intent information and the semantic relationship information, the input feature information is determined; and The hidden layer feature information and the input feature information are input into the (i+1)th deep learning unit to determine the output trajectory information.

5. The method according to claim 4, wherein, The step of determining the intent information and semantic relationship information of the target obstacle based on the sample trajectory information of the target obstacle and the first sub-model includes: Feature extraction is performed on the sample trajectory information to obtain semantic feature information; and The semantic feature information is input into the first sub-model to determine the intent information and the semantic relationship information.

6. The method according to claim 4, wherein, The step of determining the intent information and semantic relationship information of the target obstacle based on the sample trajectory information of the target obstacle and the first sub-model includes: Based on the location information of the target obstacle, determine at least one target object whose distance from the target obstacle is less than or equal to a preset distance threshold; Based on the target obstacle and the at least one target object, at least one edge relationship information is obtained; Based on the at least one edge relationship information, an adjacency matrix is ​​obtained; and The intent information and the semantic relationship information are determined based on the sample trajectory information and the first sub-model that performs data processing based on the adjacency matrix.

7. The method according to claim 4, wherein, Training the deep learning model based on the trajectory labels of the output trajectory information and the sample trajectory information includes: Based on the output trajectory information and the trajectory label, the parameters of the first sub-model and the second sub-model are adjusted to train the deep learning model.

8. The method according to claim 5, wherein, The step of inputting the semantic feature information into the first sub-model to determine the intent information and the semantic relationship information further includes: The semantic feature information is input into the first sub-model to determine the sample intent information and sample semantic relationship information, wherein the sample trajectory information has intent labels and semantic relationship labels; Based on the sample intent information and the intent label, a first loss value is determined; Based on the sample semantic relationship information and the semantic relationship label, a second loss value is determined; and Based on the first loss value and the second loss value, the first sub-model is trained to obtain the pre-trained first sub-model.

9. A trajectory data processing device, comprising: The first determining module is configured to determine the intent information and semantic relationship information of the target obstacle based on its current trajectory information, wherein the semantic relationship information is used to characterize the relationship between the target obstacle and at least one object; and The second determining module is used to determine the target trajectory information of the target obstacle based on the current trajectory information, the intent information, and the semantic relationship information. The second determining module includes: The fourth determining submodule is used to determine hidden layer feature information based on the current trajectory information; The fifth determining submodule is used to determine input feature information based on the intent information and the semantic relationship information; and The sixth determining submodule is used to determine the target trajectory information based on the hidden layer feature information and the input feature information.

10. The apparatus according to claim 9, wherein, The first determining module includes: The first feature extraction submodule is used to extract features from the current trajectory information to obtain semantic feature information; and The first determining submodule is used to determine the intent information and semantic relationship information of the target obstacle based on the semantic feature information.

11. The apparatus according to claim 9, wherein, The first determining module includes: The second determining submodule is used to determine at least one target object whose distance from the target obstacle is less than or equal to a preset distance threshold based on the location information of the target obstacle; The first acquisition submodule is used to obtain at least one edge relationship information based on the target obstacle and the at least one target object; The second obtaining submodule is used to obtain an adjacency matrix based on the at least one edge relationship information; and The third determining submodule is used to determine the intent information and semantic relationship information based on the current trajectory information and the adjacency matrix.

12. A training apparatus for a deep learning model, the deep learning model comprising a first sub-model and a second sub-model, the apparatus comprising: The third determining module is used to determine the intent information and semantic relationship information of the target obstacle based on the sample trajectory information of the target obstacle and the first sub-model, wherein the semantic relationship information is used to characterize the relationship between the target obstacle and at least one object; The fourth determining module is used to input the sample trajectory information, the intent information, and the semantic relationship information into the second sub-model to determine the output trajectory information of the target obstacle; and The training module is used to train the deep learning model based on the trajectory labels of the output trajectory information and the sample trajectory information; The second sub-model includes multiple deep learning units. The fourth determining module includes: The tenth determination submodule is used to input the sample trajectory information into the i-th deep learning unit of the second sub-model to determine the hidden layer feature information, where i is an integer greater than or equal to 1; The eleventh determining submodule is used to determine input feature information based on the intent information and the semantic relationship information; and The twelfth determination submodule is used to input the hidden layer feature information and the input feature information into the (i+1)th deep learning unit to determine the output trajectory information.

13. The apparatus according to claim 12, wherein, The third determining module includes: The second feature extraction submodule is used to extract features from the sample trajectory information to obtain semantic feature information; and The seventh determination submodule is used to input the semantic feature information into the first sub-model to determine the intent information and the semantic relationship information.

14. The apparatus according to claim 12, wherein, The third determining module includes: The eighth determination submodule is used to determine at least one target object whose distance from the target obstacle is less than or equal to a preset distance threshold based on the location information of the target obstacle; The third acquisition submodule is used to obtain at least one edge relationship information based on the target obstacle and the at least one target object; The fourth submodule is used to obtain an adjacency matrix based on the at least one edge relationship information; and The ninth determining submodule is used to determine the intent information and the semantic relationship information based on the sample trajectory information and the first sub-model that performs data processing based on the adjacency matrix.

15. The apparatus according to claim 12, wherein, The training module includes: The adjustment submodule is used to adjust the parameters of the first sub-model and the second sub-model according to the output trajectory information and the trajectory label, so as to train the deep learning model.

16. The apparatus according to claim 13, wherein, The seventh determining submodule also includes: The first determining unit is used to input the semantic feature information into the first sub-model to determine the sample intent information and the sample semantic relationship information, wherein the sample trajectory information has an intent label and a semantic relationship label; The second determining unit is used to determine a first loss value based on the sample intent information and the intent label; The third determining unit is configured to determine a second loss value based on the sample semantic relationship information and the semantic relationship label; and The training unit is used to train the first sub-model based on the first loss value and the second loss value to obtain the pre-trained first sub-model.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

20. An autonomous vehicle comprising the electronic device of claim 17.

Images