A trajectory prediction method supervised by cross-modal semantic information

Abstract

The present invention relates to the field of trajectory prediction technology, and in particular to a trajectory prediction method and device supervised by cross-modal semantic information. A trajectory prediction method supervised by cross-modal semantic information includes collecting vehicle information, driving information and map information to obtain semantic supervision information and future trajectory truth information; obtaining global background information features according to map features and historical trajectory features; performing mapping transformation according to future trajectory features to obtain predicted trajectory features closest to the truth value; performing comparative learning training on a semantically supervised trajectory prediction model to be trained according to semantic features, global background information features and predicted trajectory features closest to the truth value, obtaining a semantically supervised trajectory prediction model based on future trajectory truth information, and performing trajectory prediction according to the semantically supervised trajectory prediction model. The present invention is an accurate and efficient semantically supervised trajectory prediction method for scene semantic information.

Description

Technical Field

[0001] The present invention relates to the technical field of trajectory prediction, and in particular to a trajectory prediction method and device supervised by cross-modal semantic information. Background Art

[0002] Trajectory prediction is a technology used to predict the future position of a moving object. It is commonly used in fields such as autonomous driving, drones, robot navigation, and traffic flow prediction. In trajectory prediction, the object of interest must first be detected and identified, and its position must be tracked in consecutive frames. The development of object detection and tracking technology is crucial for trajectory prediction. There are uncertainties in trajectory prediction, including factors such as sensor noise, incomplete object occlusion, and future environmental changes. Effective uncertainty modeling can help evaluate the reliability of prediction results and provide a basis for decision making.

[0003] Trajectory prediction algorithms generally take road maps and the historical states of the agent in the scene as input. In order to encode the scene context information, early studies usually converted them into images so that they can be processed using convolutional neural networks. Due to the large amount of computation required for image information, later algorithms improved the encoding method of maps and historical states, using vectorized information instead of image information to represent context information. For example, a lane map is constructed to capture the map topology, or a graph structure is used to represent the map and historical trajectory information.

[0004] In existing methods, map information and vehicle trajectory information are generally used as supervision information of the model, and only the future trajectory information of the vehicle is used as the output of the constraint model to update the model parameters.

[0005] The label information is used as a supervisory signal, which only allows the model to fit the possible travel trajectory, but does not explicitly let the model understand the current vehicle type, driving distance, vehicle intention and other information, as well as the reasons why it drives in a specific way.

[0006] In the prior art, there is a lack of an accurate and efficient semantically supervised trajectory prediction method for scene semantic information. Summary of the invention

[0007] The embodiment of the present invention provides a trajectory prediction method and device supervised by cross-modal semantic information. The technical solution is as follows:

[0008] On the one hand, a trajectory prediction method supervised by cross-modal semantic information is provided, the method is implemented by an electronic device, and the method includes:

[0009] S1, collect vehicle information, driving information and map information, obtain semantic supervision information and future trajectory truth information;

[0010] S2. Build a model based on the encoder-decoder architecture to obtain the semantic supervision trajectory prediction model to be trained;

[0011] S3, performing feature vectorization on the semantic supervision information to obtain semantic features;

[0012] S4. According to the semantic supervision information, map features, historical trajectory features, and future trajectory features are obtained through the semantic supervision trajectory prediction model to be trained;

[0013] S5, performing feature splicing according to the map features and the historical trajectory features to obtain global background information features;

[0014] S6, performing mapping transformation according to the future trajectory feature to obtain a predicted trajectory feature closest to the true value;

[0015] S7, performing comparative learning training on the semantic supervised trajectory prediction model to be trained according to the semantic features, the global background information features and the predicted trajectory features closest to the true value, and obtaining a loss function convergence result based on the future trajectory true value information;

[0016] S8. When the loss function converges to a convergence result, a semantically supervised trajectory prediction model is obtained, and trajectory prediction is performed according to the semantically supervised trajectory prediction model; when the loss function converges to a non-convergence result, return to S1.

[0017] The semantic supervision information includes driving direction information, driving intention information, driving distance information and vehicle type information.

[0018] Optionally, performing feature stitching according to the map feature and the historical trajectory feature to obtain a global background information feature includes:

[0019] The map features and the historical trajectory features are combined to obtain overall information features;

[0020] Performing continuous vector mapping on the coordinates of the last moment of the historical trajectory feature to obtain the last moment coordinate vector; inputting the last moment coordinate vector into a preset attention module to obtain a query matrix;

[0021] According to the overall information characteristics, a key matrix and a value matrix are obtained;

[0022] Attention interaction is performed according to the query matrix, the key matrix and the value matrix to obtain global background information features.

[0023] Optionally, performing mapping transformation according to the future trajectory feature to obtain a predicted trajectory feature closest to a true value includes:

[0024] Performing dimensional mapping transformation according to the future trajectory features to obtain the vehicle driving trajectory coordinates;

[0025] Screening is performed according to the vehicle driving trajectory coordinates to obtain the predicted trajectory coordinates closest to the true value;

[0026] According to the predicted trajectory coordinates closest to the true value, a corresponding selection is made among the future trajectory features to obtain the predicted trajectory features closest to the true value.

[0027] Optionally, performing comparative learning training on the semantic supervised trajectory prediction model to be trained according to the semantic feature, the global background information feature and the predicted trajectory feature closest to the true value to obtain a loss function convergence result includes:

[0028] According to the semantic features and the global background information features, performing comparative learning training on the semantic supervised trajectory prediction model to be trained to obtain a first loss function convergence result;

[0029] According to the semantic features and the predicted trajectory features closest to the true value, performing comparative learning training on the semantic supervised trajectory prediction model to be trained to obtain a convergence result of the second loss function;

[0030] According to the future trajectory features, obtaining future trajectory coordinates; according to the future trajectory true value information, obtaining future trajectory true value coordinates; according to the future trajectory coordinates and the future trajectory true value coordinates, calculating the Gaussian mixture model loss, and obtaining a third loss function convergence result;

[0031] Obtaining a loss function convergence result according to the first loss function convergence result, the second loss function convergence result, and the third loss function convergence result;

[0032] The convergence result of the loss function includes convergence and non-convergence; wherein, convergence means when the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are converged at the same time; non-convergence means that the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are converged at different times.

[0033] On the other hand, a trajectory prediction device supervised by cross-modal semantic information is provided, and the device is applied to a trajectory prediction method supervised by cross-modal semantic information. The device includes:

[0034] The semantic supervision information acquisition module is used to collect vehicle information, driving information and map information, and obtain semantic supervision information and future trajectory truth information;

[0035] A model building module is used to build a model based on an encoder-decoder architecture to obtain a semantically supervised trajectory prediction model to be trained;

[0036] A feature vectorization module, used for performing feature vectorization on the semantic supervision information to obtain semantic features;

[0037] A feature extraction module, used to obtain map features, historical trajectory features and future trajectory features according to the semantic supervision information through the semantic supervision trajectory prediction model to be trained;

[0038] A global background information feature acquisition module, used for performing feature splicing according to the map features and the historical trajectory features to obtain global background information features;

[0039] A predicted trajectory feature acquisition module, used to perform mapping transformation according to the future trajectory feature to obtain the predicted trajectory feature closest to the true value;

[0040] A convergence result acquisition module is used to perform comparative learning training on the semantic supervised trajectory prediction model to be trained according to the semantic features, the global background information features and the predicted trajectory features closest to the true value, and obtain a loss function convergence result based on the future trajectory true value information;

[0041] The model acquisition module is used to obtain a semantically supervised trajectory prediction model when the loss function converges and perform trajectory prediction according to the semantically supervised trajectory prediction model; when the loss function converges and does not converge, return to the semantic supervision information acquisition module.

[0042] The semantic supervision information includes driving direction information, driving intention information, driving distance information and vehicle type information.

[0043] Optionally, the global background information feature acquisition module is further used to:

[0044] The map features and the historical trajectory features are combined to obtain overall information features;

[0045] Performing continuous vector mapping on the coordinates of the last moment of the historical trajectory feature to obtain the last moment coordinate vector; inputting the last moment coordinate vector into a preset attention module to obtain a query matrix;

[0046] According to the overall information characteristics, a key matrix and a value matrix are obtained;

[0047] Attention interaction is performed according to the query matrix, the key matrix and the value matrix to obtain global background information features.

[0048] Optionally, the predicted trajectory feature acquisition module is further used to:

[0049] Performing dimensional mapping transformation according to the future trajectory features to obtain the vehicle driving trajectory coordinates;

[0050] Screening is performed according to the vehicle driving trajectory coordinates to obtain the predicted trajectory coordinates closest to the true value;

[0051] According to the nearest trajectory coordinates, a corresponding selection is made among the future trajectory features to obtain a predicted trajectory feature closest to the true value.

[0052] Optionally, the convergence result acquisition module is further used to:

[0053] According to the semantic features and the global background information features, performing comparative learning training on the semantic supervised trajectory prediction model to be trained to obtain a first loss function convergence result;

[0054] According to the semantic features and the predicted trajectory features closest to the true value, performing comparative learning training on the semantic supervised trajectory prediction model to be trained to obtain a convergence result of the second loss function;

[0055] According to the future trajectory features, obtaining future trajectory coordinates; according to the future trajectory true value information, obtaining future trajectory true value coordinates; according to the future trajectory coordinates and the future trajectory true value coordinates, calculating the Gaussian mixture model loss, and obtaining a third loss function convergence result;

[0056] Obtaining a loss function convergence result according to the first loss function convergence result, the second loss function convergence result, and the third loss function convergence result;

[0057] The convergence result of the loss function includes convergence and non-convergence; wherein, convergence means when the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are converged at the same time; non-convergence means that the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are converged at different times.

[0058] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned trajectory prediction method supervised by cross-modal semantic information.

[0059] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned trajectory prediction method supervised by cross-modal semantic information.

[0060] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0061] The present invention proposes a trajectory prediction method supervised by cross-modal semantic information. Based on the semantic supervision learning method of text information, additional semantic information is introduced, and information such as vehicle type, driving distance, vehicle intention, map information, etc. is extracted into features by text encoding, and the model is supervised and prompted, so that the model can better understand the context and background of the driving task, help the model to more accurately predict the future actions of the vehicle, more accurately predict the path, reduce the prediction error of the model, and reduce the training time of the model; the present invention adopts a semantic supervision learning method so that the model does not need to rely on a large number of annotated data sets, but automatically generates supervision signals by using existing information such as vehicles, trajectories, and maps. This not only improves data utilization, but also reduces the cost of obtaining supervision information and reduces dependence on manual annotation. The semantic supervision learning method proposed by the present invention is not restricted by a specific trajectory prediction algorithm, and can be used in most trajectory prediction methods, bringing improvement to the trajectory prediction algorithm. The present invention is an accurate and efficient semantic supervision trajectory prediction method for scene semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 It is a flow chart of a trajectory prediction method supervised by cross-modal semantic information provided by an embodiment of the present invention;

[0064] Figure 2 It is a block diagram of a trajectory prediction device supervised by cross-modal semantic information provided by an embodiment of the present invention;

[0065] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0067] The embodiment of the present invention provides a trajectory prediction method supervised by cross-modal semantic information, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1The flowchart of a trajectory prediction method supervised by cross-modal semantic information is shown in FIG. 1 . The processing flow of the method may include the following steps:

[0068] S1. Collect vehicle information, driving information and map information to obtain semantic supervision information and future trajectory truth information.

[0069] Among them, the semantic supervision information includes driving direction information, driving intention information, driving distance information and vehicle type information.

[0070] In a feasible implementation manner, the semantic information data used in the present invention includes but is not limited to driving direction information, driving intention information, driving distance information, and vehicle type information.

[0071] Among them, it includes but is not limited to generating driving direction information based on the angle difference between the true value of the vehicle's historical trajectory and the future trajectory; generating driving intention information based on the angle difference between the true value of the vehicle's historical trajectory and the future trajectory and map information; generating driving distance information based on the vehicle's driving distance; generating vehicle type information based on the type of predicted trajectory intelligent body.

[0072] Future trajectory truth information refers to the actual or true information about the future trajectory corresponding to the past observed trajectory data in the trajectory prediction task. "Future" refers to the time period after the current moment, which can be a few seconds, minutes, or even longer. Future trajectory truth information is a benchmark for evaluating the performance of prediction models because it provides a comparison between the model's prediction results and the actual situation.

[0073] Typically, the future trajectory truth information consists of the real evolution data of the actually observed trajectory data in the future time period.

[0074] S2. Build a model based on the encoder-decoder architecture to obtain the semantically supervised trajectory prediction model to be trained.

[0075] In a feasible implementation manner, the semantically supervised trajectory prediction model to be trained adopted by the present invention is not limited to the model of the encoder-decoder architecture, and any trajectory prediction model of an intelligent agent in the prior art can also be adopted.

[0076] S3. Vectorize the semantic supervision information to obtain semantic features.

[0077] In a feasible implementation, the semantic supervision information is expressed as a language text in the form of a template; the language text is sent to an arbitrary text encoder to map the text information into a semantic feature of dimension D, and the semantic feature is represented by T.

[0078] S4. According to the semantic supervision information, map features, historical trajectory features and future trajectory features are obtained through the semantic supervision trajectory prediction model to be trained.

[0079] In a feasible implementation, the trajectory is predicted by using a semantically supervised trajectory prediction model to be trained, and features are extracted from the map and historical trajectories, and feature mapping is performed on future trajectories.

[0080] The map feature is represented as M1 with dimension [A, M, D]; the historical trajectory feature is represented as H1 with dimension [A, H, D], where A represents the number of surrounding vehicles; the predicted future trajectory is mapped into the future trajectory feature with dimension [A, K, D].

[0081] S5. Perform feature splicing based on map features and historical trajectory features to obtain global background information features.

[0082] Optionally, feature splicing is performed based on map features and historical trajectory features to obtain global background information features, including:

[0083] Combine map features and historical trajectory features to obtain overall information features;

[0084] Perform continuous vector mapping on the coordinates of the last moment of the historical trajectory feature to obtain the coordinate vector of the last moment; input the coordinate vector of the last moment into the preset attention module to obtain the query matrix;

[0085] According to the overall information characteristics, obtain the key matrix and value matrix;

[0086] Attention interaction is performed based on the query matrix, key matrix, and value matrix to obtain global background information features.

[0087] In a feasible implementation manner, in the present invention, the map feature M1 and the historical trajectory feature H1 are spliced ​​to synthesize the overall information feature C1, and the overall information feature dimension is [A, M+H, D].

[0088] In order to obtain features that can better represent future trajectories, a decoding structure is designed here, which uses the coordinates of the last moment of the vehicle's driving history in the semantic supervision information for encoding initialization.

[0089] Decompose the coordinates of the last moment of the input into continuous vector representations, and use the obtained vector matrix as the query matrix.

[0090] Then, according to the overall information feature C1, the key matrix and value matrix are obtained. The query matrix, key matrix and value matrix are interactively paid attention to, and finally the global background information feature F1 is obtained. The dimension of the global matrix F1 is [A, D].

[0091] S6. Perform mapping transformation according to the future trajectory features to obtain the predicted trajectory features closest to the true value.

[0092] Optionally, mapping transformation is performed according to the future trajectory features to obtain predicted trajectory features closest to the true value, including:

[0093] Perform dimension mapping transformation according to the future trajectory characteristics to obtain the vehicle's driving trajectory coordinates;

[0094] Filter according to the vehicle's driving trajectory coordinates to obtain the predicted trajectory coordinates closest to the true value;

[0095] According to the predicted trajectory coordinates closest to the true value, a corresponding selection is made among the future trajectory features to obtain the predicted trajectory features closest to the true value.

[0096] In a feasible implementation, feature mapping is performed based on the future trajectory features obtained in the above steps, and the future trajectory features of [A, K, D] dimensions are mapped into trajectory coordinates of [A, K, T, E]. We denote the feature [A, D] corresponding to the trajectory closest to the true value among the K predicted future trajectories as the predicted trajectory feature F2 closest to the true value.

[0097] Among them, K represents K possible future trajectories, D is the feature dimension, T represents the number of future time points, and E represents the coordinates x and y corresponding to the moving vehicle.

[0098] S7. According to the semantic features, global background information features and the predicted trajectory features closest to the true value, the semantic supervised trajectory prediction model to be trained is subjected to comparative learning training and supervised training, and the loss function convergence result is obtained based on the future trajectory true value information.

[0099] Optionally, the semantic supervised trajectory prediction model to be trained is subjected to comparative learning training according to the semantic features, the global background information features and the predicted trajectory features closest to the true value to obtain the loss function convergence result, including:

[0100] According to the semantic features and global background information features, the semantic supervised trajectory prediction model to be trained is trained by contrastive learning to obtain the convergence result of the first loss function;

[0101] According to the semantic features and the predicted trajectory features closest to the true value, the semantic supervised trajectory prediction model to be trained is trained by contrastive learning to obtain the convergence result of the second loss function;

[0102] According to the future trajectory characteristics, the future trajectory coordinates are obtained; according to the future trajectory true value information, the future trajectory true value coordinates are obtained; according to the future trajectory coordinates and the future trajectory true value coordinates, the Gaussian mixture model loss is calculated to obtain the convergence result of the third loss function;

[0103] Obtaining a loss function convergence result according to the first loss function convergence result, the second loss function convergence result, and the third loss function convergence result;

[0104] The convergence results of the loss function include convergence and non-convergence; wherein, convergence means when the convergence results of the first loss function, the convergence results of the second loss function and the convergence results of the third loss function are converged at the same time; non-convergence means that the convergence results of the first loss function, the convergence results of the second loss function and the convergence results of the third loss function do not converge at the same time.

[0105] In a feasible implementation manner, the design of the loss function in the present invention includes the design of the first loss function, the design of the second loss function and the design of the third loss function, and the specific design is as follows:

[0106] Contrastive loss between semantic feature T and global background information feature F1:

[0107] L_gloabl=(KL_div(logits_t,label)+KL_div(logits_f1,label)) / 2

[0108] logits_t=dot(T,F1)[0]

[0109] logits_f1=dot(T,F1)[1]

[0110] logits_t represents the similarity between F1 and T using text feature T as the label, label represents the true label, and KL_div represents the use of KL divergence as the loss function to measure the loss.

[0111] logits_f1 represents the similarity between the semantic feature T and the global background information feature F1 using the global background information feature F1 as the label, label represents the true label, and KL_div represents the use of KL divergence as the loss function.

[0112] The contrast loss between the semantic feature T and the nearest predicted trajectory feature F2 is:

[0113] L_proposal=(KL_div(logits_t,label)+KL_div(logits_f2,label)) / 2

[0114] logits_t=dot(T,F2)[0]

[0115] logits_f2=dot(T,F2)[1]

[0116] logits_t represents the similarity between F2 and T using text feature T as the label, label represents the true label, and KL_div represents the use of KL divergence as the loss function to measure the loss.

[0117] logits_f1 represents the similarity between the semantic feature T and the global background information feature F2 using the global background information feature F2 as the label, label represents the true label, and KL_div represents the use of KL divergence as the loss function.

[0118] Gaussian mixture model loss of predicted future trajectory coordinates and future trajectory true value coordinates:

[0119] L_gmm=GMM_loss(pred,label)

[0120] pred is the future trajectory coordinate predicted by the model, label is the true value coordinate of the future trajectory, and GMM_loss represents the Gaussian mixture model loss function.

[0121] S8. When the loss function converges to convergence, a semantically supervised trajectory prediction model is obtained, and trajectory prediction is performed according to the semantically supervised trajectory prediction model; when the loss function converges to non-convergence, return to S1.

[0122] In a feasible implementation, when the loss function converges to convergence, a semantically supervised trajectory prediction model is obtained. When performing trajectory prediction according to the semantically supervised trajectory prediction model described in the present invention, richer semantic information is introduced, which can better understand the semantic information in the scene and make future predictions more accurate. Based on the existing experimental data, the present invention can reduce the number of training rounds by 20%.

[0123] The present invention proposes a trajectory prediction method supervised by cross-modal semantic information. Based on the semantic supervision learning method of text information, additional semantic information is introduced, and information such as vehicle type, driving distance, vehicle intention, map information, etc. is extracted into features by text encoding, and the model is supervised and prompted, so that the model can better understand the context and background of the driving task, help the model to more accurately predict the future actions of the vehicle, more accurately predict the path, reduce the prediction error of the model, and reduce the training time of the model; the present invention adopts a semantic supervision learning method so that the model does not need to rely on a large number of annotated data sets, but automatically generates supervision signals by using existing information such as vehicles, trajectories, and maps. This not only improves data utilization, but also reduces the cost of obtaining supervision information and reduces dependence on manual annotation. The semantic supervision learning method proposed by the present invention is not restricted by a specific trajectory prediction algorithm, and can be used in most trajectory prediction methods, bringing improvement to the trajectory prediction algorithm. The present invention is an accurate and efficient semantic supervision trajectory prediction method for scene semantic information.

[0124] Figure 2 is a block diagram of a trajectory prediction device supervised by cross-modal semantic information according to an exemplary embodiment. Figure 2 , the device comprises:

[0125] The semantic supervision information acquisition module 210 is used to collect vehicle information, driving information and map information, and obtain semantic supervision information and future trajectory truth information;

[0126] A model building module 220 is used to build a model based on an encoder-decoder architecture to obtain a semantically supervised trajectory prediction model to be trained;

[0127] A feature vectorization module 230 is used to perform feature vectorization on the semantic supervision information to obtain semantic features;

[0128] A feature extraction module 240 is used to obtain map features, historical trajectory features, and future trajectory features according to the semantic supervision information through the semantic supervision trajectory prediction model to be trained;

[0129] A global background information feature acquisition module 250 is used to perform feature splicing based on map features and historical trajectory features to obtain global background information features;

[0130] The predicted trajectory feature acquisition module 260 is used to perform mapping transformation according to the future trajectory feature to obtain the predicted trajectory feature closest to the true value;

[0131] A convergence result acquisition module 270 is used to perform comparative learning training on the semantic supervised trajectory prediction model to be trained according to the semantic features, the global background information features and the predicted trajectory features closest to the true value, and obtain the loss function convergence result based on the future trajectory true value information;

[0132] The model acquisition module 280 is used to obtain the semantic supervision trajectory prediction model when the loss function converges to convergence, and perform trajectory prediction according to the semantic supervision trajectory prediction model; when the loss function converges to non-convergence, return to the semantic supervision information acquisition module.

[0133] Among them, the semantic supervision information includes driving direction information, driving intention information, driving distance information and vehicle type information.

[0134] Optionally, the global background information feature acquisition module 250 is further used to:

[0135] Combine map features and historical trajectory features to obtain overall information features;

[0136] Perform continuous vector mapping on the coordinates of the last moment of the historical trajectory feature to obtain the coordinate vector of the last moment; input the coordinate vector of the last moment into the preset attention module to obtain the query matrix;

[0137] According to the overall information characteristics, obtain the key matrix and value matrix;

[0138] Attention interaction is performed based on the query matrix, key matrix, and value matrix to obtain global background information features.

[0139] Optionally, the predicted trajectory feature acquisition module 260 is further used to:

[0140] Perform dimension mapping transformation according to the future trajectory characteristics to obtain the vehicle's driving trajectory coordinates;

[0141] Filter according to the vehicle's driving trajectory coordinates to obtain the predicted trajectory coordinates closest to the true value;

[0142] According to the coordinates of the nearest trajectory, a corresponding selection is made among the future trajectory features to obtain the predicted trajectory features closest to the true value.

[0143] Optionally, the convergence result acquisition module 270 is further used to:

[0144] According to the semantic features and global background information features, the semantic supervised trajectory prediction model to be trained is trained by contrastive learning to obtain the convergence result of the first loss function;

[0145] According to the semantic features and the predicted trajectory features closest to the true value, the semantic supervised trajectory prediction model to be trained is trained by contrastive learning to obtain the convergence result of the second loss function;

[0146] According to the future trajectory characteristics, the future trajectory coordinates are obtained; according to the future trajectory true value information, the future trajectory true value coordinates are obtained; according to the future trajectory coordinates and the future trajectory true value coordinates, the Gaussian mixture model loss is calculated to obtain the convergence result of the third loss function;

[0147] Obtaining a loss function convergence result according to the first loss function convergence result, the second loss function convergence result, and the third loss function convergence result;

[0148] The convergence results of the loss function include convergence and non-convergence; among them, convergence means when the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are converged at the same time; non-convergence means that the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function do not converge at the same time.

[0149] The present invention proposes a trajectory prediction method supervised by cross-modal semantic information. Based on the semantic supervision learning method of text information, additional semantic information is introduced, and information such as vehicle type, driving distance, vehicle intention, map information, etc. is extracted into features by text encoding, and the model is supervised and prompted, so that the model can better understand the context and background of the driving task, help the model to more accurately predict the future actions of the vehicle, more accurately predict the path, reduce the prediction error of the model, and reduce the training time of the model; the present invention adopts a semantic supervision learning method so that the model does not need to rely on a large number of annotated data sets, but automatically generates supervision signals by using existing information such as vehicles, trajectories, and maps. This not only improves data utilization, but also reduces the cost of obtaining supervision information and reduces dependence on manual annotation. The semantic supervision learning method proposed by the present invention is not restricted by a specific trajectory prediction algorithm, and can be used in most trajectory prediction methods, bringing improvement to the trajectory prediction algorithm. The present invention is an accurate and efficient semantic supervision trajectory prediction method for scene semantic information.

[0150] Figure 3 It is a structural diagram of an electronic device 300 provided in an embodiment of the present invention. The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 301 and one or more memories 302, wherein the memory 302 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 301 to implement the steps of the above-mentioned trajectory prediction method supervised by cross-modal semantic information.

[0151] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned cross-modal semantic information supervised trajectory prediction method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a tape, a floppy disk, and an optical data storage device.

[0152] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A trajectory prediction method supervised by cross-modal semantic information, characterized in that: The method comprises: S1, collect vehicle information, driving information and map information, obtain semantic supervision information and future trajectory truth information; The semantic supervision information includes driving direction information, driving intention information, driving distance information and vehicle type information; The driving direction information is generated based on the angle difference between the historical trajectory information of the vehicle and the true value information of the future trajectory; the driving intention information is generated based on the driving direction information and the map information; the driving distance information is formed based on the driving distance of the vehicle; S2. Build a model based on the encoder-decoder architecture to obtain the semantic supervision trajectory prediction model to be trained; S3, performing feature vectorization on the semantic supervision information to obtain semantic features; S4. According to the semantic supervision information, map features, historical trajectory features, and future trajectory features are obtained through the semantic supervision trajectory prediction model to be trained; S5, performing feature splicing according to the map features and the historical trajectory features to obtain global background information features; S6, performing mapping transformation according to the future trajectory feature to obtain a predicted trajectory feature closest to the true value; S7, performing comparative learning training on the semantic supervised trajectory prediction model to be trained according to the semantic features, the global background information features and the predicted trajectory features closest to the true value, and obtaining a loss function convergence result based on the future trajectory true value information; The step of performing comparative learning training on the semantic supervised trajectory prediction model to be trained based on the semantic features, the global background information features and the predicted trajectory features closest to the true value, and obtaining a loss function convergence result based on the future trajectory true value information includes: According to the semantic features and the global background information features, performing comparative learning training on the semantic supervised trajectory prediction model to be trained to obtain a first loss function convergence result; According to the semantic features and the predicted trajectory features closest to the true value, performing comparative learning training on the semantic supervised trajectory prediction model to be trained to obtain a convergence result of the second loss function; According to the future trajectory features, obtaining future trajectory coordinates; according to the future trajectory true value information, obtaining future trajectory true value coordinates; according to the future trajectory coordinates and the future trajectory true value coordinates, calculating the Gaussian mixture model loss, and obtaining a third loss function convergence result; Obtaining a loss function convergence result according to the first loss function convergence result, the second loss function convergence result, and the third loss function convergence result; The design of the loss function includes the design of the first loss function, the design of the second loss function and the design of the third loss function. The specific designs are as follows: Contrastive loss between semantic feature T and global background information feature F1: L_gloabl=(KL_div(logits_t,label)+KL_div(logits_f1,label)) / 2 logits_t=dot(T,F1)[0] logits_f1=dot(T,F1)[1] logits_t means the similarity between F1 and T with text feature T as label, label means the real label, KL_div means using KL divergence as loss function to measure loss; logits_f1 represents the similarity between the semantic feature T and the global background information feature F1 using the global background information feature F1 as the label, label represents the true label, and KL_div represents the use of KL divergence as the loss function; The contrast loss between the semantic feature T and the nearest predicted trajectory feature F2 is: L_proposal=(KL_div(logits_t,label)+KL_div(logits_f2,label)) / 2 logits_t=dot(T,F2)[0] logits_f2=dot(T,F2)[1] logits_t means taking text feature T as label, the similarity between F2 and T, label means the real label, KL_div means using KL divergence as loss function to measure loss; logits_f1 represents the similarity between the semantic feature T and the global background information feature F2 using the global background information feature F2 as the label, label represents the true label, and KL_div represents the use of KL divergence as the loss function; Gaussian mixture model loss of predicted future trajectory coordinates and future trajectory true value coordinates: L_gmm=GMM_loss(pred,label) pred is the future trajectory coordinates predicted by the model, label is the true value coordinates of the future trajectory, and GMM_loss represents the Gaussian mixture model loss function; The convergence result of the loss function includes convergence and non-convergence; wherein, convergence means when the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are converged at the same time; non-convergence means when the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are different, it is converged; S8. When the loss function converges to a convergence result, a semantically supervised trajectory prediction model is obtained, and trajectory prediction is performed according to the semantically supervised trajectory prediction model; when the loss function converges to a non-convergence result, return to S1.

2. According to claim 1, a trajectory prediction method supervised by cross-modal semantic information is characterized in that: The feature splicing is performed according to the map features and the historical trajectory features, Obtain global background information features, including: The map features and the historical trajectory features are combined to obtain overall information features; Performing continuous vector mapping on the coordinates of the last moment of the historical trajectory feature to obtain the last moment coordinate vector; inputting the last moment coordinate vector into a preset attention module to obtain a query matrix; According to the overall information characteristics, a key matrix and a value matrix are obtained; Attention interaction is performed according to the query matrix, the key matrix and the value matrix to obtain global background information features.

3. The trajectory prediction method supervised by cross-modal semantic information according to claim 1, characterized in that: The mapping transformation is performed according to the future trajectory feature to obtain the predicted trajectory feature closest to the true value, including: Performing dimensional mapping transformation according to the future trajectory features to obtain the vehicle driving trajectory coordinates; Screening is performed according to the vehicle driving trajectory coordinates to obtain the predicted trajectory coordinates closest to the true value; According to the predicted trajectory coordinates closest to the true value, a corresponding selection is made among the future trajectory features to obtain the predicted trajectory features closest to the true value.

4. A trajectory prediction device supervised by cross-modal semantic information, characterized in that: The device comprises: The semantic supervision information acquisition module is used to collect vehicle information, driving information and map information, and obtain semantic supervision information and future trajectory truth information; The semantic supervision information includes driving direction information, driving intention information, driving distance information and vehicle type information; The driving direction information is generated based on the angle difference between the historical trajectory information of the vehicle and the true value information of the future trajectory; the driving intention information is generated based on the driving direction information and the map information; the driving distance information is formed based on the driving distance of the vehicle; A model building module is used to build a model based on an encoder-decoder architecture to obtain a semantically supervised trajectory prediction model to be trained; A feature vectorization module, used for performing feature vectorization on the semantic supervision information to obtain semantic features; A feature extraction module, used to obtain map features, historical trajectory features and future trajectory features according to the semantic supervision information through the semantic supervision trajectory prediction model to be trained; A global background information feature acquisition module, used for performing feature splicing according to the map features and the historical trajectory features to obtain global background information features; A predicted trajectory feature acquisition module, used to perform mapping transformation according to the future trajectory feature to obtain the predicted trajectory feature closest to the true value; A convergence result acquisition module is used to perform comparative learning training on the semantic supervised trajectory prediction model to be trained according to the semantic features, the global background information features and the predicted trajectory features closest to the true value, and obtain a loss function convergence result based on the future trajectory true value information; The step of performing comparative learning training on the semantic supervised trajectory prediction model to be trained based on the semantic features, the global background information features and the predicted trajectory features closest to the true value, and obtaining a loss function convergence result based on the future trajectory true value information includes: According to the semantic features and the global background information features, performing comparative learning training on the semantic supervised trajectory prediction model to be trained to obtain a first loss function convergence result; According to the semantic features and the predicted trajectory features closest to the true value, performing comparative learning training on the semantic supervised trajectory prediction model to be trained to obtain a convergence result of the second loss function; According to the future trajectory features, obtaining future trajectory coordinates; according to the future trajectory true value information, obtaining future trajectory true value coordinates; according to the future trajectory coordinates and the future trajectory true value coordinates, calculating the Gaussian mixture model loss, and obtaining a third loss function convergence result; Obtaining a loss function convergence result according to the first loss function convergence result, the second loss function convergence result, and the third loss function convergence result; The design of the loss function includes the design of the first loss function, the design of the second loss function and the design of the third loss function. The specific designs are as follows: Contrastive loss between semantic feature T and global background information feature F1: L_gloabl=(KL_div(logits_t,label)+KL_div(logits_f1,label)) / 2 logits_t=dot(T,F1)[0] logits_f1=dot(T,F1)[1] logits_t means the similarity between F1 and T with text feature T as label, label means the real label, KL_div means using KL divergence as loss function to measure loss; logits_f1 represents the similarity between the semantic feature T and the global background information feature F1 using the global background information feature F1 as the label, label represents the true label, and KL_div represents the use of KL divergence as the loss function; The contrast loss between the semantic feature T and the nearest predicted trajectory feature F2 is: L_proposal=(KL_div(logits_t,label)+KL_div(logits_f2,label)) / 2 logits_t=dot(T,F2)[0] logits_f2=dot(T,F2)[1] logits_t means taking text feature T as label, the similarity between F2 and T, label means the real label, KL_div means using KL divergence as loss function to measure loss; logits_f1 represents the similarity between the semantic feature T and the global background information feature F2 using the global background information feature F2 as the label, label represents the true label, and KL_div represents the use of KL divergence as the loss function; Gaussian mixture model loss of predicted future trajectory coordinates and future trajectory true value coordinates: L_gmm=GMM_loss(pred,label) pred is the future trajectory coordinates predicted by the model, label is the true value coordinates of the future trajectory, and GMM_loss represents the Gaussian mixture model loss function; The convergence result of the loss function includes convergence and non-convergence; wherein, convergence means when the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are converged at the same time; non-convergence means when the convergence result of the first loss function, the convergence result of the second loss function and the convergence result of the third loss function are different, it is converged; The model acquisition module is used to obtain a semantically supervised trajectory prediction model when the loss function converges and perform trajectory prediction according to the semantically supervised trajectory prediction model; when the loss function converges and does not converge, return to the semantic supervision information acquisition module.

5. The trajectory prediction device supervised by cross-modal semantic information according to claim 4, characterized in that: The global background information feature acquisition module is further used to: The map features and the historical trajectory features are combined to obtain overall information features; Performing continuous vector mapping on the coordinates of the last moment of the historical trajectory feature to obtain the last moment coordinate vector; inputting the last moment coordinate vector into a preset attention module to obtain a query matrix; According to the overall information characteristics, a key matrix and a value matrix are obtained; Attention interaction is performed according to the query matrix, the key matrix and the value matrix to obtain global background information features.

6. The trajectory prediction device supervised by cross-modal semantic information according to claim 4, characterized in that: The predicted trajectory feature acquisition module is further used to: Performing dimensional mapping transformation according to the future trajectory features to obtain the vehicle driving trajectory coordinates; Screening is performed according to the vehicle driving trajectory coordinates to obtain the predicted trajectory coordinates closest to the true value; According to the nearest trajectory coordinates, a corresponding selection is made among the future trajectory features to obtain a predicted trajectory feature closest to the true value.

Images