A model training processing method and device

Abstract

Embodiments of the present application relate to a kind of model training processing method and device, the method comprises: obtaining high-precision map construction first training dataset;According to the first training dataset, the map network pre-training of map network module of LaneGCN model is carried out;Map network pre-training succeeds, then obtains vehicle road test data and constructs second training dataset;On the premise of keeping the network parameters of map network module unchanged, according to second training dataset, model training is carried out to LaneGCN model.By the present application, the training difficulty of overall model can be reduced, the training maturity of map network module is improved, and the output precision of LaneGCN model is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for model training. Background Technology

[0002] Predicting changes in the dynamic and static environment surrounding the vehicle and the behavioral evolution of other traffic participants is an essential capability for autonomous vehicles to achieve high levels of intelligent driving. Current mainstream autonomous driving prediction algorithms are implemented using deep learning models. These models take high-precision map information and observational information of traffic participants as input, and output information about the future movement trajectories of these participants. LaneGCN is one such model.

[0003] The structure of the LaneGCN model, as described in the paper "Learning Lane Graph Representations for Motion Forecasting", includes: ActorNet, MapNet, FusionNet, and Header. The Header includes a regression branch and a classification branch. The LaneGCN model comprises several modules: a participant network module for extracting participant features from the differential trajectory sequences of M traffic participants (also known as obstacles, obstacle targets, or targets); a map network module for extracting map lane features from the N lane node features; a fusion network module for fusing participant and lane features using four fusion units (participant-to-lane A2L, lane-to-lane L2L, lane-to-participant L2A, and participant-to-participant A2A), with the A2A unit outputting the participant-to-participant fused features; and a prediction head network module for predicting the future K-modal (K possible) trajectories of the m-th (1≤m≤M) participant based on the participant-to-participant fused features, and outputting the corresponding confidence score for the k-th (1≤m≤K) predicted trajectory. The regression branch of the prediction head network module is used for multimodal trajectory prediction, while the classification branch is used for multimodal confidence classification. The paper provides a training method based on the loss function L... mod =L cls +αL reg The training method for self-supervised learning is the inverse modulation method.

[0004] However, in practical applications, we have found that based solely on the aforementioned loss function L... modIt is difficult to fully train the map network module. Since the map network module is a prior module of the fusion network module, the sufficiency of training the map network module directly affects the fusion effect of the fusion network module, and the fusion effect of the fusion network module directly affects the prediction accuracy of the prediction head network module. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a method, apparatus, electronic device, and computer-readable storage medium for model training. This method involves first training a high-precision map dataset and a loss function L before performing overall model training on the LaneGCN model. map The map network module is pre-trained to achieve sufficient training results; then, while keeping the network parameters of the map network module unchanged, the loss function L is applied. mod =L cls +αL reg The LaneGCN model is trained as a whole. This invention reduces the overall training difficulty of the model, improves the training maturity of the map network module, and enhances the output accuracy of the LaneGCN model.

[0006] To achieve the above objectives, a first aspect of the present invention provides a method for model training, the method comprising:

[0007] Obtain high-precision maps to construct the first training dataset;

[0008] The map network module of the LaneGCN model is pre-trained based on the first training dataset.

[0009] If the map network is successfully pre-trained, then vehicle road test data is acquired to construct a second training dataset.

[0010] While keeping the network parameters of the map network module unchanged, the LaneGCN model is trained based on the second training dataset.

[0011] Preferably, the LaneGCN model includes a participant network module, a map network module, a fusion network module, and a prediction head network module; the fusion network module is connected to the participant network module, the map network module, and the prediction head network module, respectively.

[0012] The participant network module is used to perform participant feature extraction processing on the first input tensor of shape M*3*T1 to generate a corresponding first output tensor of shape M*128; and send the first output tensor to the fusion network module; the first input tensor includes M participant observation tensors of shape 3*T1; M is the total number of participants; T1 is the preset number of first time moments, which is 20 by default;

[0013] The map network module is used to perform map lane feature extraction processing on the second input tensor of shape N*4 to generate a corresponding second output tensor of shape N*128; and send the second output tensor to the fusion network module; the second input tensor includes N lane node vectors of shape 1*4; N is the total number of lane node vectors;

[0014] The fusion network module is used to perform feature fusion processing on the first output tensor and the second output tensor to generate a first fused feature tensor with a corresponding shape of M*128; and send the first fused feature tensor to the prediction head network module.

[0015] The prediction head network module includes a regression branch and a classification branch; the regression branch is used to perform multimodal trajectory prediction processing based on the first fusion feature tensor to generate a first prediction tensor with a corresponding shape of M*K*T2*2; the classification branch is used to perform multimodal confidence classification processing based on the first fusion feature tensor to generate a first confidence tensor with a corresponding shape of M*K; K is the total number of modes; T2 is the preset number of second time points, which defaults to 30.

[0016] The first prediction tensor includes M first participant prediction tensors of shape K*T2*2; each first participant prediction tensor corresponds one-to-one with the M participants; each first participant prediction tensor includes K first prediction trajectory tensors of shape T2*2; each first prediction trajectory tensor includes a first trajectory point vector of the second time step number T2; the first trajectory point vector is a two-dimensional trajectory point coordinate.

[0017] The first confidence tensor includes M first confidence vectors of length K; each first confidence vector corresponds one-to-one with one of the M participants; each first confidence vector includes K first confidence values; each first confidence value corresponds one-to-one with the first predicted trajectory tensor, and the first confidence value is the prediction confidence value of the corresponding first predicted trajectory tensor.

[0018] Preferably, the step of acquiring the high-precision map to construct the first training dataset specifically includes:

[0019] The first scene map set is composed of high-precision map raw data selected from a publicly available high-precision map database according to preset map filtering rules. The map filtering rules include minimum map quantity, minimum map size, maximum map accuracy, minimum number of lanes, minimum lane length, and lane segment length. The first scene map set includes multiple first scene maps. The number of first scene maps is not greater than the maximum number of maps. The map size of the first scene map is not less than the minimum map size, and the map accuracy is not less than the minimum map accuracy. The first scene map includes multiple first lanes, the number of first lanes is not less than the minimum number of lanes, and the lane length is not less than the minimum lane length. Each first lane includes a first lane centerline.

[0020] According to the lane segment length of the map filtering rules, each first lane of each first scene map is segmented to obtain a plurality of corresponding first lane segments; and the two intersection points of each first lane segment and the corresponding first lane centerline are recorded as the corresponding first starting point and first ending point;

[0021] On each of the first scene maps, the first starting point of any starting lane segment of the first lane is selected as the origin, and a two-dimensional Cartesian coordinate system is constructed with the forward direction of the starting lane segment as the positive X-axis, denoted as the corresponding first coordinate system; and the coordinates of the first starting point and the first ending point of each of the first lane segments on the current first scene map on the first coordinate system are taken as the corresponding first starting point coordinates (x... s ,y s ) and the coordinates of the first endpoint (x) e ,y e ); and the coordinates (x) of the first starting point of each of the first lane segments. s ,y s ) and the first endpoint coordinates (x) e ,y e The lane node vectors are formed into a corresponding 1*4 shape; and the first map tensor is formed from all the obtained lane node vectors; the first map tensor has a shape of N'*4, where N' is the total number of lane node vectors;

[0022] The first training dataset is composed of all the first map tensors obtained.

[0023] Preferably, the step of performing map network pre-training on the map network module of the LaneGCN model based on the first training dataset specifically includes:

[0024] Step 41: Extract the map network module separately and connect it with the preset first multilayer perception network to form the corresponding first training network;

[0025] Step 42: Extract one of the first map tensors from the first training dataset as the corresponding current map tensor;

[0026] Step 43: Statistically count the total number of lane node vectors in the current map tensor to generate the corresponding total number of lane nodes N0; calculate the current total number of input lane nodes N1 = N0 * (1-β) and the total number of lane nodes to be tested N2 = N0 * β based on a preset first occlusion ratio β; extract the lane node vectors of the total number of input lane nodes N1 from the current map tensor to form a current input tensor with shape N1 * 4; and form a current label tensor with shape N2 * 4 from the remaining total number of lane nodes to be tested N2; and use each lane node vector in the current label tensor as the corresponding lane node to be tested Ln. i ; 1≤i≤N2; The first occlusion ratio β is defaulted to 20%;

[0027] Step 44, for each of the lane nodes to be tested Ln i truth center The coordinates are determined; the true center point The coordinates are x-axis The first starting point coordinate (x) of the corresponding lane node vector s ,y s ) and the first endpoint coordinates (c, y) e The average of the x-axis of ) s +x e ) / 2, ordinate For the corresponding first starting point coordinates (x) s ,y s ) and the first endpoint coordinates (x) e ,y e The average value of the ordinate (y) s +y e ) / 2;

[0028] Step 45, based on the total number of lane nodes N2 to be tested and each of the true center points The corresponding first loss function is set as follows:

[0029]

[0030]

[0031]

[0032]

[0033] s i The lane node to be tested, Ln i The prediction center point, the prediction center point s i The coordinates are (x i ,y i ), reg() is the regression function, and d() is the smoothed L1 loss function. for norm expression, for The norm expression;

[0034] Step 45: Input the current input tensor and the current label tensor of shape N1*4 into the first training network. The map network module performs map lane feature extraction processing on the current input tensor to generate a corresponding first training tensor of shape N1*128. The first multilayer perceptron then uses the first training tensor and the current label tensor to compare each lane node Ln to be tested with the current label tensor. i The corresponding center point coordinates are used to perform regression prediction to generate a second training tensor of shape N2*2; the second training tensor includes N2 predicted center points s i coordinates (x) i ,y i );

[0035] Step 46, connect each of the lane nodes Ln to be tested. i The corresponding prediction center point s i coordinates (x) i ,y i ) and the truth center point coordinates Substitute the values ​​into the first loss function to calculate the corresponding first loss value;

[0036] Step 47: Identify whether the first loss value meets the preset first loss value convergence range; if the first loss value does not meet the first loss value convergence range, then perform reverse modulation on the network parameters of the map network module along the direction that makes the first loss function reach the minimum value, and return to step 45 to continue training when the reverse modulation is completed; if the first loss value meets the first loss value convergence range, then identify whether the current map tensor is the last first map tensor in the first training dataset; if it is not the last, then extract the next first map tensor as the new current map tensor and return to step 43 to continue training; if it is the last, then stop model training and confirm that the map network pre-training is successful.

[0037] Preferably, the step of acquiring vehicle road test data to construct a second training dataset specifically includes:

[0038] Step 51: Select a first road test data sequence from a preset vehicle road test database; the first road test data sequence consists of (T1+1)+T2 frames of first road test data that are sequentially continuous and have completed target association; the first road test data includes a first timestamp, a first vehicle coordinate, and multiple first target data groups, each first target data group including a first target identifier and a first target coordinate; the first target identifiers of the two first target data groups corresponding to the associated targets in the first road test data at different times are the same;

[0039] Step 52: Using the first vehicle coordinates of the first road test data in frame (T1+1) as the first reference point coordinates, and using the first timestamp corresponding to the first reference point coordinates minus 1 second as the first time, and using the first vehicle coordinates corresponding to the first timestamp that is earlier than the first time and closest to the first time as the second reference point coordinates; using the first reference point coordinates as the origin, and using the direction from the second reference point coordinates to the first reference point coordinates as the positive X-axis, construct a two-dimensional Cartesian coordinate system, denoted as the corresponding second coordinate system;

[0040] Step 53: Perform coordinate transformation on all the first vehicle coordinates and the first target coordinates in the second coordinate system to generate the corresponding second vehicle coordinates and second target coordinates;

[0041] Step 54: Sort the coordinates of multiple second targets with the same first target identifier in the first road test data of the previous (T1+1) frames according to time sequence to generate corresponding participant historical trajectories; and count the total number of participant historical trajectories to generate the corresponding first participant total number M1; and perform coordinate difference calculation on adjacent second target coordinates in each participant historical trajectory to generate corresponding differential coordinate sequences; if the number of differential coordinates in the differential coordinate sequence is less than the number of first time points T1, it is padded with leading zeros to ensure that the differential coordinate sequence always has the number of differential coordinates of the first time points T1, and assign a corresponding zero-padding status identifier to each differential coordinate; and the horizontal differential coordinate Δx and vertical differential coordinate Δy of each differential coordinate and the corresponding zero-padding status identifier constitute a pair of differential coordinates. The participant observation vectors are of shape 1*3; and the participant observation vectors corresponding to the first time point T1 of each participant's historical trajectory form a first observation tensor of shape T1*3; the first observation tensor is transposed to obtain a participant observation tensor of shape 3*T1; and the participant observation tensors of the obtained first total number of participants M1 form a first training tensor of shape M1*3*T1; the differential coordinates include the horizontal differential coordinates Δx and the vertical differential coordinates Δy; the zero-padding status identifier includes a first identifier value and a second identifier value. If the zero-padding status identifier is the first identifier value, the type of the corresponding differential coordinates is zero-padding differential coordinates; if the zero-padding status identifier is the second identifier value, the type of the corresponding differential coordinates is true differential coordinates.

[0042] Step 55: Sort the coordinates of multiple second targets with the same first target identifier in the first road test data of the later T2 frame according to the time sequence to generate the corresponding participant real trajectory; if the number of second target coordinates in the participant real trajectory is less than the number of second time points T2, use a preset Kalman filter to predict the trajectory coordinates of the participant real trajectory in the future multiple time points to ensure that the participant real trajectory always has the number of second target coordinates of the second time points T2; and form a corresponding first real trajectory set from all the obtained participant real trajectories;

[0043] Step 56: Query the preset high-precision map library, and extract the high-precision map that covers all the coordinates of the first vehicle and the first target according to the preset map size as the corresponding second scene map; the second scene map includes multiple second lanes, and the second lane includes the center line of the second lane;

[0044] Step 57: Divide each second lane into multiple corresponding second lane segments according to the preset lane segment length; and record the two intersection points of each second lane segment with the corresponding second lane centerline as the corresponding second start point and second end point; and use the coordinates of the second start point and second end point of each second lane segment in the second coordinate system as the corresponding second start point coordinates and second end point coordinates; and form a corresponding lane node vector of shape 1*4 by the second start point coordinates and second end point coordinates of each second lane segment; and generate a corresponding total number of lane node vectors N3 by counting the total number of lane node vectors; and form a corresponding second map tensor of shape N3*4 by the N3 lane node vectors.

[0045] Step 58: The second training data, composed of the first training tensor, the first real trajectory set, and the second map tensor, is stored in the second training dataset.

[0046] Step 59: Calculate the total number of the second training data in the second training dataset to generate a corresponding first quantity; if the first quantity is less than a preset first total threshold, return to step 51 to continue preparing training data; if the first quantity is equal to the first total threshold, stop preparing training data and output the second training dataset.

[0047] Preferably, the step of training the LaneGCN model based on the second training dataset specifically includes:

[0048] Step 61: Set the loss function L of the LaneGCN model according to the second time step number T2. mod for:

[0049] L mod =L cls +αL reg ,

[0050]

[0051]

[0052] reg(Z)=∑ j d(z j ),

[0053]

[0054] Wherein, factor α is 1.0;

[0055] L cls For the classification loss function, L reg For regression loss function;

[0056] The classification loss function L cls In this context, ∈ represents a preset marginal factor, m is the participant index, 1 ≤ m ≤ the total number of participants M, and the participant index m corresponds to the first target coordinate; k is the confidence level negative sample index, k * For the positive sample index of confidence level, 1≤k, k * ≤ Total number of modes K, but k ≠ k * ;c m，k For the m-th participant, there are K negative classification samples from the first confidence level. The positive classification sample is one of the K first confidence scores of the m-th participant; only one of the K first confidence scores of each participant is the positive classification sample, and the remaining K-1 are the negative classification samples; the straight-line distance between the trajectory point coordinates of the last first trajectory point vector of the first predicted trajectory tensor corresponding to the positive classification sample and the last second target coordinates of the corresponding participant's true trajectory is the shortest.

[0057] The prediction loss function L reg In this context, T2 is the preset number of second time points, which defaults to 30; t is the time index, 1≤t≤T2; reg() is the regression function, and d() is the smoothed L1 loss function. For the K first predicted trajectory tensors of the m-th participant, the index k of the positive sample corresponding to the confidence level is... * The coordinates of the trajectory point of the t-th first trajectory point vector on the first predicted trajectory tensor; Let t be the coordinate of the second target in the actual trajectory of the m-th participant;

[0058] Step 62: Extract the first piece of the second training data from the second training dataset as the corresponding current training data;

[0059] Step 63: Extract the first training tensor, the first real trajectory set, and the second map tensor from the current training data as the corresponding current training tensor, current real trajectory set, and current map tensor; and generate the current total number of participants M2 by counting the number of participant observation tensors in the current training tensor; and generate the current total number of lane node vectors N4 by counting the number of lane node vectors in the current map tensor.

[0060] Step 64: Input the current training tensor of shape M2*3*T1 into the participant network module for participant feature extraction processing to generate a corresponding second feature tensor of shape M2*128; input the current map tensor of shape N4*4 into the map network module for map lane feature extraction processing to generate a corresponding third feature tensor of shape N4*128; input the second and third feature tensors into the fusion network module for feature fusion processing to generate a corresponding fourth feature tensor of shape M2*128; and input the fourth feature tensor into the regression branch and the classification branch of the prediction head network module respectively for corresponding multimodal trajectory prediction processing and multimodal confidence classification processing to generate a corresponding second prediction tensor of shape M2*K*T1*2 and a second confidence tensor of shape M2*K.

[0061] The second prediction tensor includes M2 second participant prediction tensors of shape K*T2*2; the second participant prediction tensor corresponds to the first target identifier; the second participant prediction tensor includes K second prediction trajectory tensors of shape T2*2; the second prediction trajectory tensor includes a second trajectory point vector of the second time point T2; the second trajectory point vector is a two-dimensional trajectory point coordinate.

[0062] The second confidence tensor includes M2 second confidence vectors of length K; the second confidence vectors correspond to the first target identifier; the second confidence vector includes K second confidences; the second confidences correspond one-to-one with the second predicted trajectory tensor, and the second confidence is the prediction confidence of the corresponding second predicted trajectory tensor;

[0063] Step 65: Substitute the current total number of participants M2, the second prediction tensor, the second confidence tensor, and the current set of true trajectories into the loss function L of the LaneGCN model. mod Calculate the loss value to generate the corresponding second loss value;

[0064] Step 66: Identify whether the second loss value meets the preset convergence range of the second loss value; if the second loss value does not meet the convergence range of the second loss value, then perform back modulation on the network parameters of each module in the LaneGCN model except the map network module along the direction that makes the second loss function reach its minimum value, and return to step 64 to continue training when the back modulation is completed; if the second loss value meets the convergence range of the loss value, then identify whether the current training data is the last second training data in the second training dataset; if it is not the last, then extract the next second training data as the new current training data and return to step 63 to continue training; if it is the last, then stop the model training and confirm that the model training is successful.

[0065] A second aspect of the present invention provides an apparatus for implementing the model training processing method described in the first aspect above, the apparatus comprising: a first training data preparation module, a first model training module, a second training data preparation module, and a second model training module;

[0066] The first training data preparation module is used to acquire high-precision maps to construct the first training dataset;

[0067] The first model training module is used to perform map network pre-training on the map network module of the LaneGCN model based on the first training dataset.

[0068] The second training data preparation module is used to acquire vehicle road test data to construct a second training dataset when the map network pre-training is successful;

[0069] The second model training module is used to train the LaneGCN model based on the second training dataset while keeping the network parameters of the map network module unchanged.

[0070] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0071] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0072] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0073] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0074] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for model training. Before performing overall model training on the LaneGCN model, a high-precision map training dataset and loss function L are first applied. map The map network module is pre-trained to achieve sufficient training results; then, while keeping the network parameters of the map network module unchanged, the loss function L is applied. mod =L cls +αL reg The LaneGCN model is trained as a whole. This invention reduces the training difficulty of the overall model, improves the training maturity of the map network module, and enhances the output accuracy of the LaneGCN model. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of a model training processing method provided in Embodiment 1 of the present invention;

[0076] Figure 2 This is a module structure diagram of a model training processing device provided in Embodiment 2 of the present invention;

[0077] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0079] Embodiment 1 of the present invention provides a method for processing model training, such as... Figure 1 The schematic diagram shows a model training processing method provided in Embodiment 1 of the present invention. This method mainly includes the following steps:

[0080] Step 1: Obtain high-precision maps to construct the first training dataset;

[0081] Specifically, it includes: Step 11, selecting high-precision map raw data from a publicly available high-precision map database according to preset map filtering rules to form the first scene map set;

[0082] The map filtering rules include minimum map quantity, minimum map size, maximum map accuracy, minimum number of lanes, minimum lane length, and lane segment length; the first scene map set includes multiple first scene maps; the number of first scene maps is not less than the minimum map quantity; the map size of the first scene map is not less than the minimum map size, and the map accuracy is not greater than the maximum map accuracy; the first scene map includes multiple first lanes, the number of first lanes is not less than the minimum number of lanes, and the lane length is not less than the minimum lane length; the first lane includes the centerline of the first lane;

[0083] Here, the minimum map quantity, minimum map size, minimum map accuracy, minimum number of lanes, minimum lane length, and lane segment length in the map filtering rules are all pre-set rule parameters. The larger the minimum map quantity, the more scene maps are used for training in the first scene map set; the smaller the maximum map accuracy, the higher the accuracy of the scene maps used for training; the smaller the minimum number of lanes, minimum lane length, and lane segment length, the more complex the scene maps used for training; the lane segment length will be used for lane segmentation in subsequent steps to generate corresponding lane nodes; as is common knowledge about high-precision maps, each high-precision map includes lane information, i.e., the first lane, and each first lane has a lane centerline, i.e., the first lane centerline.

[0084] Step 12: Divide each first lane of each first scene map into multiple corresponding first lane segments according to the lane segment length of the map filtering rules; and record the two intersections of each first lane segment with the corresponding first lane centerline as the corresponding first start point and first end point;

[0085] Here, when processing segments, the segmentation starts from the starting position of each first lane in the first scene map by default. Every other lane segment length is cut off as the corresponding first lane segment. Each first lane segment must have two intersection points with the center line of the first lane of the current lane. Here, the intersection point closer to the starting position of the lane is recorded as the first starting point, and the intersection point farther away from the starting position of the lane is recorded as the first ending point.

[0086] Step 13: On each first scene map, select any starting point of the first lane segment as the origin and construct a two-dimensional Cartesian coordinate system with the forward direction of the starting lane segment as the positive X-axis. Record this as the corresponding first coordinate system. Then, use the coordinates of the first starting point and the first ending point of each first lane segment on the current first scene map in the first coordinate system as the corresponding first starting point coordinates (x...). s ,y s ) and the coordinates of the first endpoint (x) e ,y e ); and the coordinates of the first starting point (x) of each first lane segment.s ,y s ) and the coordinates of the first endpoint (x) e ,y e The lane node vectors are formed into a 1*4 shape; and the first map tensor is formed from all the obtained lane node vectors; the shape of the first map tensor is N'*4, where N' is the total number of lane node vectors;

[0087] Here, if we take the first starting point of any lane segment of the first lane as the vehicle's position, then the first coordinate system is actually a two-dimensional bird's-eye view vehicle coordinate system with the vehicle's position as the origin and the vehicle's direction of travel as the positive X-axis; then the coordinates of the first starting point (x s ,y s ) and the coordinates of the first endpoint (x) e ,y e The first feature vector (x, y) represents the coordinates of two feature points (first start point and first end point) in the first coordinate system, i.e., the vehicle's coordinate system, for each lane segment. The feature vector of each lane segment is defined as the lane node vector. The content of the lane node vector is composed of the coordinates of the two feature points of the corresponding lane segment in the vehicle's coordinate system; that is, the lane node vector is (x, y) = ... s ,y s ,x e ,y e Therefore, the shape of the lane node vector is 1*4; the total number of lane segments on the current first scene map, which is also the total number of lane node vectors, is N'. Then, from N... ’ Combining the vectors of each lane node yields a feature tensor of shape N'*4, which is used to represent the current first scene map, i.e., the first map tensor.

[0088] Step 14: The first training dataset is composed of all the obtained first map tensors.

[0089] Here, by using step 1 above, we can avoid being limited by the overall input data range of the LaneGCN model and search for various high-precision maps as much as possible to build the first scene map set, thereby laying a data foundation for the full training of the map network module.

[0090] Step 2: Perform map network pre-training on the map network module of the LaneGCN model based on the first training dataset;

[0091] Specifically, this includes: Step 21, extracting the map network module separately and connecting it with a preset first multilayer perceptron (MLP) to form a corresponding first training network;

[0092] Here, the first multilayer perceptron is a pre-trained neural network. The first multilayer perceptron has two inputs: the output tensor of the map network module and a pre-prepared label tensor. The purpose of the first multilayer perceptron is to first perform a fully connected operation on the output tensor of the map network module to obtain a global feature. This global feature includes the shape (position, size, orientation, etc.) features of each known lane node involved in the output tensor of the map network module. Then, based on the above global feature, a fully connected regression calculation is performed to predict the center point of all lane nodes. Here, "all lane nodes" includes known lane nodes and unknown lane nodes. Unknown lane nodes refer to lane nodes with similar shapes (size, orientation, etc.) but not continuous known lane nodes. Unknown lane nodes can be reconstructed through fully connected regression calculation. After obtaining the center points of all lane nodes, the center point of the corresponding lane node to be tested is given as the network output based on the lane node to be tested given in the label tensor. Therefore, the input of the first training network in this embodiment is actually a partial lane node vector of a map tensor, and the output is an output tensor composed of the coordinates of the center points corresponding to the remaining lane node vectors of the map tensor.

[0093] Step 22: Extract a first map tensor from the first training dataset as the corresponding current map tensor;

[0094] Step 23: Calculate the total number of lane node vectors in the current map tensor to generate the corresponding total number of lane nodes N0; calculate the total number of input lane nodes N1 = N0 * (1-β) and the total number of lane nodes to be tested N2 = N0 * β based on the preset first occlusion ratio β; extract the lane node vectors of the total number of input lane nodes N1 from the current map tensor to form the corresponding current input tensor with shape N1 * 4; and form the corresponding current label tensor with shape N2 * 4 from the remaining total number of lane nodes to be tested N2; and use each lane node vector in the current label tensor as the corresponding lane node to be tested Ln. i ;

[0095] Where 1≤i≤N2; the first occlusion ratio β is 80% by default;

[0096] Here, the total number of lane node vectors in the current map tensor is the total number of lane nodes N0. When training the map network module, a portion of the lane node vectors needs to be extracted from these N0 according to a preset extraction ratio, namely (1 - first occlusion ratio β), to form the input vector of the map network module. The remaining lane node vectors are then used to form the label tensor required by the first multilayer perception network. This operation is similar to partially occluding the lane nodes in the current map tensor, so the relevant ratio is called the occlusion ratio, i.e., the first occlusion ratio β. The default value of the first occlusion ratio β is 20%. When extracting lane node vectors from the current map tensor to form the current input tensor, the extraction is performed according to a random extraction principle. Each lane node vector in the resulting label tensor is defined as the lane node to be tested, Ln. i The lane node Ln to be tested here i This refers to the unknown lane node mentioned in step 21 above;

[0097] For example, if the current map tensor has a shape of 50*4, indicating that the total number of lane nodes N0 = 50 and the first occlusion ratio β = 20%, then the total number of lane nodes N1 = N0*(1-β) = 40, and the total number of lane nodes to be tested N2 = N0*β = 10. Forty lane node vectors are randomly extracted from the current map tensor to form a current input tensor with a shape of 40*4, and the remaining 10 lane node vectors are used to form a current label tensor with a shape of 10*4. These 10 lane node vectors in the current label tensor are sequentially denoted as the lane nodes to be tested Ln. i＝1 、Ln i＝2 ...Ln i＝10 ;

[0098] Step 24, for each lane node Ln to be tested i truth center The coordinates are determined;

[0099] Among them, the truth center point The coordinates are x-axis The first starting point coordinate (x) of the corresponding lane node vector s ,y s ) and the coordinates of the first endpoint (c, y) e The average of the x-axis of ) s +x e ) / 2, ordinate The corresponding first starting point coordinates (x) s ,y s ) and the coordinates of the first endpoint (x) e ,y e The average value of the ordinate (y) s +y e ) / 2;

[0100] Here, each lane node Ln to be tested i A corresponding set: First starting point coordinates (x) s ,y s ) and the coordinates of the first endpoint (c, y) e ); then, for each lane node Ln to be tested i truth center The coordinates should be

[0101] For example, the current label tensor is known to include 10 lane nodes Ln to be tested. i＝1 、Ln i＝2 ...Ln i＝10 This allows us to calculate 10 truth center points.

[0102] Step 25, based on the total number of lane nodes N2 to be tested and the ground truth center points of each lane... The corresponding first loss function is set as follows:

[0103]

[0104]

[0105]

[0106]

[0107] Among them, s i For the lane node Ln to be tested i The predicted center point, the predicted center point s i The coordinates are (x i ,y i ), reg() is the regression function, and d() is the smoothed L1 loss function. for norm expression, for The norm expression;

[0108] Here, the predicted center point s i It refers to the center points of the lane nodes to be tested contained in the output tensor of the first multilayer sensing network, with each predicted center point s. i Each prediction center point s corresponds to a lane node to be tested. i Corresponding to a truth center The smaller the distance between the two, the higher the maturity of the first training network; it is also known that the first multilayer perceptron of the first training network is a pre-trained mature neural network, and its network parameters remain unchanged throughout the training process, that is, the prediction center point s i With the center of truth The smaller the distance between the two, the higher the network maturity of the map network module in the first training network; therefore, the embodiment of the present invention is based on the predicted center point s i With the center of truth Regression function of the distance between the two The average value is used to construct the first loss function L. map The regression function The calculation method is shown above, using a smoothed L1 loss function with horizontal (X-axis) and vertical (Y-axis) spacing. and The sum is used to achieve this; while the calculation principle of the smoothed L1 loss function d(σ) is known from the calculation method of the smoothed L1 loss function, and is determined by the norm ‖σ‖ of the calculation factor σ in the binary domain (the first domain less than 1 and the second domain greater than or equal to 1). In the first domain less than 1, (0.5*σ) is used. 2 The calculation is performed using the method of (‖σ‖-0.5) in the second value range that is greater than or equal to 1;

[0109] Step 25: Input the current input tensor and the current label tensor of shape N1*4 into the first training network. The map network module performs map lane feature extraction processing on the current input tensor to generate the corresponding first training tensor of shape N1*128. The first multilayer perceptron uses the first training tensor and the current label tensor to analyze each lane node Ln to be tested in the current label tensor. i The corresponding center point coordinates are used to perform regression prediction to generate a second training tensor with a shape of N2*2;

[0110] The second training tensor includes N2 prediction center points s. i coordinates (x) i ,y i );

[0111] Here, the implementation of the map network module's processing of map lane feature extraction from the current input tensor can be found in the paper "Learning Lane Graph Representations for Motion Forecasting," and will not be elaborated upon here; the implementation of the first multilayer perceptron is as described above, and will not be repeated here.

[0112] For example, given that the current input tensor has a shape of 40*4 and the current label tensor has a shape of 10*4, then the resulting second training tensor will have a shape of 10*2, meaning the second training tensor consists of 10 prediction center points s. i coordinates (x) i ,y i )constitute;

[0113] Step 26, connect each lane node Ln to be tested. i The corresponding prediction center point s i coordinates (x) i ,y i and truth center coordinates Substitute the values ​​into the first loss function to calculate the corresponding first loss value;

[0114] For example, given that the total number of lane nodes to be tested is N2 = 10, and 10 prediction center points s are known. i coordinates (x) i ,y i ) and 10 truth centers coordinates Substitute into the first loss function L map The first loss value can be obtained as:

[0115]

[0116] Step 27: Identify whether the first loss value meets the preset first loss value convergence range; if the first loss value does not meet the first loss value convergence range, then perform reverse modulation on the network parameters of the map network module along the direction that makes the first loss function reach the minimum value, and return to step 25 to continue training when the reverse modulation is completed; if the first loss value meets the first loss value convergence range, then identify whether the current map tensor is the last first map tensor in the first training dataset. If it is not the last, then extract the next first map tensor as the new current map tensor and return to step 23 to continue training; if it is the last, then stop the model training and confirm that the map network pre-training is successful.

[0117] Here, the first loss value convergence range is a preset reasonable loss value error range. If the first loss value does not meet the first loss value convergence range, it means that the current training has not yet converged. Then, the network parameters of the map network module are modulated, and after modulation, the process returns to step 25 to continue training. If the first loss value meets the first loss value convergence range, it means that the training based on the current training data, that is, the current map tensor, has ended. The next training data, that is, the next first map tensor of the first training dataset, needs to be retrieved for training until all the first map tensors of the first training dataset have been used for training and have all achieved convergence.

[0118] In summary, through the training data preparation and model training in steps 1-2 of this embodiment, the overall input limitations of the LaneGCN model can be eliminated, enabling sufficient training of the map network module. This results in higher model accuracy and generalization adaptability for the map network module within the LaneGCN model. Following steps 1-2, the network parameters of the map network module can be fixed and incorporated into the LaneGCN model for overall training in subsequent steps 3-4.

[0119] Before describing the subsequent steps 3-4, let's briefly describe the module structure and data structure of the LaneGCN model.

[0120] The LaneGCN model includes a participant network module, a map network module, a fusion network module, and a prediction head network module; the fusion network module is connected to the participant network module, the map network module, and the prediction head network module, respectively.

[0121] For a detailed explanation, please refer to the paper "Learning Lane Graph Representations for Motion Forecasting";

[0122] (I) Participant Network Module

[0123] The participant network module is used to perform participant feature extraction processing on the first input tensor of shape M*3*T1 to generate the corresponding first output tensor of shape M*128; and send the first output tensor to the fusion network module.

[0124] The first input tensor includes M participant observation tensors of shape 3*T1; M is the total number of participants; T1 is the preset number of first time points, which is 20 by default.

[0125] Here, the so-called participants refer to traffic participants, that is, obstacle targets (targets) identified by the perception module of the autonomous driving system. The total number of participants M is a variable, and the number of participants at the first moment T1 is a pre-set constant, which defaults to 20. In practical applications, the number of participants at the first moment T1 refers to the number of sampling moments before the current moment. For example, if M=5, T1=20, and the current moment is t0, then the first input tensor is actually a tensor composed of 5 historical trajectories of 5 obstacle targets before the current moment t0 (i.e., 3*T1 participant observation tensors). The historical trajectory (participant observation tensor) of each obstacle target includes T1=20 trajectory points. Each trajectory point corresponds to a vector of length 3. As can be seen from the subsequent processing steps, this vector includes the coordinates (x, y) of the corresponding trajectory point and a zero-padding status indicator.

[0126] (II) Map Network Module

[0127] The map network module is used to perform map lane feature extraction processing on the second input tensor of shape N*4 to generate a corresponding second output tensor of shape N*128; and sends the second output tensor to the fusion network module;

[0128] The second input tensor includes N lane node vectors of shape 1*4; N is the total number of lane node vectors.

[0129] Here, as mentioned earlier, the data structure of the lane node vector consists of the coordinates of the corresponding first starting point and first ending point; the second input tensor consists of all lane node vectors in the current map; the total number of lane node vectors N is a variable.

[0130] (III) Converged Network Module

[0131] The fusion network module is used to perform feature fusion processing on the first output tensor and the second output tensor to generate a corresponding first fused feature tensor with a shape of M*128; and send the first fused feature tensor to the prediction head network module.

[0132] Here, as shown in the paper "Learning Lane Graph Representations for MotionForecasting", the fusion network module includes four fusion units: Actor to Lane (A2L) fusion unit, Lane to Lane (L2L) fusion unit, Lane to Actor (L2A) fusion unit, and Actor to Actor (A2A) fusion unit. The A2L fusion unit is used to perform feature fusion from actor to lane on the first output tensor and the second output tensor to output the corresponding A2L fusion feature tensor. The L2L fusion unit is used to perform feature fusion from lane to lane on the A2L fusion feature tensor to output the corresponding L2L fusion feature tensor. The L2A fusion unit is used to perform feature fusion from lane to actor on the first output tensor and the L2L fusion feature tensor to output the corresponding L2A fusion feature tensor. The A2A fusion unit is used to perform feature fusion from actor to actor on the L2A fusion feature tensor to output the corresponding A2A fusion feature tensor, which is also the first fusion feature tensor with shape M*128.

[0133] (iv) Prediction Head Network Module

[0134] The prediction head network module includes a regression branch and a classification branch; the regression branch is used to perform multimodal trajectory prediction processing based on the first fusion feature tensor to generate a first prediction tensor with a corresponding shape of M*K*T2*2; the classification branch is used to perform multimodal confidence classification processing based on the first fusion feature tensor to generate a first confidence tensor with a corresponding shape of M*K.

[0135] Where K is the total number of modes; T2 is the preset number of second time points, which is 30 by default;

[0136] The first prediction tensor includes M first participant prediction tensors of shape K*T2*2; each first participant prediction tensor corresponds one-to-one with the M participants; the first participant prediction tensor includes K first prediction trajectory tensors of shape T2*2; the first prediction trajectory tensor includes a first trajectory point vector of the second time step number T2; the first trajectory point vector is a two-dimensional trajectory point coordinate.

[0137] The first confidence tensor includes M first confidence vectors of length K; each first confidence vector corresponds one-to-one with one of the M participants; each first confidence vector includes K first confidence values; each first confidence value corresponds one-to-one with the first predicted trajectory tensor, and the first confidence value is the prediction confidence value of the corresponding first predicted trajectory tensor.

[0138] Here, the prediction head network module predicts K possible trajectories of M participants in future time periods through regression and classification branches, obtaining M*K predicted trajectories, i.e., the first predicted trajectory tensor. The output first predicted tensor is composed of the M*K first predicted trajectory tensors. Each first predicted trajectory tensor is assigned a confidence score, i.e., the first confidence score. The corresponding first confidence score is composed of K first confidence scores for each participant. The output first confidence tensor is composed of M first confidence vectors of length K.

[0139] The total number of time points for each first predicted trajectory tensor, which is also the total number of first trajectory point vectors, is equal to the number of second time points T2. Here, each first trajectory point vector corresponds to the coordinates of a predicted trajectory point, so the length of the first trajectory point vector is 2. Therefore, the shape of the first predicted trajectory tensor is T2*2.

[0140] Each trajectory may correspond to a mode. The total number of modes K can be preset to 4, which corresponds to the four possibilities of front, back, left, and right. Alternatively, it can be obtained based on the actual prediction results each time. If the total number of modes K is a preset constant, then there may be a first prediction trajectory tensor in the first prediction tensor where the trajectory point vector is all 0, indicating that there is no corresponding prediction trajectory. In this case, the corresponding first confidence level is 0.

[0141] The above description introduces the module structure and data structure of the LaneGCN model, which provides a basis for understanding the data preparation and model training processes in steps 3-4 below.

[0142] Step 3: If the map network pre-training is successful, then obtain vehicle road test data to construct the second training dataset;

[0143] Here, the second training dataset is a training dataset constructed based on the overall input requirements of the LaneGCN model, namely the input data requirements of the participant network module and the map network module; wherein, the input data of the map network module and the input data of the participant network module have spatial correlation;

[0144] Specifically, this includes: Step 31, selecting a first road test data sequence from a preset vehicle road test database;

[0145] The first road test data sequence consists of (T1+1)+T2 frames of first road test data that are sequentially continuous and have completed target association. The first road test data includes a first timestamp, a first vehicle coordinate, and multiple first target data groups. Each first target data group includes a first target identifier and a first target coordinate. The first target identifiers of the two first target data groups corresponding to the associated targets in the first road test data at different times are the same.

[0146] Here, in this embodiment of the invention, before proceeding to step 3, data storage for the vehicle road test database is performed through vehicle road tests. Specifically:

[0147] During the driving process, the vehicle performing the road test continuously perceives obstacles and targets (i.e., traffic participants) around the vehicle by calling the perception module of the vehicle's autonomous driving system to obtain multiple perception data (similar to LiDAR point clouds, camera images, etc.) at continuous moments. For each perception data obtained, the vehicle performs target detection on the current perception data based on a preset target detection model to obtain multiple target poses (including coordinates, shape, speed, orientation, etc.). Then, based on a preset target association algorithm, the target detection results of the two perception data at different moments are associated and the same target identification information is assigned to the associated targets. Then, the trajectory of the same target (i.e. the same participant) with the same target identification information is tracked to obtain the corresponding target historical trajectory. Each target historical trajectory corresponds to a target identification and consists of multiple trajectory points. Each trajectory point includes trajectory point coordinate information and corresponds to a timestamp information.

[0148] In addition, while the vehicle performing the road test task calls the perception module to track the target's historical trajectory, it also synchronously acquires the vehicle's real-time pose (including coordinates, speed, orientation, etc.) through the positioning module of the autonomous driving system, and obtains the vehicle's historical trajectory based on the vehicle's real-time pose. Similar to the target historical trajectory, the vehicle's historical trajectory also consists of multiple trajectory points, each trajectory point including trajectory point coordinate information and corresponding timestamp information. When maintaining the vehicle's historical trajectory, the vehicle will synchronize the timestamp information of the vehicle's trajectory points with the timestamp information of the target trajectory points, so that the retained vehicle trajectory points and the trajectory points of each target historical trajectory can correspond to the same synchronized timestamp information.

[0149] In addition, while generating time-synchronized target historical trajectories and self-vehicle historical trajectories, vehicles performing road test tasks also periodically segment their self-vehicle historical trajectories and the target historical trajectories of each participant based on preset segment durations to obtain trajectory segments for the self-vehicle and each participant. Then, the trajectory segments of the self-vehicle and each participant in the same time period are combined according to the synchronization timestamp information of the trajectory points to obtain the road test data sequence for that time period and stored locally. The road test data sequence includes multiple frames of road test data. Each frame of road test data includes a synchronization timestamp + self-vehicle coordinates + multiple target data groups. Each target data group includes a target identifier + target coordinates. The length of the road test data sequence is the total number of road test data.

[0150] At the end of the road test mission, the vehicle performing the road test mission will upload multiple road test data sequences saved during the current journey to the vehicle road test database for storage.

[0151] Based on the above data storage process, we know that the first road test data sequence obtained from the vehicle road test database in the current step is actually a collection of trajectory segments of a single vehicle and multiple participants; each first road test data in the first road test data sequence is actually a combination of trajectory point coordinates of a single vehicle and multiple participants at a given time point; the first timestamp of the first road test data is the synchronization timestamp information of the vehicle and multiple participants at the corresponding time point; the first vehicle coordinates of the first road test data are the coordinate information in the real-time pose of the vehicle at the corresponding time point; each first target data group of the first road test data is the target identifier information + trajectory point coordinate information of each participant at the corresponding time point, where the first target identifier is the target identifier information of the corresponding participant, and the first target coordinates are the trajectory point coordinate information of the corresponding participant at the corresponding time point; the first road test data at consecutive times refers to two adjacent first road test data in the first road test data sequence. As can be seen from the aforementioned target association description, the first target identifier in two first road test data belonging to the same participant (i.e., associated target) is the same target identifier information, that is, the first target identifier of the two first target data groups corresponding to the associated target in the first road test data at consecutive times is the same;

[0152] In this embodiment of the invention, the default length of the first road test data sequence is (T1+1)+T2. Therefore, when selecting a first road test data sequence from the vehicle road test database, a road test data sequence is first selected from the vehicle road test database as the current road test data sequence. If the length of the current road test data sequence is equal to (T1+1)+T2, then the current road test data sequence is output as the corresponding first road test data sequence. If the length of the current road test data sequence is greater than (T1+1)+T2, then multiple road test data sequence segments are obtained by sliding segmenting the current road test data sequence with a step size of 1, using (T1+1)+T2 as the truncation length. Each road test data sequence segment is then used as a candidate sequence for the first road test data sequence. This processing method in this embodiment of the invention can fully utilize the stored data in the vehicle road test database to generate rich training data sources.

[0153] For example, if the length of road test data sequence A in the vehicle road test database is 60, and (T1+1)+T2=(20+1)+30=51, then by using a sliding segment truncation method with a step size of 1 and a truncation length of 51, starting from the first road test data, we can obtain (60-51)+1=10 candidate sequences for use. This is equivalent to generating 10 first road test data sequences for backup from one road test data sequence A, and the length of each backup first road test data sequence is 51.

[0154] Step 32: Take the first vehicle coordinates of the first road test data in frame (T1+1) as the first reference point coordinates, and take the first time stamp corresponding to the first reference point coordinates minus 1 second as the first time, and take the first vehicle coordinates corresponding to the first time stamp that is earlier than the first time and closest to the first time as the second reference point coordinates; construct a two-dimensional Cartesian coordinate system with the first reference point coordinates as the origin and the direction from the second reference point coordinates to the first reference point coordinates as the positive X-axis, and denote it as the corresponding second coordinate system;

[0155] Here, with the first time point T1 being 20 by default and the second time point T2 being 30 by default, the first road test data sequence includes (T1+1)+T2=51 frames of first road test data; the second coordinate system is actually a two-dimensional bird's-eye view vehicle coordinate system with the time point corresponding to (T1+1)=21 frames of first road test data as the current time point t0, and the first vehicle coordinate corresponding to the current time point t0 as the origin, and the vehicle's driving direction (orientation) from the previous second to the current time point t0 as the positive X-axis;

[0156] Step 33: Perform coordinate transformation on all first vehicle coordinates and first target coordinates in the second coordinate system to generate corresponding second vehicle coordinates and second target coordinates;

[0157] Here, the actual process is to transform all the trajectory point coordinates (the coordinates of the trajectory points of the vehicle and each reference) in the first road test data sequence to a unified vehicle coordinate system, namely the second coordinate system.

[0158] Step 34: Sort the coordinates of multiple second targets with the same first target identifier in the first (T1+1) frame of the first road test data according to their chronological order to generate the corresponding participant historical trajectories; count the total number of participant historical trajectories to generate the corresponding total number of first participants M1; and perform coordinate difference calculation on the adjacent second target coordinates in each participant's historical trajectory to generate the corresponding differential coordinate sequence; if the number of differential coordinates in the differential coordinate sequence is less than the number of first time points T1, padding with leading zeros is used to ensure that the differential coordinate sequence always has the number of differential coordinates of the first time point T1, and for each differential coordinate... Each participant is assigned a corresponding zero-padding state identifier; the horizontal difference coordinates Δx and vertical difference coordinates Δy of each difference coordinate, along with the corresponding zero-padding state identifiers, constitute a participant observation vector of shape 1*3; the participant observation vectors of the first time step number T1 corresponding to each participant's historical trajectory constitute a first observation tensor of shape T1*3; the first observation tensor is transposed to obtain a participant observation tensor of shape 3*T1; and the participant observation tensors of the obtained first total number of participants M1 constitute a first training tensor of shape M1*3*T1.

[0159] The differential coordinates include horizontal differential coordinates △x and vertical differential coordinates △y; the zero-padding status identifier includes a first identifier value and a second identifier value. If the zero-padding status identifier is the first identifier value, the corresponding differential coordinate type is zero-padding differential coordinates; if the zero-padding status identifier is the second identifier value, the corresponding differential coordinate type is true differential coordinates.

[0160] For example, the first drive test data sequence includes (T1+1)+T2=51 frames of first drive test data;

[0161] Suppose that the first drive test data from frame 1 to frame 51 includes two first target data groups (first target data group 1 and 2), and the first drive test data from frame 11 to frame 50 also includes a first target data group 3; wherein, the first target identifiers of first target data groups 1 and 2 are identifiers A and B, respectively, and the first target identifier of first target data group 3 is identifier C;

[0162] Given T1 = 20, suppose that in the first (20+1) = 21 frames of the first road test data sequence, the coordinates of the first target corresponding to the first target data groups 1 and 2 are respectively p A1,r p B1,r The corresponding second target coordinates are p A2,r p B2,r , 1≤r≤21; the first target coordinate 3 corresponds to the first target coordinate p. C1,r* The corresponding second target coordinates are p C2,r* ,11≤r * ≤21;

[0163] Therefore, sorting the coordinates of multiple second targets with the same first target identifier in the first 21 frames of the first road test data in chronological order yields the historical trajectories of three participants:

[0164] Participant's historical trajectory 1{p A2,r＝1 ,p A2,r＝2 ,p A2,r＝3 …p A2,r＝20 ,p A2,r＝21 The trajectory length of participant's historical trajectory 1 is 21;

[0165] Participant historical trajectory 2{p B2,r＝1 ,p B2,r＝2 ,p B2,r＝3 …p B2,r＝20 ,p B2,r＝21 The trajectory length of participant's historical trajectory 2 is 21;

[0166] Participant historical trajectory 3{p C2,r*＝11 ,p C2,r*＝12 ,p C2,r*＝13 …p C2,r*＝20 ,pC2,r*＝21 The trajectory length of participant's historical trajectory 3 is 11;

[0167] The total number of first participants, M1 = 3;

[0168] By performing coordinate difference calculations on the adjacent second target coordinates in the participant's historical trajectories 1, 2, and 3, three difference coordinate sequences can be obtained:

[0169] Difference coordinate sequence 1{△p A2,r＝1 =p A2,r＝2 -p A2,r＝1 ,…△p A2,r＝20 =p A2,r＝21 -p A2,r＝20}, the difference coordinates Δp in the difference coordinate sequence 1 A2,r The quantity is 20, which is equal to T1 = 20, so no zeros need to be added. That is to say, from △p A2,r＝1 To △p A2,r＝20 All are true differential coordinates, and the corresponding zero-padding status identifiers are all the second identifier value;

[0170] Difference coordinate sequence 2{△p B2,r＝1 =p B2,r＝2 -p B2,r＝1 ,…△p B2,r＝20 =p B2,r＝21 -p B2,r＝20}, the difference coordinates Δp in the difference coordinate sequence 2 B2,r The quantity is 20, which is equal to T1 = 20, so no zeros need to be added. That is to say, from △p B2,r＝1 To △p B2,r＝20 All are true differential coordinates, and the corresponding zero-padding status identifiers are all the second identifier value;

[0171] Difference coordinate sequence 3{△p C2,r*＝11 =p C2,r*＝12 -p C2,r*＝11 ,…△p C2,r*＝20 =p C2,r*＝21 -p C2,r*＝20}, the difference coordinates Δp in the difference coordinate sequence 3 C2,r* The quantity is 10, which is insufficient for T1=20. Zero-padding is needed in the difference coordinate sequence 3 to ensure that the difference coordinates Δp in the difference coordinate sequence 3 are equal. C2,r* With a quantity of 20, the zero-padded difference coordinate sequence 3 is:

[0172] {△p C2,r＝1 ,…△p C2,r＝10 ,△p C2,r＝11 …△p C2,r＝20},

[0173] From △p C2,r＝1 To △p C2,r＝10The zero-filled differential coordinates and the corresponding zero-filled state identifier are the first identifier value, △p C2,r＝11 To △p C2,r＝20 The true differential coordinates and the corresponding zero-padding status identifier are the second identifier value;

[0174] Substituting the horizontal difference coordinates Δx and the vertical difference coordinates Δy, which are included in the difference coordinates, into difference coordinate sequences 1 and 2 and the zero-padded difference coordinate sequence 3, we can obtain:

[0175] Difference coordinate sequence 1{(△x) A2,r＝1 ,△y A2,r＝1 ),…(△x A2,r＝20 ,△y A2,r＝20 )},

[0176] Difference coordinate sequence 2{(△x) B2,r＝1 ,△y B2,r＝1 ),…(△x B2,r＝20 ,△y B2,r＝20 )},

[0177] Difference coordinate sequence 3{(0,0),(0,0),(0,0),(0,0),(0,0),(0,0),(0,0),(0,0),(0,0),(0,0),(0,0),(△x C2,r＝11 ,△y C2,r＝11 )…(△x C2,r＝20 ,△y C2,r＝20 )};

[0178] Let the first identifier be 0 and the second identifier be 1, then,

[0179] From the difference coordinate sequence 1, we can obtain the observation vectors of 20 participants as follows: Participant observation vector A1(△x) A2,r＝1 ,△y A2,r＝1 (Second identifier value = 1)... Participant observation vector A20(△x) A2,r＝20 ,△y A2,r＝20 The second identifier value is 1); thus, the first observation tensor A corresponding to the identifier A is {participant observation vector A1, participant observation vector A2, ..., participant observation vector A20}; here, the shape of the participant observation vector A1 is 1*3, the shape of the first observation tensor A is 20*3, and the first observation tensor A is transposed to obtain the corresponding participant observation tensor A, then the shape of the participant observation tensor A is 3*20;

[0180] From the difference coordinate sequence 2, the observation vectors of the 20 participants can be obtained as follows: Participant observation vector B1(△x) B2,r＝1 ,△y B2,r＝1 (Second identifier value = 1) ... Participant observation vector B20(△x) B2,r＝20 ,△yB2,r＝20 The second identifier value is 1); thus, the first observation tensor B corresponding to the identifier B is {participant observation vector B1, participant observation vector B2, ..., participant observation vector B20}; here, the shape of the participant observation vector B1 is 1*3, the shape of the first observation tensor B is 20*3, and the transpose of the first observation tensor B yields the corresponding participant observation tensor B, so the shape of the participant observation tensor B is 3*20;

[0181] From the difference coordinate sequence 2, we can obtain the observation vectors of 20 participants as follows: participant observation vector C1(0,0, first identifier value = 0)...participant observation vector C10(0,0, first identifier value = 0), participant observation vector C11(△x)... C2,r＝11 ,△y C2,r＝11 (Second identifier value = 1) ... Participant observation vector C20(△x) C2,r＝20 ,△y C2,r＝20 The second identifier value is 1); thus, the first observation tensor C corresponding to the identifier C is {participant observation vector C1, participant observation vector C2, ..., participant observation vector C20}; here, the shape of the participant observation vector C1 is 1*3, the shape of the first observation tensor C is 20*3, and the first observation tensor C is transposed to obtain the corresponding participant observation tensor C, then the shape of the participant observation tensor C is 3*20;

[0182] The first training tensor is composed of the M1 = 3 participant observation tensors A, B, and C corresponding to the labels A, B, and C. The shape of the first training tensor is 3*3*20.

[0183] Step 35: Sort the coordinates of multiple second targets with the same first target identifier in the first road test data of the later T2 frame according to the time sequence to generate the corresponding real trajectory of the participant; if the number of second target coordinates in the real trajectory of the participant is less than the number of second time points T2, use a preset Kalman filter to predict the trajectory coordinates of the real trajectory of the participant in the future multiple time points to ensure that the real trajectory of the participant always has the number of second target coordinates of the second time points T2; and form the corresponding first real trajectory set by all the real trajectories of the participants obtained.

[0184] For example, continuing the above example, it is known that the first road test data sequence includes (T1+1)+T2 = 51 frames of first road test data, where T2 = 30. The first 29 frames of the last 30 frames of the first road test data sequence each include three first target data groups (first target data groups 1, 2, and 3), but the last frame of the first road test data only includes two first target data groups (first target data groups 1 and 2). The first target identifiers of first target data groups 1, 2, and 3 are A, B, and C, respectively, and the corresponding second target coordinates are p.A2,g p B2,g p C2,o , 22≤g≤51, 22≤o≤50;

[0185] Therefore, by sorting the coordinates of multiple second targets with the same first target identifier in the last 30 frames of the first road test data in chronological order, we can obtain the actual trajectories of the three participants:

[0186] Participant's actual trajectory 1{p A2,g＝22 ,p A2,g＝23 …p A2,g＝51 The number of second target coordinates in the participant's actual trajectory 1 is 30, which is equal to the number of second time points T2 = 30, so no trajectory supplementation is needed;

[0187] Participant's actual trajectory 2{p B2,g＝22 ,p B2,g＝23 …p B2,g＝51 The number of second target coordinates in the participant's actual trajectory 2 is 30, which is equal to the number of second time points T2 = 30, so no trajectory supplementation is needed;

[0188] Participant's actual trajectory 3{p C2,o＝22 ,p C2,o＝23 …p C2,o＝50 The number of second target coordinates in the participant's true trajectory 3 is 29, which is insufficient for the number of second time points T2 = 30. Therefore, a preset Kalman filter needs to be used to analyze the participant's true trajectory 3 at point p. C2,o＝50 The trajectory coordinates at the next time step are then predicted to ensure that the participant's true trajectory 3 always has the second time step number T2 = 30 second target coordinates; the participant's true trajectory 3 after prediction is {p C2,g＝22 ,p C2,g＝23 …p C2,g＝51}, where p C2,g＝51 The predicted coordinates of the second target;

[0189] Then, the real trajectories of all participants are combined into a first real trajectory set {participant real trajectory 1, participant real trajectory 2, participant real trajectory 3};

[0190] Here, the first set of real trajectories prepared in the current step is the ground truth data corresponding to the first training tensor prepared in step 34 above. In subsequent steps, it will be used to compare the predicted trajectory output by the LaneGCN model corresponding to the first training tensor.

[0191] Step 36: Query the preset high-precision map library, and extract the high-precision map that covers all the coordinates of the first vehicle and the first target according to the preset map size as the corresponding second scene map;

[0192] The second scene map includes multiple second lanes, and each second lane includes a center line.

[0193] Here, the input data of the map network module of the LaneGCN model and the input data of the participant network module have spatial correlation. Therefore, in preparing the corresponding training data, this embodiment of the invention searches for a high-precision map area that covers all the coordinates of the first vehicle and the first target from the known high-precision map library as a candidate area, and performs map cropping on the candidate area based on the pre-set map size information to obtain the corresponding second scene map.

[0194] Step 37: Divide each second lane into multiple corresponding second lane segments according to the preset lane segment length; record the two intersection points of each second lane segment with the corresponding second lane centerline as the corresponding second start point and second end point; use the coordinates of the second start point and second end point of each second lane segment in the second coordinate system as the corresponding second start point coordinates and second end point coordinates; form a lane node vector of shape 1*4 from the second start point coordinates and second end point coordinates of each second lane segment; count the total number of lane node vectors to generate a total of N3 lane node vectors; and form a second map tensor of shape N3*4 from the N3 lane node vectors.

[0195] Here, when processing segments, the segmentation starts from the beginning position of each second lane in the second scene map by default. Every other lane segment is cut off as the corresponding second lane segment. Each second lane segment must have two intersection points with the center line of the current lane's second lane. Here, the intersection point closer to the lane's beginning position is recorded as the second starting point, and the intersection point farther from the lane's beginning position is recorded as the second ending point. The feature vector of each lane segment is defined as a lane node vector. The content of the lane node vector is composed of the coordinates of the two feature points of the corresponding lane segment in the vehicle coordinate system, i.e., the second coordinate system. Therefore, the shape of the lane node vector is 1*4. The total number of lane segments on the second scene map, which is also the total number of lane node vectors, is N3. Therefore, combining N3 lane node vectors can obtain a feature tensor with a shape of N3*4, i.e., the second map tensor, which represents the current second scene map. The second map tensor prepared in this step is the lane data corresponding to the first training tensor prepared in step 34 above. In subsequent steps, it will be input into the LaneGCN model together with the first training tensor for multimodal trajectory prediction.

[0196] Step 38: The second training data, consisting of the first training tensor, the first set of real trajectories, and the second map tensor, is stored in the second training dataset.

[0197] Step 39: Calculate the total number of second training data in the second training dataset to generate a corresponding first quantity; if the first quantity is less than the preset first total threshold, return to step 31 to continue preparing training data; if the first quantity is equal to the first total threshold, stop preparing training data and output the second training dataset.

[0198] Here, the first total threshold is a pre-set total threshold for the second training dataset. As long as the total amount of training data in the second training dataset reaches this threshold, data preparation can be stopped and the subsequent steps can be taken to train the model. Otherwise, if the total amount of training data does not reach this threshold, the process should return to step 31 to continue preparing training data.

[0199] Step 4: Train the LaneGCN model using the second training dataset while keeping the network parameters of the map network module unchanged.

[0200] Specifically, this includes: Step 41, setting the loss function L of the LaneGCN model based on the number of steps T2 at the second time point. mod for:

[0201] L mod =L cls +αL reg ,

[0202]

[0203]

[0204] reg(Z)=∑ j d(z j ),

[0205]

[0206] Wherein, factor α is 1.0;

[0207] L cls For the classification loss function, L reg For regression loss function;

[0208] Classification loss function L cls In this context, ∈ represents a predefined marginal factor, m is the participant index (1 ≤ m ≤ total number of participants M), and the participant index m corresponds to the first target coordinate; k is the confidence level negative sample index, k * For the positive sample index of confidence level, 1≤k, k * ≤ Total number of modes K, but k ≠ k * ;c m,k For the K negative classification samples with first confidence scores of the m-th participant, c m,k*For the m-th participant, there are only one positive classification sample among the K first confidence scores; for each participant, there is only one positive classification sample among the K first confidence scores, and the remaining K-1 are all negative classification samples; the straight-line distance between the coordinates of the last first trajectory point vector of the first predicted trajectory tensor corresponding to the positive classification sample and the coordinates of the last second target of the corresponding participant's true trajectory is the shortest.

[0209] Prediction loss function L reg In this context, T2 is the preset number of second time points, which defaults to 30; t is the time index, 1≤t≤T2; reg() is the regression function, and d() is the smoothed L1 loss function. Let k be the index of the positive sample corresponding to the confidence level in the K first predicted trajectory tensors of the m-th participant. * The coordinates of the trajectory point of the t-th first trajectory point vector on the first predicted trajectory tensor; Let t be the coordinate of the second target in the actual trajectory of the m-th participant;

[0210] Here, the loss function L of the LaneGCN model mod For a detailed explanation, please refer to the paper "Learning Lane Graph Representations for Motion Forecasting";

[0211] Step 42: Extract the first second training data from the second training dataset as the corresponding current training data;

[0212] Step 43: Extract the first training tensor, the first real trajectory set, and the second map tensor from the current training data as the corresponding current training tensor, current real trajectory set, and current map tensor; and generate the total number of current participants M2 by counting the number of participant observation tensors in the current training tensor; and generate the total number of current lane node vectors N4 by counting the number of lane node vectors in the current map tensor.

[0213] Step 44: Input the current training tensor of shape M2*3*T1 into the participant network module for participant feature extraction to generate the corresponding second feature tensor of shape M2*128; input the current map tensor of shape N4*4 into the map network module for map lane feature extraction to generate the corresponding third feature tensor of shape N4*128; input the second and third feature tensors into the fusion network module for feature fusion to generate the corresponding fourth feature tensor of shape M2*128; and input the fourth feature tensor into the regression branch and classification branch of the prediction head network module respectively for multimodal trajectory prediction and multimodal confidence classification to generate the corresponding second prediction tensor of shape M2*K*T1*2 and the second confidence tensor of shape M2*K.

[0214] The second prediction tensor includes M2 second participant prediction tensors of shape K*T2*2; the second participant prediction tensor corresponds to the first target identifier; the second participant prediction tensor includes K second prediction trajectory tensors of shape T2*2; the second prediction trajectory tensor includes a second trajectory point vector of the second time point T2; the second trajectory point vector is a two-dimensional trajectory point coordinate.

[0215] The second confidence tensor includes M2 second confidence vectors of length K; the second confidence vectors correspond to the first target identifier; the second confidence vector includes K second confidences; the second confidences correspond one-to-one with the second predicted trajectory tensor, and the second confidence is the prediction confidence of the corresponding second predicted trajectory tensor;

[0216] For example, let the total number of modalities K = 4. Take the first training tensor {participant observation tensor A, B, C} composed of participant observation tensors A, B, and C in the previous example as the current training tensor, the first set of true trajectories {participant true trajectory 1, participant true trajectory 2, participant true trajectory 3} as the current true trajectory set, and the second map tensor as the current map tensor. Input the current training tensor and the current map tensor into the LaneGCN model and perform calculations through the participant network module, map network module, fusion network module, and prediction head network module to obtain a second prediction tensor with shape 3*4*20*2 and shape 3*4 with M2 = 3 and K = 4.

[0217] Therefore, the second prediction tensor includes three second participant prediction tensors of shape 4*30*2, namely second participant prediction tensors A, B, and C; each second participant prediction tensor contains four second prediction trajectory tensors of shape 30*2, namely second prediction trajectory tensors A1, A2, A3, A4, B1, B2, B3, B4, and C1, C2, C3, C4; each second prediction trajectory tensor (A1-A4, B1-B4, C1-C4) includes 30 second trajectory point vectors. That is, the coordinates of 30 two-dimensional trajectory points, 1≤k'≤K (i.e., 1≤k'≤4), 1≤m≤M2 (i.e., 1≤m≤3), 1≤t≤T2 (i.e., 1≤t≤30);

[0218] The second confidence tensor consists of three second confidence vectors of length 4, namely second confidence vectors A, B, and C; each second confidence vector contains four second confidence values ​​c. m，k′ ;

[0219] Step 45: Substitute the current total number of participants M2, the second prediction tensor, the second confidence tensor, and the current set of true trajectories into the loss function L of the LaneGCN model. mod Calculate the loss value to generate the corresponding second loss value;

[0220] Specifically, this includes: Step 451, traversing the real trajectories of each participant in the current real trajectory set; during traversal, the real trajectory of the currently traversed participant is taken as the current real trajectory, the last second target coordinate of the current real trajectory is taken as the current coordinate, and the first target identifier corresponding to the current real trajectory is taken as the current target identifier; and the straight-line distance between the trajectory point coordinates corresponding to the last second trajectory point vector of each second predicted trajectory tensor corresponding to the current target identifier and the current coordinate is calculated to generate the corresponding first straight-line distance; and the shortest of all the obtained first straight-line distances is selected as the corresponding current shortest distance, and the second predicted trajectory tensor corresponding to the current shortest distance is taken as the corresponding matching predicted trajectory tensor, and the modal branch index corresponding to the matching predicted trajectory tensor is taken as the corresponding matching modal index, and the corresponding second confidence is extracted from the second confidence tensor by the current target identifier + the matching modal index as the corresponding matching confidence; and the corresponding confidence positive sample index k is determined by the matching modal index. * The corresponding positive classification sample is determined by the matching confidence score. And the second confidence tensor that corresponds to the current target identifier and is not associated with positive classification samples The other second confidence scores for the matching samples are all set to the corresponding negative classification sample c. m，k ;

[0221] For example, iterating through the real trajectories of the three participants in the current real trajectory set;

[0222] When the current true trajectory is participant's true trajectory 1, the current coordinates are the coordinates of the last second target of participant's true trajectory 1, and the current target identifier is identifier A. The straight-line distances between the current coordinates and the coordinates of the trajectory points corresponding to the last second trajectory point vector of the four second predicted trajectory tensors (A1-A4) corresponding to identifier A are calculated to obtain four first straight-line distances. Assuming the first straight-line distance to the second predicted trajectory tensor A1 is the shortest, then the second predicted trajectory tensor A1 is the matching predicted trajectory tensor. The modal branch index corresponding to the matching predicted trajectory tensor is 1, i.e., the matching modal index is 1, and the corresponding confidence positive sample index k is... * =1, the positive classification sample extracted from the second confidence tensor that matches the modality index 1 of label A+ should be Then the negative classification sample c corresponding to label A m，k Then it includes c m＝1，k＝2 c m＝1，k＝3 c m＝1，k＝4 ;

[0223] When the current true trajectory is participant's true trajectory 2, the current coordinates are the coordinates of the last second target in participant's true trajectory 2, and the current target identifier is identifier B. The straight-line distances between the current coordinates and the coordinates of the trajectory points corresponding to the last second trajectory point vector in the four second predicted trajectory tensors (B1-B4) corresponding to identifier B are calculated to obtain four first straight-line distances. Assuming the first straight-line distance to the second predicted trajectory tensor B2 is the shortest, then the second predicted trajectory tensor B2 is the matching predicted trajectory tensor. The modal branch index corresponding to the matching predicted trajectory tensor is 2, i.e., the matching modal index is 2, and the corresponding confidence positive sample index k. * =2, the positive classification sample extracted from the second confidence tensor corresponding to the modality index 2 matching the label B+ should be Then the negative classification sample c corresponding to label B m，k Then it includes c m＝2，k＝1 c m＝2，k＝3 c m＝2，k＝4 ;

[0224] When the current true trajectory is participant's true trajectory 3, the current coordinates are the coordinates of the last second target in participant's true trajectory 3, and the current target identifier is identifier C. The straight-line distances between the current coordinates and the coordinates of the trajectory points corresponding to the last second trajectory point vector of the four second predicted trajectory tensors (C1-C4) corresponding to identifier C are calculated to obtain four first straight-line distances. Assuming the first straight-line distance to the second predicted trajectory tensor C3 is the shortest, then the second predicted trajectory tensor C3 is the matching predicted trajectory tensor. The modal branch index corresponding to the matching predicted trajectory tensor is 3, i.e., the matching modal index is 3, and the corresponding confidence positive sample index k. * =3, the positive classification sample extracted from the second confidence tensor corresponding to the modality index 3 matching the label C+ should be 3. Then the negative classification sample c corresponding to the identifier C m，k Then it includes c m＝3，k＝1 c m＝3，k＝2 c m＝3，k＝4 ;

[0225] Step 452: Select the positive sample index k corresponding to the confidence level from each second participant's prediction tensor. * The corresponding second predicted trajectory tensor is used as the corresponding positive sample predicted trajectory tensor, and the vectors of each second trajectory point in the positive sample predicted trajectory tensor are used as the vectors of the second trajectory points. As the corresponding positive sample trajectory point vector

[0226] For example, given an identifier A, the confidence level of the positive sample index k is known. * =1, then the second predicted trajectory tensor A1 is the positive sample predicted trajectory tensor of participant A; then, the vectors of each second trajectory point in the second predicted trajectory tensor A1 That is, the vector of positive sample trajectory points corresponding to A.

[0227] Given that for identifier B, the confidence level is positive and the sample index k is positive. * = 2, then the second predicted trajectory tensor B2 is the positive sample predicted trajectory tensor of participant B; then, the vectors of each second trajectory point in the second predicted trajectory tensor B2 That is, the vector of positive sample trajectory points corresponding to identifier B.

[0228] Given an identifier C, the confidence level and the positive sample index k * =3, then the second predicted trajectory tensor C3 is the positive sample predicted trajectory tensor of participant C; then, the vectors of each second trajectory point in the second predicted trajectory tensor C3 That is, the vector of positive sample trajectory points corresponding to identifier B.

[0229] Step 453: Use the second target coordinates of the real trajectory of each participant in the current real trajectory set as the corresponding real trajectory point vector.

[0230] For example, continuing the previous example, it is known that the current set of real trajectories includes participant's real trajectories 1, 2, and 3;

[0231] For participant's actual trajectory 1, extract the coordinates of each second target as the corresponding actual trajectory point vector.

[0232] For participant's actual trajectory 2, extract the coordinates of each second target as the corresponding actual trajectory point vector.

[0233] For participant's actual trajectory 3, extract the coordinates of each second target as the corresponding actual trajectory point vector.

[0234] Step 455, obtain all positive classification samples negative classification sample c m，k Positive sample trajectory point vector and the vector of the true trajectory point And the current total number of participants M2 are substituted into the loss function L. mod Calculate the loss value to generate the corresponding second loss value;

[0235] For example, the total number of participants M2 = 3 and all positive classification samples obtained. negative classification sample c m，k (c m＝1，k＝2 c m＝1，k＝3 c m＝1，k＝4 c m＝2，k＝1 c m＝2，k＝3 c m＝2，k＝4 c m＝3，k＝1 c m＝3，k＝2 c m＝3，k＝4 ), positive sample trajectory point vector and the vector of the true trajectory point Substitute into the loss function L mod ,

[0236] L mod =L cls +1.0*L reg ,

[0237]

[0238]

[0239] Solving the reg() function is similar to step 25 above, and will not be further broken down here; from the above calculation steps, it can be seen that the loss function L mod Since all parameters are known data, the corresponding second loss value can be calculated.

[0240] Step 46: Identify whether the second loss value meets the preset convergence range of the second loss value; if the second loss value does not meet the convergence range of the second loss value, then perform back modulation on the network parameters of each module in the LaneGCN model except the map network module in the direction that makes the second loss function reach the minimum value, and return to step 44 to continue training when the back modulation is completed; if the second loss value meets the convergence range of the loss value, then identify whether the current training data is the last second training data in the second training dataset. If it is not the last, then extract the next second training data as the new current training data and return to step 43 to continue training. If it is the last, then stop the model training and confirm that the model training is successful.

[0241] Here, the convergence range of the second loss value is a preset reasonable error range of the loss value. If the second loss value does not meet the convergence range of the second loss value, it means that the current training has not yet converged. Then, the network parameters of each module in the LaneGCN model, except for the map network module, are back-modulated and the training is returned to step 44 after modulation. If the second loss value meets the convergence range of the second loss value, it means that the training based on the current training data has ended. The next training data, that is, the next second training data of the second training dataset, needs to be retrieved for training until all the second training data of the second training dataset has been used for training and has achieved convergence.

[0242] In summary, through the training data preparation and model training in steps 3-4 of this embodiment of the invention, the LaneGCN model is trained based on a pre-trained mature map network module, which reduces the overall training difficulty of the LaneGCN model and improves the overall output accuracy of the LaneGCN model.

[0243] Figure 2 This is a block diagram of a model training processing device provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a first training data preparation module 201, a first model training module 202, a second training data preparation module 203, and a second model training module 204.

[0244] The first training data preparation module 201 is used to obtain high-precision maps and construct the first training dataset.

[0245] The first model training module 202 is used to perform map network pre-training on the map network module of the LaneGCN model based on the first training dataset.

[0246] The second training data preparation module 203 is used to acquire vehicle road test data and construct a second training dataset when the map network pre-training is successful.

[0247] The second model training module 204 is used to train the LaneGCN model based on the second training dataset while keeping the network parameters of the map network module unchanged.

[0248] The model training processing device provided in this embodiment of the invention can execute the method steps in the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0249] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the first training data preparation module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0250] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0251] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0252] Figure 3 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be the aforementioned terminal device or server, or it can be a terminal device or server connected to the aforementioned terminal device or server that implements the method of the embodiments of the present invention. Figure 3As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0253] exist Figure 3 The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0254] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0255] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0256] This invention also provides a chip for executing instructions, which is used to perform the processing steps described in the foregoing method embodiments.

[0257] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for model training. Before performing overall model training on the LaneGCN model, a high-precision map training dataset and loss function L are first applied. map The map network module is pre-trained to achieve sufficient training results; then, while keeping the network parameters of the map network module unchanged, the loss function L is applied. mod =L cls +αL reg The LaneGCN model is trained as a whole. This invention reduces the training difficulty of the overall model, improves the training maturity of the map network module, and enhances the output accuracy of the LaneGCN model.

[0258] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0259] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0260] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for processing model training, characterized in that, The method includes: Obtain high-precision maps to construct the first training dataset; The map network module of the LaneGCN model is pre-trained based on the first training dataset. If the map network is successfully pre-trained, then vehicle road test data is acquired to construct a second training dataset. While keeping the network parameters of the map network module unchanged, the LaneGCN model is trained based on the second training dataset. The first training dataset includes multiple first map tensors; the first map tensors include multiple lane node vectors; the lane node vectors have a 1*4 shape and are composed of a set of corresponding first starting point coordinates (x, y, y). s ,y s ) and the coordinates of the first endpoint (x) e ,y e )composition; Specifically, the step of pre-training the map network module of the LaneGCN model based on the first training dataset includes: Step 41: Extract the map network module separately and connect it with the preset first multilayer perception network to form the corresponding first training network; Step 42: Extract one of the first map tensors from the first training dataset as the corresponding current map tensor; Step 43: Statistically count the total number of lane node vectors in the current map tensor to generate the corresponding total number of lane nodes N0; calculate the current total number of input lane nodes N1 = N0 * (1-β) and the total number of lane nodes to be tested N2 = N0 * β based on a preset first occlusion ratio β; extract the lane node vectors of the total number of input lane nodes N1 from the current map tensor to form a current input tensor with a shape of N1 * 4; and form a current label tensor with a shape of N2 * 4 from the remaining total number of lane nodes to be tested N2; and use each lane node vector in the current label tensor as the corresponding lane node to be tested Ln. i ; 1≤i≤N2; The first occlusion ratio β is defaulted to 20%; Step 44, for each of the lane nodes to be tested Ln i truth center The coordinates are determined; the true center point The coordinates are ( , ), x-axis The first starting point coordinate (x) of the corresponding lane node vector s ,y s ) and the first endpoint coordinates (x) e ,y e The average of the x-axis of ) s +x e ) / 2, ordinate For the corresponding first starting point coordinates (x) s ,y s ) and the first endpoint coordinates (x) e ,y e The average value of the ordinate (y) s +y e ) / 2; Step 45, based on the total number of lane nodes N2 to be tested and each of the true center points The corresponding first loss function is set as follows: , ， ， ； s i The lane node to be tested, Ln i The prediction center point, the prediction center point s i The coordinates are (x i ,y i ), reg() is the regression function, d() is the smoothing L1 loss function, for norm expression, for The norm expression; Step 45: Input the current input tensor and the current label tensor of shape N1*4 into the first training network. The map network module performs map lane feature extraction processing on the current input tensor to generate a corresponding first training tensor of shape N1*128. The first multilayer perceptron then uses the first training tensor and the current label tensor to compare each lane node Ln to be tested with the current label tensor. i The corresponding center point coordinates are used to perform regression prediction to generate a second training tensor of shape N2*2; the second training tensor includes N2 predicted center points s i coordinates (x) i ,y i ); Step 46, connect each of the lane nodes Ln to be tested. i The corresponding prediction center point s i coordinates (x) i ,y i and the truth center point coordinates ( , Substitute the first loss function into the first loss function to calculate the corresponding first loss value; Step 47: Identify whether the first loss value meets the preset first loss value convergence range; if the first loss value does not meet the first loss value convergence range, then perform reverse modulation on the network parameters of the map network module along the direction that makes the first loss function reach the minimum value, and return to step 45 to continue training when the reverse modulation is completed; if the first loss value meets the first loss value convergence range, then identify whether the current map tensor is the last first map tensor in the first training dataset; if it is not the last, then extract the next first map tensor as the new current map tensor and return to step 43 to continue training; if it is the last, then stop model training and confirm that the map network pre-training is successful.

2. The model training processing method according to claim 1, characterized in that, The LaneGCN model includes a participant network module, a map network module, a fusion network module, and a prediction head network module; the fusion network module is connected to the participant network module, the map network module, and the prediction head network module, respectively. The participant network module is used to perform participant feature extraction processing on the first input tensor of shape M*3*T1 to generate a corresponding first output tensor of shape M*128; and send the first output tensor to the fusion network module; the first input tensor includes M participant observation tensors of shape 3*T1; M is the total number of participants; T1 is the preset number of first time moments, which is 20 by default; The map network module is used to perform map lane feature extraction processing on the second input tensor of shape N*4 to generate a corresponding second output tensor of shape N*128; and send the second output tensor to the fusion network module; the second input tensor includes N lane node vectors of shape 1*4; N is the total number of lane node vectors; The fusion network module is used to perform feature fusion processing on the first output tensor and the second output tensor to generate a first fused feature tensor with a corresponding shape of M*128; and send the first fused feature tensor to the prediction head network module. The prediction head network module includes a regression branch and a classification branch; the regression branch is used to perform multimodal trajectory prediction processing based on the first fusion feature tensor to generate a first prediction tensor with a corresponding shape of M*K*T2*2; the classification branch is used to perform multimodal confidence classification processing based on the first fusion feature tensor to generate a first confidence tensor with a corresponding shape of M*K; K is the total number of modes; T2 is the preset number of second time points, which defaults to 30. The first prediction tensor includes M first participant prediction tensors of shape K*T2*2; each first participant prediction tensor corresponds one-to-one with the M participants; each first participant prediction tensor includes K first prediction trajectory tensors of shape T2*2; each first prediction trajectory tensor includes a first trajectory point vector of the second time step number T2; the first trajectory point vector is a two-dimensional trajectory point coordinate. The first confidence tensor includes M first confidence vectors of length K; each first confidence vector corresponds one-to-one with one of the M participants; each first confidence vector includes K first confidence values; each first confidence value corresponds one-to-one with the first predicted trajectory tensor, and the first confidence value is the prediction confidence value of the corresponding first predicted trajectory tensor.

3. The model training processing method according to claim 2, characterized in that, The process of acquiring high-precision maps and constructing the first training dataset specifically includes: The first scene map set is composed of high-precision map raw data selected from a publicly available high-precision map database according to preset map filtering rules. The map filtering rules include minimum map quantity, minimum map size, maximum map accuracy, minimum number of lanes, minimum lane length, and lane segment length. The first scene map set includes multiple first scene maps. The number of first scene maps is not greater than the maximum number of maps. The map size of the first scene map is not less than the minimum map size, and the map accuracy is not less than the minimum map accuracy. The first scene map includes multiple first lanes, the number of first lanes is not less than the minimum number of lanes, and the lane length is not less than the minimum lane length. Each first lane includes a first lane centerline. According to the lane segment length of the map filtering rules, each first lane of each first scene map is segmented to obtain a plurality of corresponding first lane segments; and the two intersection points of each first lane segment and the corresponding first lane centerline are recorded as the corresponding first starting point and first ending point; On each of the first scene maps, the first starting point of any starting lane segment of the first lane is selected as the origin, and a two-dimensional Cartesian coordinate system is constructed with the forward direction of the starting lane segment as the positive X-axis, denoted as the corresponding first coordinate system; and the coordinates of the first starting point and the first ending point of each of the first lane segments on the current first scene map on the first coordinate system are taken as the corresponding first starting point coordinates (x... s ,y s ) and the coordinates of the first endpoint (x) e ,y e ); and the coordinates (x) of the first starting point of each of the first lane segments. s ,y s ) and the first endpoint coordinates (x) e ,y e The lane node vectors are composed of a 1x4 shape; and all the obtained lane node vectors are used to form a first map tensor; the first map tensor has an N shape. ’ *4, N ’ The total number of the lane node vectors; The first training dataset is composed of all the first map tensors obtained.

4. The model training processing method according to claim 2, characterized in that, The process of acquiring vehicle road test data to construct the second training dataset specifically includes: Step 51: Select a first road test data sequence from a preset vehicle road test database; the first road test data sequence consists of (T1+1)+T2 frames of first road test data that are sequentially continuous and have completed target association; the first road test data includes a first timestamp, a first vehicle coordinate, and multiple first target data groups, each first target data group including a first target identifier and a first target coordinate; the first target identifiers of the two first target data groups corresponding to the associated targets in the first road test data at different times are the same; Step 52: Using the first vehicle coordinates of the first road test data in frame (T1+1) as the first reference point coordinates, and using the first timestamp corresponding to the first reference point coordinates minus 1 second as the first time, and using the first vehicle coordinates corresponding to the first timestamp that is earlier than the first time and closest to the first time as the second reference point coordinates; using the first reference point coordinates as the origin, and using the direction from the second reference point coordinates to the first reference point coordinates as the positive X-axis, construct a two-dimensional Cartesian coordinate system, denoted as the corresponding second coordinate system; Step 53: Perform coordinate transformation on all the first vehicle coordinates and the first target coordinates in the second coordinate system to generate the corresponding second vehicle coordinates and second target coordinates; Step 54: Sort the coordinates of multiple second targets with the same first target identifier in the first road test data of the previous (T1+1) frames according to time sequence to generate corresponding participant historical trajectories; and count the total number of participant historical trajectories to generate the corresponding first participant total number M1; and perform coordinate difference calculation on adjacent second target coordinates in each participant historical trajectory to generate corresponding differential coordinate sequences; if the number of differential coordinates in the differential coordinate sequence is less than the number of first time points T1, it is padded with leading zeros to ensure that the differential coordinate sequence always has the number of differential coordinates of the first time points T1, and assign a corresponding zero-padding status identifier to each differential coordinate; and the horizontal differential coordinate Δx and vertical differential coordinate Δy of each differential coordinate and the corresponding zero-padding status identifier constitute a pair of differential coordinates. The participant observation vectors are of shape 1*3; and the participant observation vectors corresponding to the first time point T1 of each participant's historical trajectory form a first observation tensor of shape T1*3; the first observation tensor is transposed to obtain a participant observation tensor of shape 3*T1; and the participant observation tensors of the obtained first total number of participants M1 form a first training tensor of shape M1*3*T1; the differential coordinates include the horizontal differential coordinates Δx and the vertical differential coordinates Δy; the zero-padding status identifier includes a first identifier value and a second identifier value. If the zero-padding status identifier is the first identifier value, the type of the corresponding differential coordinates is zero-padding differential coordinates; if the zero-padding status identifier is the second identifier value, the type of the corresponding differential coordinates is true differential coordinates. Step 55: Sort the coordinates of multiple second targets with the same first target identifier in the first road test data of the later T2 frame according to the time sequence to generate the corresponding participant real trajectory; if the number of second target coordinates in the participant real trajectory is less than the number of second time points T2, use a preset Kalman filter to predict the trajectory coordinates of the participant real trajectory in the future multiple time points to ensure that the participant real trajectory always has the number of second target coordinates of the second time points T2; and form a corresponding first real trajectory set from all the obtained participant real trajectories; Step 56: Query the preset high-precision map library, and extract the high-precision map that covers all the coordinates of the first vehicle and the first target according to the preset map size as the corresponding second scene map; the second scene map includes multiple second lanes, and the second lane includes the center line of the second lane; Step 57: Divide each second lane into multiple corresponding second lane segments according to the preset lane segment length; and record the two intersection points of each second lane segment with the corresponding second lane centerline as the corresponding second start point and second end point; and use the coordinates of the second start point and second end point of each second lane segment in the second coordinate system as the corresponding second start point coordinates and second end point coordinates; and form a corresponding lane node vector of shape 1*4 by the second start point coordinates and second end point coordinates of each second lane segment; and generate a corresponding total number of lane node vectors N3 by counting the total number of lane node vectors; and form a corresponding second map tensor of shape N3*4 by the N3 lane node vectors. Step 58: The second training data, composed of the first training tensor, the first real trajectory set, and the second map tensor, is stored in the second training dataset. Step 59: Calculate the total number of the second training data in the second training dataset to generate a corresponding first quantity; if the first quantity is less than a preset first total threshold, return to step 51 to continue preparing training data; if the first quantity is equal to the first total threshold, stop preparing training data and output the second training dataset.

5. The model training processing method according to claim 4, characterized in that, The step of training the LaneGCN model based on the second training dataset specifically includes: Step 61: Set the loss function L of the LaneGCN model according to the second time step number T2. mod for: ， , , , ； Wherein, factor α is 1.0; L cls For the classification loss function, L reg For regression loss function; The classification loss function L cls middle, Let m be a preset marginal factor, where 1 ≤ m ≤ total number of participants M, and the participant index m corresponds to the first target coordinate; k is the confidence level negative sample index, k * For the positive sample index of confidence level, 1≤k, k * ≤ Total number of modes K, but k ≠ k * c m,k For the m-th participant, there are K negative classification samples from the first confidence level. The positive classification sample is one of the K first confidence scores of the m-th participant; only one of the K first confidence scores of each participant is the positive classification sample, and the remaining K-1 are the negative classification samples; the straight-line distance between the trajectory point coordinates of the last first trajectory point vector of the first predicted trajectory tensor corresponding to the positive classification sample and the last second target coordinates of the corresponding participant's true trajectory is the shortest. The regression loss function L reg In this context, T2 is the preset number of second time points, which defaults to 30; t is the time index, 1≤t≤T2; reg() is the regression function, and d() is the smoothed L1 loss function; For the K first predicted trajectory tensors of the m-th participant, the index k of the positive sample corresponding to the confidence level is... * The coordinates of the trajectory point of the t-th first trajectory point vector on the first predicted trajectory tensor; Let t be the coordinate of the second target in the actual trajectory of the m-th participant; Step 62: Extract the first piece of the second training data from the second training dataset as the corresponding current training data; Step 63: Extract the first training tensor, the first real trajectory set, and the second map tensor from the current training data as the corresponding current training tensor, current real trajectory set, and current map tensor; and generate the current total number of participants M2 by counting the number of participant observation tensors in the current training tensor; and generate the current total number of lane node vectors N4 by counting the number of lane node vectors in the current map tensor. Step 64: Input the current training tensor of shape M2*3*T1 into the participant network module for participant feature extraction processing to generate a corresponding second feature tensor of shape M2*128; input the current map tensor of shape N4*4 into the map network module for map lane feature extraction processing to generate a corresponding third feature tensor of shape N4*128; input the second and third feature tensors into the fusion network module for feature fusion processing to generate a corresponding fourth feature tensor of shape M2*128; and input the fourth feature tensor into the regression branch and the classification branch of the prediction head network module respectively for corresponding multimodal trajectory prediction processing and multimodal confidence classification processing to generate a corresponding second prediction tensor of shape M2*K*T1*2 and a second confidence tensor of shape M2*K. The second prediction tensor includes M2 second participant prediction tensors of shape K*T2*2; the second participant prediction tensor corresponds to the first target identifier; the second participant prediction tensor includes K second prediction trajectory tensors of shape T2*2; the second prediction trajectory tensor includes a second trajectory point vector of the second time point T2; the second trajectory point vector is a two-dimensional trajectory point coordinate. The second confidence tensor includes M2 second confidence vectors of length K; the second confidence vectors correspond to the first target identifier; the second confidence vector includes K second confidences; the second confidences correspond one-to-one with the second predicted trajectory tensor, and the second confidence is the prediction confidence of the corresponding second predicted trajectory tensor; Step 65: Substitute the current total number of participants M2, the second prediction tensor, the second confidence tensor, and the current set of true trajectories into the loss function L of the LaneGCN model. mod Calculate the loss value to generate the corresponding second loss value; Step 66: Identify whether the second loss value meets the preset convergence range of the second loss value; if the second loss value does not meet the convergence range of the second loss value, then perform back modulation on the network parameters of each module in the LaneGCN model except the map network module along the direction that makes the second loss function reach its minimum value, and return to step 64 to continue training when the back modulation is completed; if the second loss value meets the convergence range of the loss value, then identify whether the current training data is the last second training data in the second training dataset; if it is not the last, then extract the next second training data as the new current training data and return to step 63 to continue training; if it is the last, then stop the model training and confirm that the model training is successful.

6. An apparatus for implementing the processing method for model training according to any one of claims 1-5, characterized in that, The device includes: a first training data preparation module, a first model training module, a second training data preparation module, and a second model training module; The first training data preparation module is used to acquire high-precision maps to construct the first training dataset; The first model training module is used to perform map network pre-training on the map network module of the LaneGCN model based on the first training dataset. The second training data preparation module is used to acquire vehicle road test data to construct a second training dataset when the map network pre-training is successful; The second model training module is used to train the LaneGCN model based on the second training dataset while keeping the network parameters of the map network module unchanged.

7. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-5; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-5.

Images