A processing method for predicting a state and a category of an obstacle based on multi-class perception features

Abstract

Embodiments of the present application relate to a processing method for predicting obstacle state and category based on multi-category perception features, which comprises: acquiring perception data sequences of a vehicle-mounted laser radar, a vehicle-mounted millimeter wave radar and a vehicle-mounted camera in a first nearest time period T; dividing the first time period T into a plurality of first time points according to a preset first time interval Δt, and taking the number of the first time points as the corresponding time point number L; performing multi-category perception source data preprocessing according to the laser radar point cloud, the millimeter wave radar point cloud, the image sequence, all the first time points and the time point number L; inputting a multi-category perception source time feature tensor set and a multi-category perception source time mask tensor into a preset target state understanding and classification network to perform obstacle target state prediction and obstacle target category prediction processing. Through the present application, end-to-end state+category synchronous prediction can be realized based on an overall solution, and the prediction accuracy can also be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a processing method for predicting obstacle state and category based on multi-sensing feature. BACKGROUND

[0002] There are two common obstacle state prediction methods at present: rule-based state prediction method and deep learning-based state prediction method. The rule-based prediction method is similar to using Kalman filter to predict the obstacle state by establishing observation-state equation set, and the deep learning-based prediction method is similar to using deep learning model to predict the state. Compared with the rule-based prediction method, the deep learning-based prediction method is simpler to implement, but has lower accuracy. There are two common obstacle classification prediction methods at present: point cloud-based classification prediction method and image-based classification prediction method. The former is prone to have low type recognition rate, and the latter is prone to have inaccurate target positioning. Moreover, in the current common application scenarios, obstacle state prediction and obstacle category prediction cannot be solved by an overall solution. Therefore, how to realize end-to-end state+category synchronous prediction and improve prediction accuracy by an overall solution is a problem to be solved by the present application. SUMMARY

[0003] The purpose of the present application is to provide a processing method for predicting obstacle state and category based on multi-sensing feature, electronic equipment and computer readable storage medium, which fuses multi-sensing data (laser radar, millimeter wave radar, camera, etc.) and multi-data features (basic data features, target detection features), constructs a target state understanding and classification network to perform end-to-end state+category synchronous prediction, and inputs the fused features into the target state understanding and classification network to obtain the state and category prediction results of the obstacle. Through the present application, end-to-end state+category synchronous prediction can be realized by an overall solution, and prediction accuracy can be improved based on multi-sensing feature.

[0004] To achieve the above purpose, the first aspect of the embodiment of the present application provides a processing method for predicting obstacle state and category based on multi-sensing feature, which comprises:

[0005] Obtain the sensing data sequence of the vehicle-mounted laser radar, vehicle-mounted millimeter wave radar and vehicle-mounted camera in the first time period T as the corresponding first laser radar point cloud sequence, first millimeter wave radar point cloud sequence and first image sequence;

[0006] Divide the first time period T into a plurality of first time points according to a preset first time interval At, and take the number of the first time points as the corresponding time point number L.

[0007] performing multi-class perception source data preprocessing on the first laser radar point cloud sequence, the first millimeter wave radar point cloud sequence, the first image sequence, all the first time points and the time point quantity L to obtain a corresponding multi-class perception source time feature tensor set and a corresponding multi-class perception source time mask tensor;

[0008] inputting the multi-class perception source time feature tensor set and the multi-class perception source time mask tensor into a preset target state understanding and classification network to perform obstacle target state prediction and obstacle target category prediction processing to obtain a corresponding first obstacle prediction data set; the first obstacle prediction data set includes a plurality of first obstacle prediction data; the first obstacle prediction data includes first obstacle state data and first obstacle type data.

[0009] Preferably, the first laser radar point cloud sequence includes a plurality of first laser radar point clouds, each of which corresponds to a first sampling time point;

[0010] The first millimeter wave radar point cloud sequence includes a plurality of first millimeter wave radar point clouds, each of which corresponds to a second sampling time point;

[0011] The first image sequence includes a plurality of first images, each of which corresponds to a third sampling time point;

[0012] The multi-class perception source time feature tensor set includes a first point cloud time feature tensor, a first point cloud target time feature tensor, a second point cloud target time feature tensor, a first image time feature tensor and a first image target time feature tensor; the shape of the first point cloud time feature tensor is B×L×F1, the shape of the first point cloud target time feature tensor is B×L×F2, the shape of the second point cloud target time feature tensor is B×L×F3, the shape of the first image time feature tensor is B×L×F4, and the shape of the first image target time feature tensor is B×L×F5; the number of obstacles B≥1, and the feature dimensions F1, F2, F3, F4 and F5≥1;

[0013] The shape of the multi-class perception source time mask tensor is B×L×N, and the number of perception sources N is 5 by default; the multi-class perception source time mask tensor is fused from the time mask tensors with the shape of B×L of the number of perception sources N, which are respectively: a first point cloud time mask tensor, a first point cloud target time mask tensor, a second point cloud target time mask tensor, a first image time mask tensor and a first image target time mask tensor.

[0014] Preferably, the multi-class perception source data preprocessing according to the first lidar point cloud sequence, the first millimeter wave radar point cloud sequence, the first image sequence, all the first time points and the time point number L obtains a corresponding multi-class perception source time feature tensor set and a corresponding multi-class perception source time mask tensor, specifically including:

[0015] The lidar point cloud data preprocessing according to the first lidar point cloud sequence, all the first time points and the time point number L obtains corresponding first point cloud time feature tensors, first point cloud target time feature tensors, first point cloud time mask tensors and first point cloud target time mask tensors;

[0016] The millimeter wave radar point cloud data preprocessing according to the first millimeter wave radar point cloud sequence, all the first time points and the time point number L obtains corresponding second point cloud target time feature tensors and second point cloud target time mask tensors;

[0017] The camera image data preprocessing according to the first image sequence, all the first time points and the time point number L obtains corresponding first image time feature tensors, first image target time feature tensors, first image time mask tensors and first image target time mask tensors;

[0018] The first point cloud time feature tensors, the first point cloud target time feature tensors, the second point cloud target time feature tensors, the first image time feature tensors and the first image target time feature tensors are combined to obtain a corresponding multi-class perception source time feature tensor set;

[0019] The first point cloud time mask tensors, the first point cloud target time mask tensors, the second point cloud target time mask tensors, the first image time mask tensors and the first image target time mask tensors are tensor fused to generate a corresponding multi-class perception source time mask tensor with a shape of BxLxN;

[0020] The multi-class perception source time feature tensor set and the multi-class perception source time mask tensor obtained are output as a multi-class perception source data preprocessing result.

[0021] Further, the lidar point cloud data preprocessing according to the first lidar point cloud sequence, all the first time points and the time point number L obtains corresponding first point cloud time feature tensors, first point cloud target time feature tensors, first point cloud time mask tensors and first point cloud target time mask tensors, specifically including:

[0022] performing point cloud feature extraction processing on each of the first lidar point clouds of the first lidar point cloud sequence to obtain a corresponding first point cloud feature tensor with a shape of BxF1; and performing obstacle target detection processing on each of the first point cloud feature tensors to obtain a corresponding first point cloud target feature tensor with a shape of BxF2; the first point cloud feature tensor and the first point cloud target feature tensor both correspond one-to-one to the first sampling time points;

[0023] initializing a tensor sequence composed of L zero tensors with a shape of BxF1 as a corresponding first tensor sequence; and initializing a tensor sequence composed of L zero tensors with a shape of BxF2 as a corresponding second tensor sequence; the first tensor sequence includes L first tensors; the first tensor has a shape of BxF1, is a zero tensor, and corresponds one-to-one to the first time points; the second tensor sequence includes L second tensors; the second tensor has a shape of BxF2, is a zero tensor, and corresponds one-to-one to the first time points;

[0024] performing traversal on each of the first tensors of the first tensor sequence; during traversal, taking the first tensor being currently traversed as a corresponding current tensor, taking the first time point corresponding to the current tensor as a corresponding current time point, and confirming whether the current time point has a matching first sampling time point; if it is confirmed that the current time point has a matching first sampling time point, updating the current tensor to the first point cloud feature tensor corresponding to the matching first sampling time point; and after traversal, performing tensor conversion on the first tensor sequence according to a set tensor shape BxLxF1 to obtain a corresponding first point cloud time feature tensor;

[0025] performing traversal on each of the second tensors of the second tensor sequence; during traversal, taking the second tensor being currently traversed as a corresponding current tensor, taking the first time point corresponding to the current tensor as a corresponding current time point, and confirming whether the current time point has a matching first sampling time point; if it is confirmed that the current time point has a matching first sampling time point, updating the current tensor to the first point cloud target feature tensor corresponding to the matching first sampling time point; and after traversal, performing tensor conversion on the second tensor sequence according to a set tensor shape BxLxF2 to obtain a corresponding first point cloud target time feature tensor;

[0026] initializing two all-zero tensors with shapes of BxL as the first point cloud time mask tensor and the first point cloud target time mask tensor; the first point cloud time mask tensor comprises first column vectors with lengths of the obstacle number B for the time point number L; the first point cloud target time mask tensor comprises second column vectors with lengths of the obstacle number B for the time point number L; the first column vectors and the second column vectors both correspond to the first time points one by one;

[0027] iterating through each of the first time points; when iterating, taking the first time point currently iterated as a corresponding current time point; and confirming whether the current time point has a matching first sampling time point; if it is confirmed that the current time point has a matching first sampling time point, updating the first column vector corresponding to the current time point in the first point cloud time mask tensor as an all-1 column vector, and updating the second column vector corresponding to the current time point in the first point cloud target time mask tensor as an all-1 column vector;

[0028] outputting the obtained first point cloud time feature tensor, the first point cloud target time feature tensor, the first point cloud time mask tensor and the first point cloud target time mask tensor as a laser radar point cloud data preprocessing result.

[0029] Further, the millimeter wave radar point cloud data preprocessing according to the first millimeter wave radar point cloud sequence, all the first time points and the time point number L obtains corresponding second point cloud target time feature tensors and second point cloud target time mask tensors, and specifically includes:

[0030] performing point cloud feature extraction processing on each of the first millimeter wave radar point clouds of the first millimeter wave radar point cloud sequence and obtaining a corresponding second point cloud target feature tensor with a shape of BxF3 based on the extracted features and obstacle target detection processing; the second point cloud target feature tensor corresponds to the second sampling time point one by one;

[0031] initializing a tensor sequence composed of all-zero tensors with shapes of BxF3 for the time point number L as a corresponding third tensor sequence; the third tensor sequence comprises third tensors for the time point number L; the third tensors have shapes of BxF3, the third tensors are all-zero tensors, and the third tensors correspond to the first time points one by one;

[0032] traversing each of the third tensors of the third tensor sequence; when traversing, taking the third tensor currently traversed as a corresponding current tensor, and taking the first time point corresponding to the current tensor as a corresponding current time point; and confirming whether the current time point has a matching second sampling time point; if it is confirmed that the current time point has a matching second sampling time point, updating the current tensor to the second point cloud target feature tensor corresponding to the matching second sampling time point; when the traversal ends, performing tensor conversion on the third tensor sequence according to a set tensor shape B x L x F3 to obtain a corresponding second point cloud target time feature tensor;

[0033] initializing a full-zero tensor with a shape of B x L as a corresponding second point cloud target time mask tensor; the second point cloud target time mask tensor includes third column vectors with a length of the obstacle quantity B for the time point quantity L; the third column vectors correspond to the first time points one by one;

[0034] traversing each of the first time points; when traversing, taking the first time point currently traversed as a corresponding current time point; and confirming whether the current time point has a matching second sampling time point; if it is confirmed that the current time point has a matching second sampling time point, updating the third column vector corresponding to the current time point in the second point cloud target time mask tensor to a full-1 column vector;

[0035] outputting the obtained second point cloud target time feature tensor and the second point cloud target time mask tensor as millimeter wave radar point cloud data preprocessing results.

[0036] Further, the camera image data preprocessing according to the first image sequence, all the first time points and the time point quantity L obtains corresponding first image time feature tensors, first image target time feature tensors, first image time mask tensors and first image target time mask tensors, and specifically includes:

[0037] performing image feature extraction processing on each of the first images of the first image sequence to obtain a corresponding first image feature tensor with a shape of B x F4; and performing obstacle target detection processing on each of the first image feature tensors to obtain a corresponding first image target feature tensor with a shape of B x F5; the first image feature tensors and the first image target feature tensors correspond to the third sampling time points one by one;

[0038] initializing a tensor sequence consisting of L number of all-zero tensors with shape BxF4 as a corresponding fourth tensor sequence; and initializing a tensor sequence consisting of L number of all-zero tensors with shape BxF5 as a corresponding fifth tensor sequence; the fourth tensor sequence comprises L number of fourth tensors; the fourth tensor has shape BxF4, and is an all-zero tensor, and corresponds to the first time point; the fifth tensor sequence comprises L number of fifth tensors; the fifth tensor has shape BxF5, and is an all-zero tensor, and corresponds to the first time point;

[0039] traversing each fourth tensor of the fourth tensor sequence; when traversing, taking the fourth tensor being currently traversed as a corresponding current tensor, and taking the first time point corresponding to the current tensor as a corresponding current time point; and confirming whether the current time point has a matching third sampling time point; if it is confirmed that the current time point has a matching third sampling time point, updating the current tensor to the first image feature tensor corresponding to the matching third sampling time point; when the traversal ends, performing tensor conversion on the fourth tensor sequence according to a set tensor shape BxLxF4 to obtain a corresponding first image time feature tensor;

[0040] traversing each fifth tensor of the fifth tensor sequence; when traversing, taking the fifth tensor being currently traversed as a corresponding current tensor, and taking the first time point corresponding to the current tensor as a corresponding current time point; and confirming whether the current time point has a matching third sampling time point; if it is confirmed that the current time point has a matching third sampling time point, updating the current tensor to the first image target feature tensor corresponding to the matching third sampling time point; when the traversal ends, performing tensor conversion on the fifth tensor sequence according to a set tensor shape BxLxF5 to obtain a corresponding first image target time feature tensor;

[0041] initializing two all-zero tensors with shape BxL as a corresponding first image time mask tensor and a first image target time mask tensor; the first image time mask tensor comprises L number of fourth column vectors with length B; the first image target time mask tensor comprises L number of fifth column vectors with length B; the fourth column vector and the fifth column vector both correspond to the first time point;

[0042] traversing each of the first time points; when traversing, taking the first time point currently traversed as a corresponding current time point; and confirming whether the current time point has a matching third sampling time point; if it is confirmed that the current time point has a matching third sampling time point, updating the fourth column vector corresponding to the current time point in the first image time mask tensor to a full 1 column vector, and updating the fifth column vector corresponding to the current time point in the first image target time mask tensor to a full 1 column vector;

[0043] outputting the obtained first image time feature tensor, the first image target time feature tensor, the first image time mask tensor and the first image target time mask tensor as a camera image data preprocessing result.

[0044] Preferably, the target state understanding and classification network comprises an embedding layer, a position encoding layer, a fusion layer, a group normalization layer, a state encoding layer, a state prediction branch network, a classification prediction branch network and a prediction data processing module;

[0045] The embedding layer comprises five fully connected networks, which are first, second, third, fourth and fifth fully connected networks respectively;

[0046] The position encoding layer comprises five encoding modules, which are first, second, third, fourth and fifth encoding modules respectively; the input ends of the first, second, third, fourth and fifth encoding modules are connected with the output ends of the corresponding first, second, third, fourth and fifth fully connected networks;

[0047] The first, second, third, fourth and fifth input ends of the fusion layer are connected with the output ends of the first, second, third, fourth and fifth encoding modules respectively;

[0048] The input end of the group normalization layer is connected with the output end of the fusion layer;

[0049] The state coding layer is implemented based on an Encoder module of a Transformer model; the state coding layer comprises three-level state coding modules; an input end of the first-level state coding module is connected with an output end of the fusion layer; an input end of the second-level state coding module is connected with an output end of the first-level state coding module; an input end of the third-level state coding module is connected with an output end of the second-level state coding module; an output end of the third-level state coding module is an output end of the state coding layer; each level of the state coding module comprises a multi-head attention network, a first addition and normalization network, a feedforward neural network and a second addition and normalization network; an input end of the multi-head attention network is connected with an input end of the state coding module; a first input end of the first addition and normalization network is connected with an input end of the multi-head attention network, and a second input end is connected with an output end of the multi-head attention network; an input end of the feedforward neural network is connected with an output end of the first addition and normalization network; a first input end of the second addition and normalization network is connected with an input end of the feedforward neural network, and a second input end is connected with an output end of the feedforward neural network; an output end of the second addition and normalization network is an output end of the state coding module;

[0050] The state prediction branch network is composed of multiple levels of fully connected networks; an input end of the state prediction branch network is connected with an output end of the state coding layer;

[0051] The classification prediction branch network is composed of multiple levels of fully connected networks; an input end of the classification prediction branch network is connected with an output end of the state coding layer;

[0052] A first input end of the prediction data processing module is connected with an output end of the state prediction branch network, and a second input end is connected with an output end of the classification prediction branch network.

[0053] Preferably, the multi-class perception source time feature tensor set and the multi-class perception source time mask tensor are input into a preset target state understanding and classification network to obtain a corresponding first obstacle prediction data set through obstacle target state prediction and obstacle target category prediction processing, specifically comprising:

[0054] The target state understanding and classification network inputs the first point cloud time feature tensor, the first point cloud target time feature tensor, the second point cloud target time feature tensor, the first image time feature tensor and the first image target time feature tensor of the input multi-class perception source time feature tensor set into corresponding first, second, third, fourth and fifth fully connected networks in the embedding layer for fully connected processing to obtain corresponding first, second, third, fourth and fifth fully connected tensors; the shapes of the first, second, third, fourth and fifth fully connected tensors are all BxLxAl; the feature dimension Al is 128 by default;

[0055] The first, second, third, fourth and fifth fully connected tensors are input into corresponding first, second, third, fourth and fifth encoding modules in the position encoding layer for position encoding processing to obtain corresponding first, second, third, fourth and fifth position encoding tensors; the shapes of the first, second, third, fourth and fifth position encoding tensors are all BxLxAl;

[0056] The first, second, third, fourth and fifth position encoding tensors are input into the fusion layer for tensor splicing and fusion processing to obtain a corresponding first fusion tensor; the shape of the first fusion tensor is BxLx A2, and the feature dimension A2=(Al*N);

[0057] The first fusion tensor is input into the group normalization layer for group normalization processing to obtain a corresponding first normalization tensor; the shape of the first normalization tensor is BxLx A2;

[0058] The first normalization tensor and the multi-class perception source time mask tensor are input into the state encoding layer, and the three-level state encoding modules of the state encoding layer are sequentially encoded to obtain a corresponding first state encoding tensor; the shape of the first state encoding tensor is BxLx A3, and A3 is a preset state tensor feature dimension;

[0059] The first state encoding tensor is input into the state prediction branch network, and the multi-level fully connected network of the state prediction branch network is used for obstacle target state regression prediction processing to obtain a corresponding first state prediction tensor; the shape of the first state prediction tensor is Bx A4, and A4 is a preset state prediction tensor feature dimension; the first state prediction tensor includes a first state prediction vector of the obstacle number B; the vector length of the first state prediction vector is the feature dimension A4, and the vector data of the first state prediction vector includes first state prediction data of the feature dimension A4; the first state prediction includes a first obstacle speed;

[0060] The first state prediction tensor and the first classification prediction tensor are input into the prediction data processing module for output data integration processing to obtain the first obstacle prediction data set corresponding thereto.

[0061] The first state prediction tensor and the first classification prediction tensor are input into the prediction data processing module for output data integration processing to obtain the first obstacle prediction data set corresponding thereto.

[0062] The first obstacle prediction data set obtained is taken as the target state understanding and classification network processing result output.

[0063] Further, the first, second, third, fourth and fifth full connection tensors are input into the first, second, third, fourth and fifth encoding modules in the position encoding layer for position encoding processing to obtain the first, second, third, fourth and fifth position encoding tensors corresponding thereto, specifically including:

[0064] The first, second, third, fourth and fifth encoding modules perform position encoding on the input first, second, third, fourth and fifth full connection tensors based on the position encoding algorithm of the Transformer model to obtain the first, second, third, fourth and fifth position encoding tensors corresponding thereto.

[0065] Further, the first normalization tensor and the multi-class perception source time mask tensor are input into the state encoding layer, and the first state encoding tensor corresponding thereto is obtained by performing step-by-step encoding processing on the state encoding layer through the three-level state encoding modules of the state encoding layer, specifically including:

[0066] The state coding layer inputs the first normalized tensor and the multi-class perceptual source time mask tensor into the multi-head attention network of the first level of the state coding module for multi-head attention coding processing to obtain a corresponding first attention tensor; inputs the first attention tensor and the first normalized tensor into the first addition and normalization network of the first level of the state coding module for tensor addition and normalization processing to obtain a corresponding second normalized tensor; inputs the second normalized tensor into the feedforward neural network of the first level of the state coding module for nonlinear transformation processing to obtain a corresponding first transformed tensor; and inputs the first transformed tensor and the second normalized tensor into the second addition and normalization network of the first level of the state coding module for tensor addition and normalization processing to obtain a corresponding third normalized tensor;

[0067] The third normalized tensor and the multi-class perceptual source time mask tensor are further input into the multi-head attention network of the second level of the state coding module of the state coding layer for multi-head attention coding processing to obtain a corresponding second attention tensor; the second attention tensor and the third normalized tensor are input into the first addition and normalization network of the second level of the state coding module for tensor addition and normalization processing to obtain a corresponding fourth normalized tensor; the fourth normalized tensor is input into the feedforward neural network of the second level of the state coding module for nonlinear transformation processing to obtain a corresponding second transformed tensor; and the second transformed tensor and the fourth normalized tensor are input into the second addition and normalization network of the second level of the state coding module for tensor addition and normalization processing to obtain a corresponding fifth normalized tensor;

[0068] The fifth normalized tensor and the multi-class perceptual source time mask tensor are further input into the multi-head attention network of the third level of the state coding module of the state coding layer for multi-head attention coding processing to obtain a corresponding third attention tensor; the third attention tensor and the fifth normalized tensor are input into the first addition and normalization network of the third level of the state coding module for tensor addition and normalization processing to obtain a corresponding sixth normalized tensor; the sixth normalized tensor is input into the feedforward neural network of the third level of the state coding module for nonlinear transformation processing to obtain a corresponding third transformed tensor; and the third transformed tensor and the sixth normalized tensor are input into the second addition and normalization network of the third level of the state coding module for tensor addition and normalization processing to obtain a corresponding seventh normalized tensor; and the seventh normalized tensor is output as the corresponding first state coding tensor.

[0069] Further, the first state prediction tensor and the first classification prediction tensor are input into the prediction data processing module for output data integration processing to obtain a corresponding first obstacle prediction data set, specifically comprising:

[0070] The prediction data processing module records each first state prediction vector of the first state prediction tensor as a corresponding first state prediction vector S i , i is an obstacle index, 1≤i≤B;and records each first classification prediction vector of the first classification prediction tensor as a corresponding first classification prediction vector Y i ;all first state prediction data of each first state prediction vector S i are extracted to form corresponding first obstacle state data;the first classification prediction probability corresponding to the obstacle target type with the maximum probability in each first classification prediction vector Y i is taken as corresponding first obstacle type data;according to the corresponding relationship of the obstacle index i, the first obstacle state data and the first obstacle type data corresponding to the same obstacle index i are combined to form corresponding first obstacle prediction data;the first obstacle prediction data of the obtained obstacle number B are combined to form a corresponding first obstacle prediction data set;and the first obstacle prediction data set is output as the output data integration processing result.

[0071] The second aspect of the embodiment of the application provides an electronic device, comprising a memory, a processor and a transceiver;

[0072] The processor is used for coupling with the memory, reading and executing instructions in the memory to realize the method steps of the first aspect;

[0073] The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transceiving.

[0074] The third aspect of the embodiment of the application provides a computer readable storage medium, and the computer readable storage medium stores computer instructions, when the computer instructions are executed by a computer, the computer instructions make the computer execute the instructions of the method in the first aspect.

[0075] The embodiment of the present application provides a processing method for predicting obstacle state and category based on multi-class perception features, an electronic device and a computer readable storage medium, multi-class data features (basic data features, target detection features) of multi-class perception data (laser radar, millimeter wave radar, camera, etc.) are fused, and a target state understanding and classification network is constructed to perform end-to-end state+category synchronous prediction, and then the fused features are input into the target state understanding and classification network for prediction to obtain the state and category prediction results of the obstacle. Through the present application, the end-to-end state+category synchronous prediction is realized based on a whole solution, and the prediction accuracy of the solution is improved by fusing multi-class perception features. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 A processing method for predicting obstacle state and category based on multi-class perception features provided for the embodiment one of the present application;

[0077] Figure 2 A module structure diagram of the target state understanding and classification network provided for the embodiment one of the present application;

[0078] Figure 3 A structure schematic diagram of an electronic device provided for the embodiment two of the present application. DETAILED DESCRIPTION

[0079] In order to make the object, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0080] The automatic driving system of the vehicle can realize end-to-end state+category synchronous prediction of the obstacle target by using the processing method for predicting obstacle state and category based on multi-class perception features provided by the embodiment one of the present application. Figure 1 A processing method for predicting obstacle state and category based on multi-class perception features provided for the embodiment one of the present application is shown in FIG. 1, which mainly includes the following steps: Figure 1

[0081] Step 1, acquiring the perception data sequence of the vehicle-mounted laser radar, vehicle-mounted millimeter wave radar and vehicle-mounted camera in the nearest first time period T as the corresponding first laser radar point cloud sequence, first millimeter wave radar point cloud sequence and first image sequence;

[0082] ​The first laser radar point cloud sequence includes a plurality of first laser radar point clouds, each of which corresponds to a first sampling time point; the first millimeter wave radar point cloud sequence includes a plurality of first millimeter wave radar point clouds, each of which corresponds to a second sampling time point; and the first image sequence includes a plurality of first images, each of which corresponds to a third sampling time point.

[0083] Here, the automatic driving system of the embodiment one of the application can obtain the perception data sequences of the vehicle-mounted laser radar, vehicle-mounted millimeter wave radar and vehicle-mounted camera in the first time period T as the corresponding first laser radar point cloud sequence, first millimeter wave radar point cloud sequence and first image sequence through the perception module of the system; the sampling data of the three sequences may not be synchronized, that is, the sequence length and the sampling frequency of the three sequences may be different.

[0084] Step 2, divide the first time period T into a plurality of first time points according to a preset first time interval Δt, and take the number of the first time points as the corresponding time point number L.

[0085] Here, the sampling frequency corresponding to the first time interval Δt of the embodiment one of the application = 1 / Δt is greater than the sampling frequency of the above three types of perception data sequences, and the automatic driving system of the embodiment one of the application uses a faster sampling frequency to plan the timeline, the purpose of which is to make all the first, second and third sampling time points of the above three types of perception data sequences fall on a corresponding first time point.

[0086] Step 3, performing multi-type perception source data preprocessing according to the first laser radar point cloud sequence, the first millimeter wave radar point cloud sequence, the first image sequence, all the first time points and the time point number L to obtain a corresponding multi-type perception source time feature tensor set and a corresponding multi-type perception source time mask tensor;

[0087] The multi-type perception source time feature tensor set includes a first point cloud time feature tensor, a first point cloud target time feature tensor, a second point cloud target time feature tensor, a first image time feature tensor and a first image target time feature tensor; the shape of the first point cloud time feature tensor is B×L×F1, the shape of the first point cloud target time feature tensor is B×L×F2, the shape of the second point cloud target time feature tensor is B×L×F3, the shape of the first image time feature tensor is B×L×F4, and the shape of the first image target time feature tensor is B×L×F5; the number of obstacles B≥1, and the feature dimensions F1, F2, F3, F4 and F5≥1.

[0088] The shape of the multi-class perception source time mask tensor is BxLxN, and the number of perception sources N is 5 by default; the multi-class perception source time mask tensor is fused by the time mask tensors with the shape of BxL of the number of perception sources N, which are respectively: a first point cloud time mask tensor, a first point cloud target time mask tensor, a second point cloud target time mask tensor, a first image time mask tensor and a first image target time mask tensor;

[0089] Specifically, it comprises: step 31, performing laser radar point cloud data preprocessing according to the first laser radar point cloud sequence, all first time points and the number L of time points to obtain corresponding first point cloud time feature tensors, first point cloud target time feature tensors, first point cloud time mask tensors and first point cloud target time mask tensors;

[0090] Specifically, it comprises: step 311, performing point cloud feature extraction processing on each first laser radar point cloud of the first laser radar point cloud sequence to obtain a corresponding first point cloud feature tensor with the shape of BxF1; and performing obstacle target detection processing on each first point cloud feature tensor to obtain a corresponding first point cloud target feature tensor with the shape of BxF2;

[0091] Wherein, the first point cloud feature tensor and the first point cloud target feature tensor are one-to-one corresponding to the first sampling time points;

[0092] Here, the automatic driving system of the embodiment one of the application can perform dynamic configuration based on system computing resource when performing point cloud feature extraction processing on each first laser radar point cloud, and can use the backbone feature extraction network of the PointNet model to perform point cloud basic feature extraction when the computing resource and speed are sufficient, and can use the Multi-Layer Perceptron (MLP) neural network to perform point cloud basic feature extraction when the computing resource and speed are insufficient; the automatic driving system of the embodiment one of the application can use a conventional point cloud target detection network to output point cloud target detection features when performing obstacle target detection processing on each first point cloud feature tensor, which will not be described here;

[0093] Step 312, initializing a tensor sequence composed of time point number L full zero tensors with the shape of BxF1 as a corresponding first tensor sequence; and initializing a tensor sequence composed of time point number L full zero tensors with the shape of BxF2 as a corresponding second tensor sequence;

[0094] Wherein, the first tensor sequence includes time point number L first tensors; the shape of the first tensor is BxF1, the first tensor is a full zero tensor, and the first tensor is one-to-one corresponding to the first time point; the second tensor sequence includes time point number L second tensors; the shape of the second tensor is BxF2, the second tensor is a full zero tensor, and the second tensor is one-to-one corresponding to the first time point;

[0095] Step 313, each first tensor of the first tensor sequence is traversed; when traversing, the currently traversed first tensor is taken as a corresponding current tensor, and the first time point corresponding to the current tensor is taken as a corresponding current time point; and whether the current time point has a matched first sampling time point is confirmed; if it is confirmed that the current time point has a matched first sampling time point, the current tensor is updated to the first point cloud feature tensor corresponding to the matched first sampling time point; when the traversal is completed, the first tensor sequence is tensor-converted according to the set tensor shape B x L x F1 to obtain a corresponding first point cloud time feature tensor;

[0096] Here, the automatic driving system of the embodiment one of the present application can mix the point cloud basic features corresponding to all the first laser radar point clouds, i.e., the first point cloud feature tensors, into a high-dimensional tensor with a fixed shape of B x L, i.e., the first point cloud time feature tensor, through the current step;

[0097] Step 314, each second tensor of the second tensor sequence is traversed; when traversing, the currently traversed second tensor is taken as a corresponding current tensor, and the first time point corresponding to the current tensor is taken as a corresponding current time point; and whether the current time point has a matched first sampling time point is confirmed; if it is confirmed that the current time point has a matched first sampling time point, the current tensor is updated to the first point cloud target feature tensor corresponding to the matched first sampling time point; when the traversal is completed, the second tensor sequence is tensor-converted according to the set tensor shape B x L x F2 to obtain a corresponding first point cloud target time feature tensor;

[0098] Here, the automatic driving system of the embodiment one of the present application can mix the point cloud target detection features corresponding to all the first laser radar point clouds, i.e., the first point cloud target feature tensors, into a high-dimensional tensor with a fixed shape of B x L, i.e., the first point cloud target time feature tensor, through the current step;

[0099] Step 315, two all-zero tensors with a shape of B x L are initialized as a corresponding first point cloud time mask tensor and a first point cloud target time mask tensor;

[0100] The first point cloud time mask tensor includes a first column vector with a length of an obstacle number B for a time point number L; the first point cloud target time mask tensor includes a second column vector with a length of an obstacle number B for a time point number L; the first column vector and the second column vector are in one-to-one correspondence with the first time point;

[0101] Step 316, iterate through each first time point; when iterating, take the currently iterated first time point as a corresponding current time point; and confirm whether the current time point has a matching first sampling time point; if it is confirmed that the current time point has a matching first sampling time point, update the first column vector corresponding to the current time point in the first point cloud time mask tensor to a full 1 column vector, and update the second column vector corresponding to the current time point in the first point cloud target time mask tensor to a full 1 column vector;

[0102] Here, the first point cloud time mask tensor and the first point cloud target time mask tensor of the embodiment one of the application will be used in the subsequent operation of the multi-head attention network;

[0103] Step 317, output the obtained first point cloud time feature tensor, first point cloud target time feature tensor, first point cloud time mask tensor and first point cloud target time mask tensor as the laser radar point cloud data preprocessing result;

[0104] Step 32, perform millimeter wave radar point cloud data preprocessing according to the first millimeter wave radar point cloud sequence, all first time points and the time point quantity L to obtain a corresponding second point cloud target time feature tensor and a second point cloud target time mask tensor;

[0105] Specifically, it includes: step 321, performing point cloud feature extraction processing on each first millimeter wave radar point cloud of the first millimeter wave radar point cloud sequence and obstacle target detection processing based on the extracted features to obtain a second point cloud target feature tensor with a shape of BxF3;

[0106] Wherein, the second point cloud target feature tensor corresponds to the second sampling time point one by one;

[0107] Here, the automatic driving system of the embodiment one of the application can perform dynamic configuration based on system computing resource when performing point cloud feature extraction processing on each first millimeter wave radar point cloud, and can use a backbone feature extraction network such as a PointNet model, a backbone feature extraction network of a VoxelNet model, a backbone feature extraction network of a PointPillars model, etc. to perform point cloud basic feature extraction when the computing resource and speed are sufficient, and can use a multi-layer perception neural network to perform point cloud basic feature extraction when the computing resource and speed are insufficient; the automatic driving system of the embodiment one of the application can use a conventional point cloud target detection network to output point cloud target detection features when performing obstacle target detection processing on the extracted features of each first millimeter wave radar point cloud, which will not be described here;

[0108] Step 322, initialize a tensor sequence composed of a zero tensor with a shape of BxF3 of the time point quantity L as a corresponding third tensor sequence;

[0109] The third tensor sequence includes third tensors of the number L of time points; the shape of the third tensor is BxF3, the third tensor is a full zero tensor, and the third tensor corresponds to the first time point one by one;

[0110] Step 323, each third tensor of the third tensor sequence is traversed; when traversing, the currently traversed third tensor is taken as a corresponding current tensor, and the first time point corresponding to the current tensor is taken as a corresponding current time point; and it is confirmed whether the current time point has a matched second sampling time point; if it is confirmed that the current time point has a matched second sampling time point, the current tensor is updated to a second point cloud target feature tensor corresponding to the matched second sampling time point; when the traversal is completed, the third tensor sequence is tensor-converted according to the set tensor shape BxLxF3 to obtain a corresponding second point cloud target time feature tensor;

[0111] Here, the automatic driving system of the embodiment one of the present application can mix all the point cloud target detection features corresponding to the first millimeter wave radar point cloud, i.e., the second point cloud target feature tensor, into a high-dimensional tensor with a fixed shape of BxL, i.e., the second point cloud target time feature tensor, at the current step;

[0112] Step 324, a full zero tensor with a shape of BxL is initialized as a corresponding second point cloud target time mask tensor; the second point cloud target time mask tensor includes third column vectors of the number B of obstacles with the number L of time points;

[0113] The third column vector corresponds to the first time point one by one;

[0114] Step 325, each first time point is traversed; when traversing, the currently traversed first time point is taken as a corresponding current time point; and it is confirmed whether the current time point has a matched second sampling time point; if it is confirmed that the current time point has a matched second sampling time point, the third column vector corresponding to the current time point in the second point cloud target time mask tensor is updated to a full one column vector;

[0115] Here, the second point cloud target time mask tensor of the embodiment one of the present application will be used in the operation of the subsequent multi-head attention network;

[0116] Step 326, the obtained second point cloud target time feature tensor and the second point cloud target time mask tensor are used to output the millimeter wave radar point cloud data preprocessing result;

[0117] Step 33, camera image data preprocessing is performed according to the first image sequence, all the first time points, and the number L of time points to obtain a corresponding first image time feature tensor, a first image target time feature tensor, a first image time mask tensor, and a first image target time mask tensor;

[0118] Specifically comprising: step 331, performing image feature extraction processing on each first image of the first image sequence to obtain a corresponding first image feature tensor with a shape of BxF4; and performing obstacle target detection processing on each first image feature tensor to obtain a corresponding first image target feature tensor with a shape of BxF5;

[0119] The first image feature tensor and the first image target feature tensor each correspond to a third sampling time point;

[0120] Here, the automatic driving system of the embodiment one of the present application can perform image basic feature extraction based on a conventional image feature extraction network when performing image feature extraction processing on each first image; the automatic driving system of the embodiment one of the present application can use a conventional image target detection network to output image target detection features when performing obstacle target detection processing on each first image feature tensor, which will not be described here;

[0121] Step 332, initializing a tensor sequence composed of L time points of all-zero tensors with a shape of BxF4 as a corresponding fourth tensor sequence; and initializing a tensor sequence composed of L time points of all-zero tensors with a shape of BxF5 as a corresponding fifth tensor sequence;

[0122] The fourth tensor sequence includes L fourth tensors, each with a shape of BxF4; the fourth tensors are all-zero tensors, and each fourth tensor corresponds to a first time point; the fifth tensor sequence includes L fifth tensors, each with a shape of BxF5; the fifth tensors are all-zero tensors, and each fifth tensor corresponds to a first time point;

[0123] Step 333, traversing each fourth tensor of the fourth tensor sequence; during traversal, the currently traversed fourth tensor is taken as a corresponding current tensor, and the first time point corresponding to the current tensor is taken as a corresponding current time point; and it is confirmed whether the current time point has a matching third sampling time point; if it is confirmed that the current time point has a matching third sampling time point, the current tensor is updated to the first image feature tensor corresponding to the matching third sampling time point; after traversal, the fourth tensor sequence is tensor-converted according to a set tensor shape BxLxF4 to obtain a corresponding first image time feature tensor;

[0124] Here, the automatic driving system of the embodiment one of the present application can mix all image basic features corresponding to the first images, i.e., the first image feature tensors, into a high-dimensional tensor with a fixed shape of BxL, i.e., the first image time feature tensor, through the current step;

[0125] Step 334, each fifth tensor of the fifth tensor sequence is traversed; when traversing, the currently traversed fifth tensor is taken as a corresponding current tensor, and a first time point corresponding to the current tensor is taken as a corresponding current time point; and whether the current time point has a matched third sampling time point is confirmed; if it is confirmed that the current time point has a matched third sampling time point, the current tensor is updated to a first image target feature tensor corresponding to the matched third sampling time point; when the traversal is completed, the fifth tensor sequence is tensor-converted according to a set tensor shape B x L x F5 to obtain a corresponding first image target time feature tensor;

[0126] Here, the automatic driving system of the embodiment one of the present application can mix the image target detection features corresponding to all the first images, i.e., the first image target feature tensor, into a high-dimensional tensor with a fixed shape of B x L, i.e., the first image target time feature tensor, at the current step;

[0127] Step 335, two all-zero tensors with a shape of B x L are initialized as a corresponding first image time mask tensor and a first image target time mask tensor;

[0128] The first image time mask tensor includes fourth column vectors with a length of obstacle quantity B at a time point quantity L; the first image target time mask tensor includes fifth column vectors with a length of obstacle quantity B at a time point quantity L; the fourth column vectors and the fifth column vectors are one-to-one corresponding to the first time points;

[0129] Step 336, each first time point is traversed; when traversing, the currently traversed first time point is taken as a corresponding current time point; and whether the current time point has a matched third sampling time point is confirmed; if it is confirmed that the current time point has a matched third sampling time point, a fourth column vector corresponding to the current time point in the first image time mask tensor is updated to an all-1 column vector, and a fifth column vector corresponding to the current time point in the first image target time mask tensor is updated to an all-1 column vector;

[0130] Here, the first image time mask tensor and the first image target time mask tensor of the embodiment one of the present application will be used in the operation of the subsequent multi-head attention network;

[0131] Step 337, the obtained first image time feature tensor, first image target time feature tensor, first image time mask tensor and first image target time mask tensor are output as a camera image data preprocessing result;

[0132] Step 34, the obtained first point cloud time feature tensor, first point cloud target time feature tensor, second point cloud target time feature tensor, first image time feature tensor and first image target time feature tensor are combined to form a corresponding multi-class perception source time feature tensor set;

[0133] Step 35, tensor fusion is performed on the obtained first point cloud time mask tensor, first point cloud target time mask tensor, second point cloud target time mask tensor, first image time mask tensor and first image target time mask tensor to generate a corresponding multi-class perception source time mask tensor with a shape of BxLxN;

[0134] Step 36, the obtained multi-class perception source time feature tensor set and multi-class perception source time mask tensor are output as a multi-class perception source data preprocessing result.

[0135] Step 4, the multi-class perception source time feature tensor set and the multi-class perception source time mask tensor are input into a preset target state understanding and classification network to perform obstacle target state prediction and obstacle target category prediction processing to obtain a corresponding first obstacle prediction data set;

[0136] The first obstacle prediction data set includes a plurality of first obstacle prediction data; the first obstacle prediction data includes first obstacle state data and first obstacle type data.

[0137] Here, the target state understanding and classification network provided by the embodiment one of the application is a comprehensive neural network which can fuse multi-class data features (basic features, target detection features) of multi-class perception data (laser radar, millimeter wave radar, camera, etc.) and realize end-to-end state + category synchronous prediction based on the fused features; the comprehensive neural network is as shown in the following figure: Figure 2 The module structure diagram of the target state understanding and classification network provided by the embodiment one of the application includes an embedding layer, a position coding layer, a fusion layer, a group normalization layer, a state coding layer, a state prediction branch network, a classification prediction branch network and a prediction data processing module.

[0138] 1) The embedding layer includes five full connection networks, which are first, second, third, fourth and fifth full connection networks; here, the embedding layer of the embodiment one of the application is to unify the shapes of the tensors in the multi-class perception source time feature tensor set so as to facilitate the fusion processing of the subsequent modules;

[0139] 2) The position coding layer includes five coding modules, which are first, second, third, fourth and fifth coding modules; the input ends of the first, second, third, fourth and fifth coding modules are connected with the output ends of the corresponding first, second, third, fourth and fifth full connection networks; here, each coding module of the position coding layer of the embodiment one of the application is realized by position coding based on the Positional Encoding algorithm of the Transformer model;

[0140] 3) the first, second, third, fourth and fifth inputs of the fusion layer are connected with the outputs of the first, second, third, fourth and fifth encoding modules respectively; here, the fusion layer of the embodiment one of the application is to splice and fuse all the input tensors;

[0141] 4) the input of the group normalization layer is connected with the output of the fusion layer; here, the group normalization layer of the embodiment one of the application is to perform group normalization processing on the input tensor;

[0142] 5) the state encoding layer is realized based on the Encoder module of the Transformer model; the state encoding layer includes three-level state encoding modules; the input of the first-level state encoding module is connected with the output of the fusion layer; the input of the second-level state encoding module is connected with the output of the first-level state encoding module; the input of the third-level state encoding module is connected with the output of the second-level state encoding module; the output of the third-level state encoding module is the output of the state encoding layer; each level of the state encoding module includes a multi-head attention network, a first addition and normalization network, a feedforward neural network and a second addition and normalization network; the input of the multi-head attention network is connected with the input of the state encoding module; the first input of the first addition and normalization network is connected with the input of the multi-head attention network, and the second input is connected with the output of the multi-head attention network; the input of the feedforward neural network is connected with the output of the first addition and normalization network; the first input of the second addition and normalization network is connected with the input of the feedforward neural network, and the second input is connected with the output of the output of the feedforward neural network; the output of the second addition and normalization network is the output of the state encoding module; here, the state encoding layer of the embodiment one of the application is similar to the structure of the Encoder module of the Transformer model, the type of operation logic and the use principle of the mask are similar, and are not repeated here;

[0143] 6) the state prediction branch network is composed of multiple levels of fully connected networks; the input of the state prediction branch network is connected with the output of the state encoding layer; here, the state prediction branch network of the embodiment one of the application is a conventional deep learning state regression prediction network, which can predict the motion state of the obstacle, such as predicting the speed of the obstacle;

[0144] 7) the classification prediction branch network is composed of multiple levels of fully connected networks; the input of the classification prediction branch network is connected with the output of the state encoding layer; here, the classification prediction branch network of the embodiment one of the application is a conventional deep learning classification prediction network, which can classify and identify the target type of the obstacle;

[0145] 8) The first input end of the prediction data processing module is connected with the output end of the state prediction branch network, and the second input end is connected with the output end of the classification prediction branch network.

[0146] Step 4 specifically comprises: step 41, the target state understanding and classification network inputs the first point cloud time feature tensor of the input multi-class perception source time feature tensor set, the first point cloud target time feature tensor, the second point cloud target time feature tensor, the first image time feature tensor and the first image target time feature tensor into the corresponding first, second, third, fourth and fifth fully connected networks in the embedding layer respectively for fully connected processing to obtain the corresponding first, second, third, fourth and fifth fully connected tensors;

[0147] Wherein, the shapes of the first, second, third, fourth and fifth fully connected tensors are BxLxAl; the feature dimension Al is 128 by default;

[0148] Step 42, input the first, second, third, fourth and fifth fully connected tensors into the corresponding first, second, third, fourth and fifth encoding modules in the position encoding layer for position encoding processing to obtain the corresponding first, second, third, fourth and fifth position encoding tensors;

[0149] Wherein, the shapes of the first, second, third, fourth and fifth position encoding tensors are BxLxAl;

[0150] Specifically, the first, second, third, fourth and fifth encoding modules perform position encoding on the input first, second, third, fourth and fifth fully connected tensors based on the position encoding algorithm of the Transformer model to obtain the corresponding first, second, third, fourth and fifth position encoding tensors;

[0151] Here, the specific encoding implementation can refer to the position encoding algorithm implementation of the Transformer model given in the paper Attention Is All You Need, which will not be described here;

[0152] Step 43, input the first, second, third, fourth and fifth position encoding tensors into the fusion layer for tensor splicing fusion processing to obtain the corresponding first fusion tensor;

[0153] Wherein, the shape of the first fusion tensor is BxLx A2, and the feature dimension A2=(Al*N);

[0154] Step 44, input the first fusion tensor into the group normalization layer for group normalization processing to obtain the corresponding first normalization tensor;

[0155] Wherein, the shape of the first normalization tensor is BxLx A2.

[0156] Step 45, input the first normalized tensor and the multi-class perceptual source time mask tensor into the state encoding layer, and sequentially encode the input through the three-level state encoding modules of the state encoding layer to obtain a corresponding first state encoding tensor;

[0157] The shape of the first state encoding tensor is BxLxA3, and A3 is a preset state tensor feature dimension.

[0158] Specifically, step 451, the state encoding layer inputs the input first normalized tensor and the multi-class perceptual source time mask tensor into the multi-head attention network of the first-level state encoding module for multi-head attention encoding processing to obtain a corresponding first attention tensor; inputs the first attention tensor and the first normalized tensor into the first addition and normalization network of the first-level state encoding module for tensor addition and normalization processing to obtain a corresponding second normalized tensor; inputs the second normalized tensor into the feedforward neural network of the first-level state encoding module for nonlinear transformation processing to obtain a corresponding first transformed tensor; and inputs the first transformed tensor and the second normalized tensor into the second addition and normalization network of the first-level state encoding module for tensor addition and normalization processing to obtain a corresponding third normalized tensor.

[0159] Step 452, input the third normalized tensor and the multi-class perceptual source time mask tensor into the multi-head attention network of the second-level state encoding module of the state encoding layer for multi-head attention encoding processing to obtain a corresponding second attention tensor; input the second attention tensor and the third normalized tensor into the first addition and normalization network of the second-level state encoding module for tensor addition and normalization processing to obtain a corresponding fourth normalized tensor; input the fourth normalized tensor into the feedforward neural network of the second-level state encoding module for nonlinear transformation processing to obtain a corresponding second transformed tensor; and input the second transformed tensor and the fourth normalized tensor into the second addition and normalization network of the second-level state encoding module for tensor addition and normalization processing to obtain a corresponding fifth normalized tensor.

[0160] Step 453, the fifth normalized tensor and the multi-class perception source time mask tensor are input into the multi-head attention network of the third level state encoding module of the state encoding layer for multi-head attention encoding processing to obtain a corresponding third attention tensor; the third attention tensor and the fifth normalized tensor are input into the first addition and normalization network of the third level state encoding module for tensor addition and normalization processing to obtain a corresponding sixth normalized tensor; the sixth normalized tensor is input into the feedforward neural network of the third level state encoding module for nonlinear transformation processing to obtain a corresponding third transformed tensor; the third transformed tensor and the sixth normalized tensor are input into the second addition and normalization network of the third level state encoding module for tensor addition and normalization processing to obtain a corresponding seventh normalized tensor; and the seventh normalized tensor is output as a corresponding first state encoding tensor;

[0161] Step 46, the first state encoding tensor is input into the state prediction branch network, and obstacle target state regression prediction processing is performed on the first state encoding tensor by the multi-level fully connected network of the state prediction branch network to obtain a corresponding first state prediction tensor;

[0162] The shape of the first state prediction tensor is BxA4, A4 is a preset state prediction tensor feature dimension; the first state prediction tensor includes B first state prediction vectors; the vector length of the first state prediction vector is the feature dimension A4, and the vector data of the first state prediction vector includes first state prediction data of the feature dimension A4; the first state prediction includes a first obstacle speed.

[0163] Step 47, the first state encoding tensor is input into the classification prediction branch network, and obstacle target classification prediction processing is performed on the first state encoding tensor by the multi-level fully connected network of the classification prediction branch network to obtain a corresponding first classification prediction tensor;

[0164] The shape of the first classification prediction tensor is BxA5, A5 is a preset classification prediction tensor feature dimension; the first classification prediction tensor includes B first classification prediction vectors; the vector length of the first classification prediction vector is the feature dimension A5, and the first classification prediction vector includes first classification prediction probabilities of the feature dimension A5; each first classification prediction probability corresponds to an obstacle target type.

[0165] Step 48, the first state prediction tensor and the first classification prediction tensor are input into the prediction data processing module for output data integration processing to obtain a corresponding first obstacle prediction data set.

[0166] Specifically, the prediction data processing module records each first state prediction vector of the first state prediction tensor as a corresponding first state prediction vector S i, i is the obstacle index, 1≤i≤B; and each first classification prediction vector of the first classification prediction tensor is denoted as Y i ; and all first state prediction data of each first state prediction vector S i is extracted to form corresponding first obstacle state data; and the obstacle target type corresponding to the first classification prediction probability with the maximum probability in each first classification prediction vector Y i is taken as corresponding first obstacle type data; and the first obstacle state data and the first obstacle type data corresponding to the same obstacle index i are combined to form corresponding first obstacle prediction data according to the corresponding relationship of the obstacle index i; and the first obstacle prediction data of the obtained obstacle number B is combined to form a corresponding first obstacle prediction data set; and the first obstacle prediction data set is taken as output data integration processing result output;

[0167] Step 49, the obtained first obstacle prediction data set is taken as target state understanding and classification network processing result output.

[0168] Here, the automatic driving system of the vehicle realizes feature fusion of data features of multiple perception data sources (vehicle-mounted laser radar, vehicle-mounted millimeter wave radar and vehicle-mounted camera) through the above steps 1-4, and completes the synchronous prediction of the state+category of the obstacle target based on the fused features.

[0169] It should be noted that the automatic driving system of the vehicle can not only perform the synchronous prediction of the state+category based on the data features of the multiple perception data sources (vehicle-mounted laser radar, vehicle-mounted millimeter wave radar and vehicle-mounted camera) as shown above through the method of the embodiment of the application, but also can further perform the synchronous prediction of the state+category based on the data features of the multiple perception data sources+state filter data sources on the basis of the above scheme, where the state filter is a state filter (such as a Kalman filter) for predicting the obstacle state of each obstacle. The processing steps of the synchronous prediction of the state+category based on the data features of the multiple perception data sources+state filter data sources are as follows:

[0170] In the aforementioned step 1, while obtaining the perception data sequence of the vehicle-mounted laser radar, vehicle-mounted millimeter wave radar and vehicle-mounted camera in the first recent period T, additionally obtain the state data sequence of the state filter output for predicting the obstacle state of each obstacle at multiple time points in the first recent period T as the corresponding first obstacle state data sequence; the first obstacle state data sequence includes multiple first obstacle state data; the first obstacle state data includes one or more obstacle states; the obstacle state is composed of part or all of the heading angle, acceleration, speed, position and target detection box parameters; each first obstacle state data sequence corresponds to an obstacle target; each first obstacle state data corresponds to a first state time point;

[0171] After the aforementioned step 2, when the multi-class perception source data preprocessing is performed in the aforementioned step 3, in addition to the aforementioned steps 31-36, additionally initialize a tensor sequence composed of all-zero tensors with a shape of BxF6 in the number of time points L as the corresponding sixth tensor sequence, the sixth tensor sequence includes multiple sixth tensors with a shape of BxF6, and the feature dimension F6 is the number of obstacle states; and iterate through each sixth tensor of the sixth tensor sequence; when iterating, the currently iterated sixth tensor is taken as the corresponding current tensor, and the first time point corresponding to the current tensor is taken as the corresponding current time point; and confirm whether the current time point has a matching first state time point; if it is confirmed that the current time point has a matching first state time point, the matching first state time point is recorded as the current state time point, a first state tensor with a shape of BxF6 is composed of the first obstacle state data of the number B of obstacles corresponding to the current state time point, and the current tensor is updated to the first state tensor; when the iteration is completed, the sixth tensor sequence is tensor-converted according to the set tensor shape BxLxF3 to obtain the corresponding first state feature tensor;

[0172] And fine-tune the target state understanding and classification network in the aforementioned step 4, specifically: add a fully connected network similar to the first, second, third, fourth, and fifth fully connected networks to the embedding layer for fully connected processing, denoted as the sixth fully connected network, and add an encoding module based on the position encoding algorithm of the Transformer model to the position encoding layer of the target state understanding and classification network for encoding processing, denoted as the sixth encoding module, and add a sixth input end to the fusion layer, and connect the input of the sixth encoding module to the output of the sixth fully connected network, and connect the sixth input end of the fusion layer to the output end of the sixth encoding module; and configure the network parameters of the sixth fully connected network so that the output tensor shape of the fully connected network is B×L×A1; and configure the network parameters of the sixth encoding module so that the output tensor shape of the module is B×L×A1; and modify the output tensor shape of the fusion layer to B×L×A2 * , the feature dimension A2 * =(A1*N * ), the number of perception and state sources N * =the number of perception sources N+1.

[0173] And at the same time as inputting the multi-class perception source time feature tensor set and the multi-class perception source time mask tensor into the preset target state understanding and classification network in the aforementioned step 4 for obstacle target state prediction and obstacle target category prediction processing, also input the first state feature tensor into the target state understanding and classification network; and at the aforementioned step 41, additionally input the first state feature tensor into the embedding layer sixth fully connected network for fully connected processing to obtain the corresponding sixth fully connected tensor with a shape of B×L×A1; and at the aforementioned step 42, additionally input the sixth fully connected tensor into the corresponding sixth encoding module in the position encoding layer for position encoding processing to obtain the corresponding sixth position encoding tensor with a shape of B×L×A1; and at the aforementioned step 43, input the sixth position encoding tensor together with the first, second, third, fourth, and fifth position encoding tensors into the fusion layer for tensor splicing and fusion processing to obtain the second fusion tensor with a shape of B×L×A2 * ; and at the aforementioned step 44, input the second fusion tensor into the group normalization layer for group normalization processing to obtain the third normalized tensor with a shape of B×L×A2 * ; and at the aforementioned step 45, input the third normalized tensor and the multi-class perception source time mask tensor into the state encoding layer, and perform step-by-step encoding processing through the three-level state encoding modules of the state encoding layer to obtain the corresponding third encoding tensor with a shape of B×L×A2 *a second state encoding tensor; and inputting the second state encoding tensor into the state prediction branch network at the aforementioned step 46, performing obstacle target state regression prediction processing through the multi-level fully connected network of the state prediction branch network to obtain a corresponding second state prediction tensor; and inputting the second state encoding tensor into the classification prediction branch network at the aforementioned step 47, performing obstacle target classification prediction processing through the multi-level fully connected network of the classification prediction branch network to obtain a corresponding second classification prediction tensor; and inputting the second state prediction tensor and the second classification prediction tensor into the prediction data processing module for output data integration processing to obtain a corresponding second obstacle prediction data set at the aforementioned step 48; and outputting the obtained second obstacle prediction data set as the target state understanding and classification network processing result at the aforementioned step 49.

[0174] It also needs to be explained that the automatic driving system of the vehicle can not only perform state + category synchronous prediction based on the data features of the multiple perception data sources (referred to as the first scheme) and state + category synchronous prediction based on the data features of the multiple perception data sources + state filter data source (referred to as the second scheme) as shown above, but also can adopt the adjustment mode of the target state understanding and classification network in the processing steps of the state + category synchronous prediction based on the data features of the multiple perception data sources + state filter data source as described above, and further implement a dynamic prediction mode that adaptively matches the specific number of perception sources of the multiple perception data sources on the basis of the above two schemes, which is specifically: confirming the type set of multiple perception source types (vehicle-mounted laser radar type, vehicle-mounted millimeter wave radar type, and vehicle-mounted camera type) corresponding to the multiple perception data sequences currently obtained, and confirming whether the obstacle state data sequence output by the state filter is currently obtained; and based on the processing steps in steps 1-3 of the aforementioned first scheme, performing perception source data preprocessing on the perception data sequences corresponding to each perception source type; and when it is confirmed that the obstacle state data sequence output by the state filter is currently obtained, processing the obstacle state data sequence based on the processing steps of the aforementioned second scheme; and adaptively fine-tuning the target state understanding and classification network based on the multiple perception data sequences and / or the obstacle state data sequence currently obtained at the aforementioned step 4, and performing obstacle target state prediction and obstacle target category prediction processing based on the fine-tuned target state understanding and classification network to obtain a corresponding obstacle prediction data set.

[0175] Figure 3 A structural schematic diagram of an electronic device provided by the second embodiment of the present application. The electronic device can be the terminal device or the server as described above, or a terminal device or server connected to the terminal device or server as described above and implementing the method of the present application. As shown in FIG. 2, the electronic device includes a processor 21, a memory 22, a communication interface 23, and a power supply 24. The processor 21, the memory 22, the communication interface 23, and the power supply 24 are connected to each other through a bus. Figure 3As shown, the electronic device can include a processor 301 (e.g., a CPU), a memory 302, and a transceiver 303. The transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiving action of the transceiver 303. The memory 302 can store various instructions for completing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device related to the embodiments of the present application further includes a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize the communication connection between elements. The communication port 306 described above is used for the connection communication between the electronic device and other peripherals.

[0176] In Figure 3 The system bus 305 mentioned in the foregoing can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (e.g., a client, a read-write library, and a read-only library). The memory can include a Random Access Memory (RAM), and can also include a Non-Volatile Memory, such as at least one disk memory.

[0177] The processor described above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), a Graphics Processing Unit (GPU), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0178] It should be noted that the embodiments of the present application also provide a computer readable storage medium, which stores instructions, and when the instructions run on a computer, the computer executes the method and processing procedure provided in the foregoing embodiments.

[0179] The embodiment of the present application also provides a chip for running instructions, which is used for executing the processing steps described in the foregoing method embodiment.

[0180] The embodiment of the present application provides a processing method for predicting states and categories of obstacles based on multi-category perception features, an electronic device and a computer readable storage medium, feature fusion is performed on multi-category data features (basic data features, target detection features) and / or data features of state filters of multi-category perception data (laser radar, millimeter wave radar, camera, etc.), a target state understanding and classification network is constructed to perform end-to-end state+category synchronous prediction, and then the fused features are input into the target state understanding and classification network for prediction to obtain state and category prediction results of the obstacles. Through the present application, end-to-end state+category synchronous prediction is realized based on a whole solution, and the prediction accuracy of the solution is improved by fusing multi-category perception features and / or state features.

[0181] Those skilled in the art should further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both, and in order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0182] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0183] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application, and it should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for predicting obstacle status and category based on multi-class perception features, characterized in that: The method comprises: acquiring a perception data sequence of a vehicle-mounted laser radar, a vehicle-mounted millimeter wave radar and a vehicle-mounted camera in a first time period T as a corresponding first laser radar point cloud sequence, a first millimeter wave radar point cloud sequence and a first image sequence; dividing the first time period T into a plurality of first time points at a preset first time interval Δt, and taking the number of the first time points as a corresponding time point number L; performing multi-class perception source data preprocessing according to the first laser radar point cloud sequence, the first millimeter wave radar point cloud sequence, the first image sequence, all the first time points and the time point number L to obtain a corresponding multi-class perception source time feature tensor set and a corresponding multi-class perception source time mask tensor; inputting the multi-class perception source time feature tensor set and the multi-class perception source time mask tensor into a preset target state understanding and classification network to perform obstacle target state prediction and obstacle target category prediction processing to obtain a corresponding first obstacle prediction data set; the first obstacle prediction data set comprises a plurality of first obstacle prediction data; the first obstacle prediction data comprises first obstacle state data and first obstacle type data; wherein the target state understanding and classification network comprises an embedding layer, a position encoding layer, a fusion layer, a group normalization layer, a state encoding layer, a state prediction branch network, a classification prediction branch network and a prediction data processing module; the state encoding layer is implemented based on an Encoder module of a Transformer model; the state encoding layer comprises three levels of state encoding modules; an input end of a first level of the state encoding modules is connected with an output end of the fusion layer; an input end of a second level of the state encoding modules is connected with an output end of the first level of the state encoding modules; an input end of a third level of the state encoding modules is connected with an output end of the second level of the state encoding modules; an output end of the third level of the state encoding modules is an output end of the state encoding layer; each level of the state encoding modules comprises a multi-head attention network, a first sum and normalization network, a feedforward neural network and a second sum and normalization network; an input end of the multi-head attention network is connected with an input end of the state encoding module; a first input end of the first sum and normalization network is connected with an input end of the multi-head attention network, and a second input end is connected with an output end of the multi-head attention network; an input end of the feedforward neural network is connected with an output end of the first sum and normalization network; a first input end of the second sum and normalization network is connected with an input end of the feedforward neural network, and a second input end is connected with an output end of an output end of the feedforward neural network; an output end of the second sum and normalization network is an output end of the state encoding module.

2. The processing method for predicting obstacle state and category based on multi-class perception features according to claim 1, wherein the first laser radar point cloud sequence comprises a plurality of first laser radar point clouds, each of which corresponds to a first sampling time point; The first millimeter wave radar point cloud sequence includes a plurality of first millimeter wave radar point clouds, each of which corresponds to a second sampling time point; The first image sequence includes a plurality of first images, each of which corresponds to a third sampling time point; The multi-class perception source time feature tensor set includes a first point cloud time feature tensor, a first point cloud target time feature tensor, a second point cloud target time feature tensor, a first image time feature tensor, and a first image target time feature tensor; the shape of the first point cloud time feature tensor is B×L×F1, the shape of the first point cloud target time feature tensor is B×L×F2, the shape of the second point cloud target time feature tensor is B×L×F3, the shape of the first image time feature tensor is B×L×F4, and the shape of the first image target time feature tensor is B×L×F5; Obstacle The number B is greater than or equal to 1, and the feature dimensions F1, F2, F3, F4, and F5 are greater than or equal to 1; The shape of the multi-class perception source time mask tensor is B×L×N, and the number N of perception sources is 5 by default; the multi-class perception source time mask tensor is fused from the time mask tensors with the shape of B×L of the number N of perception sources, which are respectively: a first point cloud time mask tensor, a first point cloud target time mask tensor, a second point cloud target time mask tensor, a first image time mask tensor, and a first image target time mask tensor.

3. The processing method for predicting the state and class of an obstacle based on multi-class perception features according to claim 2, characterized in that, The multi-class perception source data preprocessing according to the first lidar point cloud sequence, the first millimeter wave radar point cloud sequence, the first image sequence, all the first time points, and the number L of time points obtains a corresponding multi-class perception source time feature tensor set and a corresponding multi-class perception source time mask tensor, and specifically includes: The lidar point cloud data preprocessing according to the first lidar point cloud sequence, all the first time points, and the number L of time points obtains the first point cloud time feature tensor, the first point cloud target time feature tensor, the first point cloud time mask tensor, and the first point cloud target time mask tensor; The millimeter wave radar point cloud data preprocessing according to the first millimeter wave radar point cloud sequence, all the first time points, and the number L of time points obtains the second point cloud target time feature tensor and the second point cloud target time mask tensor; The camera image data preprocessing according to the first image sequence, all the first time points, and the number L of time points obtains the first image time feature tensor, the first image target time feature tensor, the first image time mask tensor, and the first image target time mask tensor; The multi-class perception source time feature tensor set is composed of the first point cloud time feature tensor, the first point cloud target time feature tensor, the second point cloud target time feature tensor, the first image time feature tensor, and the first image target time feature tensor; tensor fusion on the first point cloud time mask tensor, the first point cloud target time mask tensor, the second point cloud target time mask tensor, the first image time mask tensor and the first image target time mask tensor to generate a multi-class perception source time mask tensor with a shape of BxLxN corresponding thereto; output the multi-class perception source time feature tensor set and the multi-class perception source time mask tensor obtained as a multi-class perception source data preprocessing result.

4. The processing method for predicting the state and class of an obstacle based on multi-class perception features according to claim 3, characterized in that, The laser radar point cloud data preprocessing according to the first laser radar point cloud sequence, all the first time points and the time point quantity L to obtain the first point cloud time feature tensor, the first point cloud target time feature tensor, the first point cloud time mask tensor and the first point cloud target time mask tensor corresponding thereto specifically comprises: perform point cloud feature extraction processing on each first laser radar point cloud of the first laser radar point cloud sequence to obtain a first point cloud feature tensor with a shape of BxF1 corresponding thereto; and perform obstacle target detection processing on each first point cloud feature tensor to obtain a first point cloud target feature tensor with a shape of BxF2 corresponding thereto; the first point cloud feature tensor and the first point cloud target feature tensor both correspond one-to-one to the first sampling time points; initialize a tensor sequence composed of full-zero tensors with a shape of BxF1 in the time point quantity L as a first tensor sequence corresponding thereto; and initialize a tensor sequence composed of full-zero tensors with a shape of BxF2 in the time point quantity L as a second tensor sequence corresponding thereto; the first tensor sequence comprises first tensors in the time point quantity L; the first tensor has a shape of BxF1, is a full-zero tensor and corresponds one-to-one to the first time points; the second tensor sequence comprises second tensors in the time point quantity L; the second tensor has a shape of BxF2, is a full-zero tensor and corresponds one-to-one to the first time points; perform traversal on each first tensor of the first tensor sequence; during traversal, take the first tensor currently traversed as a current tensor corresponding thereto, take the first time point corresponding to the current tensor as a current time point corresponding thereto, and confirm whether the current time point has a matching first sampling time point; if it is confirmed that the current time point has a matching first sampling time point, update the current tensor to the first point cloud feature tensor corresponding to the matching first sampling time point; and perform tensor conversion on the first tensor sequence according to a set tensor shape BxLxF1 to obtain the first point cloud time feature tensor corresponding thereto at the end of traversal; traversing each of the second tensors in the second tensor sequence; during the traversing, taking the currently traversed second tensor as a corresponding current tensor, taking the first time point corresponding to the current tensor as a corresponding current time point, and confirming whether the current time point has a matching first sampling time point; if it is confirmed that the current time point has a matching first sampling time point, updating the current tensor to the first point cloud target feature tensor corresponding to the matching first sampling time point; and after the traversing, performing tensor conversion on the second tensor sequence according to a set tensor shape B x L x F2 to obtain a corresponding first point cloud target time feature tensor; initializing two all-zero tensors with a shape of B x L as corresponding first point cloud time mask tensors and first point cloud target time mask tensors; the first point cloud time mask tensors include first column vectors with a length of the obstacle number B at the time point number L; the first point cloud target time mask tensors include second column vectors with a length of the obstacle number B at the time point number L; the first column vectors and the second column vectors correspond to the first time points one by one; traversing each of the first time points; during the traversing, taking the currently traversed first time point as a corresponding current time point, and confirming whether the current time point has a matching first sampling time point; if it is confirmed that the current time point has a matching first sampling time point, updating the first column vector corresponding to the current time point in the first point cloud time mask tensor to an all-1 column vector, and updating the second column vector corresponding to the current time point in the first point cloud target time mask tensor to an all-1 column vector; outputting the obtained first point cloud time feature tensor, first point cloud target time feature tensor, first point cloud time mask tensor and first point cloud target time mask tensor as a laser radar point cloud data preprocessing result.

5. The processing method for predicting the state and class of an obstacle based on multi-class perception features according to claim 3, characterized in that, the millimeter wave radar point cloud data preprocessing according to the first millimeter wave radar point cloud sequence, all the first time points and the time point number L to obtain corresponding second point cloud target time feature tensors and second point cloud target time mask tensors, specifically including: performing point cloud feature extraction processing on each of the first millimeter wave radar point clouds in the first millimeter wave radar point cloud sequence and obstacle target detection processing based on the extracted features to obtain a second point cloud target feature tensor with a shape of B x F3; the second point cloud target feature tensor corresponds to the second sampling time points one by one; initializing a tensor sequence composed of all-zero tensors with a shape of B x F3 at the time point number L as a corresponding third tensor sequence; the third tensor sequence includes third tensors at the time point number L; the third tensors have a shape of B x F3, are all-zero tensors, and correspond to the first time points one by one; traversing each of the third tensors of the third tensor sequence; during the traversal, taking the currently traversed third tensor as a corresponding current tensor, and taking the first time point corresponding to the current tensor as a corresponding current time point; and confirming whether the current time point has a matching second sampling time point; if it is confirmed that the current time point has a matching second sampling time point, updating the current tensor to the second point cloud target feature tensor corresponding to the matching second sampling time point; and after the traversal, performing tensor conversion on the third tensor sequence according to a set tensor shape B x L x F3 to obtain a corresponding second point cloud target time feature tensor; initializing a full-zero tensor with a shape of B x L as a corresponding second point cloud target time mask tensor; the second point cloud target time mask tensor includes third column vectors with a length of the obstacle quantity B for the time point quantity L; the third column vectors correspond to the first time points one by one; traversing each of the first time points; during the traversal, taking the currently traversed first time point as a corresponding current time point; and confirming whether the current time point has a matching second sampling time point; if it is confirmed that the current time point has a matching second sampling time point, updating the third column vector corresponding to the current time point in the second point cloud target time mask tensor to a full-1 column vector; outputting the obtained second point cloud target time feature tensor and the second point cloud target time mask tensor as millimeter wave radar point cloud data preprocessing results.

6. The processing method for predicting the state and class of an obstacle based on multi-class perception features according to claim 3, wherein, The camera image data preprocessing according to the first image sequence, all the first time points and the time point quantity L to obtain the corresponding first image time feature tensor, the first image target time feature tensor, the first image time mask tensor and the first image target time mask tensor specifically includes: performing image feature extraction processing on each of the first images of the first image sequence to obtain a first image feature tensor with a shape of B x F4; and performing obstacle target detection processing on each of the first image feature tensors to obtain a first image target feature tensor with a shape of B x F5; the first image feature tensor and the first image target feature tensor correspond to the third sampling time points one by one; initializing a tensor sequence composed of full-zero tensors with a shape of B x F4 for the time point quantity L as a corresponding fourth tensor sequence; and initializing a tensor sequence composed of full-zero tensors with a shape of B x F5 for the time point quantity L as a corresponding fifth tensor sequence; the fourth tensor sequence includes fourth tensors for the time point quantity L; the fourth tensor has a shape of B x F4, the fourth tensor is a full-zero tensor, and the fourth tensor corresponds to the first time points one by one; the fifth tensor sequence includes fifth tensors for the time point quantity L; the fifth tensor has a shape of B x F5, the fifth tensor is a full-zero tensor, and the fifth tensor corresponds to the first time points one by one; traversing each fourth tensor of the fourth tensor sequence; during the traversal, taking the currently traversed fourth tensor as a corresponding current tensor, and taking the first time point corresponding to the current tensor as a corresponding current time point; and confirming whether the current time point has a matching third sampling time point; if it is confirmed that the current time point has a matching third sampling time point, updating the current tensor to the first image feature tensor corresponding to the matching third sampling time point; and after the traversal, performing tensor conversion on the fourth tensor sequence according to a set tensor shape B x L x F4 to obtain a corresponding first image time feature tensor; traversing each fifth tensor of the fifth tensor sequence; during the traversal, taking the currently traversed fifth tensor as a corresponding current tensor, and taking the first time point corresponding to the current tensor as a corresponding current time point; and confirming whether the current time point has a matching third sampling time point; if it is confirmed that the current time point has a matching third sampling time point, updating the current tensor to the first image target feature tensor corresponding to the matching third sampling time point; and after the traversal, performing tensor conversion on the fifth tensor sequence according to a set tensor shape B x L x F5 to obtain a corresponding first image target time feature tensor; initializing two all-zero tensors with a shape of B x L as the first image time mask tensor and the first image target time mask tensor; the first image time mask tensor includes fourth column vectors with a length of the obstacle number B at the time point number L; the first image target time mask tensor includes fifth column vectors with a length of the obstacle number B at the time point number L; the fourth column vectors and the fifth column vectors correspond to the first time points one by one; traversing each first time point; during the traversal, taking the currently traversed first time point as a corresponding current time point; and confirming whether the current time point has a matching third sampling time point; if it is confirmed that the current time point has a matching third sampling time point, updating the fourth column vector corresponding to the current time point in the first image time mask tensor to an all-1 column vector, and updating the fifth column vector corresponding to the current time point in the first image target time mask tensor to an all-1 column vector; outputting the first image time feature tensor, the first image target time feature tensor, the first image time mask tensor, and the first image target time mask tensor obtained as a camera image data preprocessing result.

7. The processing method for predicting an obstacle state and category based on multi-category perception features according to claim 2, wherein the embedding layer includes five fully connected networks, which are first, second, third, fourth, and fifth fully connected networks, respectively. The position encoding layer includes five encoding modules, namely a first, a second, a third, a fourth and a fifth encoding module; the input ends of the first, second, third, fourth and fifth encoding modules are connected with the output ends of the corresponding first, second, third, fourth and fifth fully connected networks; The first, second, third, fourth and fifth input ends of the fusion layer are connected with the output ends of the first, second, third, fourth and fifth encoding modules respectively; The input end of the group normalization layer is connected with the output end of the fusion layer; The state prediction branch network is composed of a plurality of fully connected networks; the input end of the state prediction branch network is connected with the output end of the state encoding layer; The classification prediction branch network is composed of a plurality of fully connected networks; the input end of the classification prediction branch network is connected with the output end of the state encoding layer; The first input end of the prediction data processing module is connected with the output end of the state prediction branch network, and the second input end is connected with the output end of the classification prediction branch network.

8. The processing method for predicting the state and class of an obstacle based on multi-class perception features according to claim 7, characterized in that, The target state understanding and classification network inputs the first point cloud time feature tensor, the first point cloud target time feature tensor, the second point cloud target time feature tensor, the first image time feature tensor and the first image target time feature tensor of the input multi-class perception source time feature tensor set into the corresponding first, second, third, fourth and fifth fully connected networks in the embedding layer for fully connected processing to obtain corresponding first, second, third, fourth and fifth fully connected tensors; the shapes of the first, second, third, fourth and fifth fully connected tensors are all B×L×A1; the feature dimension A1 is 128 by default; The first, second, third, fourth and fifth fully connected tensors are input into the corresponding first, second, third, fourth and fifth encoding modules in the position encoding layer for position encoding processing to obtain corresponding first, second, third, fourth and fifth position encoding tensors; the shapes of the first, second, third, fourth and fifth position encoding tensors are all B×L×A1; The first, second, third, fourth and fifth position encoding tensors are input into the fusion layer for tensor splicing and fusion processing to obtain a corresponding first fusion tensor; the shape of the first fusion tensor is B×L×A2, and the feature dimension A2=(A1*N); The first fusion tensor is input into the group normalization layer for group normalization processing to obtain a corresponding first normalization tensor; the shape of the first normalization tensor is B×L×A2; The first normalization tensor is input into the state prediction branch network for state prediction processing to obtain a corresponding first state prediction tensor; the shape of the first state prediction tensor is B×L×A2; The first normalized tensor and the multi-class perception source time mask tensor are input into the state coding layer, and a corresponding first state coding tensor is obtained by performing step-by-step coding processing through three-level state coding modules of the state coding layer; the shape of the first state coding tensor is B×L×A3, A3 is a preset state tensor feature dimension; The first state coding tensor is input into the state prediction branch network, and a corresponding first state prediction tensor is obtained by performing obstacle target state regression prediction processing through a multi-level fully connected network of the state prediction branch network; the shape of the first state prediction tensor is B×A4, A4 is a preset state prediction tensor feature dimension; the first state prediction tensor includes a first state prediction vector of the obstacle number B; the vector length of the first state prediction vector is the feature dimension A4, and the vector data of the first state prediction vector includes first state prediction data of the feature dimension A4; the first state prediction includes a first obstacle speed; The first state coding tensor is input into the classification prediction branch network, and a corresponding first classification prediction tensor is obtained by performing obstacle target classification prediction processing through a multi-level fully connected network of the classification prediction branch network; the shape of the first classification prediction tensor is B×A5, A5 is a preset classification prediction tensor feature dimension; the first classification prediction tensor includes a first classification prediction vector of the obstacle number B; the vector length of the first classification prediction vector is the feature dimension A5, and the first classification prediction vector includes a first classification prediction probability of the feature dimension A5; each first classification prediction probability corresponds to an obstacle target type; The first state prediction tensor and the first classification prediction tensor are input into the prediction data processing module for output data integration processing to obtain a corresponding first obstacle prediction data set; The obtained first obstacle prediction data set is output as a target state understanding and classification network processing result.

9. The processing method for predicting the state and class of an obstacle based on multi-class perception features according to claim 8, characterized in that, The first, second, third, fourth and fifth encoding modules in the position coding layer are input into the first, second, third, fourth and fifth encoding modules for position coding processing to obtain corresponding first, second, third, fourth and fifth position coding tensors, and the specific operations include: The first, second, third, fourth and fifth encoding modules perform position coding on the input first, second, third, fourth and fifth fully connected tensors based on the position coding algorithm of the Transformer model to obtain the corresponding first, second, third, fourth and fifth position coding tensors.

10. The processing method for predicting the state and class of an obstacle based on multi-class perception features according to claim 8, wherein, The first normalized tensor and the multi-class perception source time mask tensor are input into the state coding layer, and a corresponding first state coding tensor is obtained by performing step-by-step coding processing through three-level state coding modules of the state coding layer, and the specific operations include: The state coding layer inputs the first normalized tensor and the multi-class perceptual source time mask tensor into the multi-head attention network of the first level state coding module for multi-head attention coding processing to obtain a corresponding first attention tensor; inputs the first attention tensor and the first normalized tensor into the first addition and normalization network of the first level state coding module for tensor addition and normalization processing to obtain a corresponding second normalized tensor; inputs the second normalized tensor into the feedforward neural network of the first level state coding module for nonlinear transformation processing to obtain a corresponding first transformed tensor; and inputs the first transformed tensor and the second normalized tensor into the second addition and normalization network of the first level state coding module for tensor addition and normalization processing to obtain a corresponding third normalized tensor; The third normalized tensor and the multi-class perceptual source time mask tensor are further input into the multi-head attention network of the second level state coding module of the state coding layer for multi-head attention coding processing to obtain a corresponding second attention tensor; the second attention tensor and the third normalized tensor are input into the first addition and normalization network of the second level state coding module for tensor addition and normalization processing to obtain a corresponding fourth normalized tensor; the fourth normalized tensor is input into the feedforward neural network of the second level state coding module for nonlinear transformation processing to obtain a corresponding second transformed tensor; and the second transformed tensor and the fourth normalized tensor are input into the second addition and normalization network of the second level state coding module for tensor addition and normalization processing to obtain a corresponding fifth normalized tensor; The fifth normalized tensor and the multi-class perceptual source time mask tensor are further input into the multi-head attention network of the third level state coding module of the state coding layer for multi-head attention coding processing to obtain a corresponding third attention tensor; the third attention tensor and the fifth normalized tensor are input into the first addition and normalization network of the third level state coding module for tensor addition and normalization processing to obtain a corresponding sixth normalized tensor; the sixth normalized tensor is input into the feedforward neural network of the third level state coding module for nonlinear transformation processing to obtain a corresponding third transformed tensor; and the third transformed tensor and the sixth normalized tensor are input into the second addition and normalization network of the third level state coding module for tensor addition and normalization processing to obtain a corresponding seventh normalized tensor; and the seventh normalized tensor is output as the first state coding tensor.

11. The processing method for predicting a state and a class of an obstacle based on multi-class perception features according to claim 8, wherein, The first state prediction tensor and the first classification prediction tensor are input into the prediction data processing module for output data integration processing to obtain a corresponding first obstacle prediction data set, specifically comprising: The prediction data processing module records each first state prediction vector of the first state prediction tensor as a corresponding first state prediction vector S i , i is an obstacle index, 1≤i≤B; and records each first classification prediction vector of the first classification prediction tensor as a corresponding first classification prediction vector Y i ; extracts all first state prediction data of each first state prediction vector S i to form corresponding first obstacle state data; extracts the first classification prediction probability corresponding to the maximum probability in each first classification prediction vector Y i as corresponding first obstacle type data; groups the first obstacle state data and the first obstacle type data corresponding to the same obstacle index i to form corresponding first obstacle prediction data according to the corresponding relationship of the obstacle index i; groups the first obstacle prediction data of the obtained obstacle number B to form a corresponding first obstacle prediction data set; and outputs the first obstacle prediction data set as the output data integration processing result.

12. An electronic device, comprising: comprising: a memory, a processor and a transceiver; The processor is used to couple with the memory, read and execute the instructions in the memory to realize the method of any one of claims 1-11; The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transceiving.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, when the computer instructions are executed by a computer, the computer executes the method in any one of claims 1-11.

Images