Early warning method and device, vehicle, medium, program product

Abstract

The application provides a pre-warning method and device, vehicle, medium and program product. The method comprises the following steps: acquiring first environment point cloud data of a vehicle in a current time window; determining a first adjacency matrix of the first environment point cloud data; determining an attention value between any two connected nodes in the first adjacency matrix; wherein the attention value is used to represent position information and semantic correlation strength between the any two connected nodes; predicting second environment point cloud data of the vehicle at a next time based on the attention value; and outputting an alarm signal in the case that an obstacle exists in a blind area of the vehicle based on the first environment point cloud data and the second environment point cloud data.

Description

Technical Field

[0001] This application relates to active safety technology, specifically to an early warning method and device, vehicle, medium, and program product. Background Technology

[0002] Currently, urban road traffic faces numerous challenges from a complex traffic environment comprised of motor vehicles, non-motor vehicles, and pedestrians. Traffic problems in this complex environment are intricate and multifaceted, with various modes of transportation sharing space, resulting in chaotic traffic order and frequent traffic accidents.

[0003] Blind spots are areas that cannot be directly observed by the driver or onboard image sensors. These areas may conceal pedestrians, bicycles, other vehicles, or obstacles. In complex traffic environments, intersections, high-speed driving, adverse weather conditions, and nighttime driving are all considered high-risk scenarios. In these scenarios, the existence of blind spots often leads to delayed reactions from drivers or vehicle driver assistance systems when accidents occur, or the systems fail to provide accurate and timely warnings, resulting in frequent traffic accidents. This is a major safety challenge in complex road traffic. Summary of the Invention

[0004] This application provides an early warning method, device, vehicle, medium, and program product, which is beneficial to improving the accuracy of predicted second environmental point cloud data.

[0005] In a first aspect, embodiments of this application provide an early warning method, the method comprising: acquiring first environmental point cloud data of a vehicle within a current time window; determining a first adjacency matrix of the first environmental point cloud data; determining an attention value between any two connected nodes in the first adjacency matrix; wherein the attention value is used to characterize the positional information and semantic association strength between the two connected nodes; predicting second environmental point cloud data of the vehicle at the next moment based on the attention value; and outputting an alarm signal when it is determined, based on the first environmental point cloud data and the second environmental point cloud data, that an obstacle exists in the blind spot of the vehicle.

[0006] It is understood that in the early warning method provided in this application embodiment, considering that there are interactions between points in the first environmental point cloud data, and the degree of interaction also varies, the attention value between any two adjacent nodes in the first adjacency matrix of the determined first environmental point cloud data, that is, the positional information and semantic association strength between any two adjacent nodes, is used to predict the second environmental point cloud data at the next moment. In other words, spatial information is introduced in the prediction process to improve the accuracy of the predicted second environmental point cloud data, thereby improving the safety of the vehicle during driving.

[0007] In some embodiments, determining the attention value between any two connected nodes in the first adjacency matrix includes: determining a set of representation vectors for the first adjacency matrix; wherein the set of representation vectors includes: the representation vector of any node in the first adjacency matrix; and determining the attention value between any two connected nodes based on the representation vectors of any two connected nodes.

[0008] It is understood that in the early warning method provided in this application embodiment, the attention value between two connected nodes is not directly determined based on the first adjacency matrix. Instead, the nodes in the first adjacency matrix are represented by representation vectors, and then the attention value between any two connected nodes is determined based on the representation vectors. In this way, graph data is transformed from discrete, unstructured adjacency relationships into continuous, computable mathematical expressions. This allows for compatibility with machine learning or deep learning algorithms and facilitates the learning of semantic information between nodes based on representation vectors, thereby improving the accuracy of determining attention values, increasing the prediction accuracy of environmental data for the vehicle in the next moment, and ultimately improving vehicle safety during driving.

[0009] In some embodiments, determining the representation vector set of the first adjacency matrix includes: performing a dimensionality increase operation on the i-th node in the first adjacency matrix to obtain the representation vector of the i-th node; wherein i is greater than 0 and less than or equal to the number of nodes in the first adjacency matrix; and obtaining the representation vector set based on the representation vectors of the nodes in the first adjacency matrix.

[0010] It is understood that in the early warning method provided in this application embodiment, the first adjacency matrix records the low-dimensional features between nodes. Performing a dimensionality increase operation on the nodes in the first adjacency matrix can capture high-dimensional features, thereby encoding structural information into a dense vector. Furthermore, subtle differences in low-dimensional features are more easily distinguished in the high-dimensional vector. Thus, mapping nodes to a higher-dimensional space is beneficial for capturing finer-grained feature representations between nodes, thereby improving the accuracy of determining attention values, increasing the prediction accuracy of environmental data for the vehicle in the next moment, and ultimately improving vehicle safety during driving.

[0011] In some embodiments, predicting the second environmental point cloud data of the vehicle at the next time step based on the attention value includes: performing a first process on the attention value between the i-th node of the first adjacency matrix and the j-th neighbor node of the i-th node; wherein the first process includes: normalizing the attention value between the i-th node and the j-th neighbor node of the i-th node to obtain a normalized value; wherein j is greater than 0 and less than or equal to the number of neighbor nodes of the i-th node; if the normalized value is less than a normalization threshold, deleting the connection between the i-th node and the j-th neighbor node of the i-th node; iteratively executing the first process until all nodes in the first adjacency matrix have performed the first process to obtain an attention graph structure; and predicting the second environmental point cloud data of the vehicle at the next time step based on the attention graph structure.

[0012] It is understood that in the early warning method provided in this application embodiment, the attention value between any two connected nodes is normalized, and the connection between the two connected nodes is deleted if the normalized value is less than or equal to a normalization threshold. That is, if the attention value between any two connected nodes is lower than a certain threshold, their connection is deleted. This improves the differentiation information between nodes, avoids the problem of excessive homogenization of node features due to too many connections, and allows the model to selectively aggregate neighborhood information during the prediction process, thereby avoiding over-smoothing when aggregating high-order neighborhood features and improving the accuracy of the prediction results.

[0013] In some embodiments, predicting the second environmental point cloud data of the vehicle at the next time step based on the attention graph structure includes: constructing a second adjacency matrix of the attention graph structure; filling the second adjacency matrix with a normalized value greater than or equal to the normalization threshold to obtain an attention graph adjacency matrix; and predicting the second environmental point cloud data of the vehicle at the next time step based on the attention graph adjacency matrix.

[0014] It is understood that in the early warning method provided in this application embodiment, filling the normalized value that is greater than or equal to the normalization threshold into the second adjacency matrix corresponding to the attention graph structure is beneficial to further improve the differential information between nodes, thereby further avoiding the over-smoothing phenomenon when aggregating high-order neighborhood features; and in the prediction process, it enables the model to focus on important connections, avoid low-quality edges from distracting attention, and enable the model to capture feature information more accurately, thereby improving the accuracy of the prediction results.

[0015] In some embodiments, predicting the second environmental point cloud data of the vehicle at the next time step based on the attention map adjacency matrix includes: extracting environmental features from the attention map adjacency matrix to obtain feature vectors of environmental point cloud data at different times within the current time window; aggregating the feature vectors at different times to obtain a feature vector sequence; and inputting the feature vector sequence into a trained multilayer self-attention network to predict the second environmental point cloud data to obtain the second environmental point cloud data.

[0016] It is understood that in the early warning method provided in this application embodiment, the feature vectors obtained from environmental feature extraction of the adjacency matrix of the attention map at different times are aggregated to obtain a feature vector sequence; this feature vector sequence is then used as the input of a trained multi-layer attention network. In this way, the trained multi-layer attention network can capture the temporal features of minute temporal movements and, combined with spatial information, collaboratively improve the accuracy of the predicted second environmental point cloud data.

[0017] In some embodiments, the trained multilayer self-attention network is trained based on a contrastive loss function of sinusoidal similarity.

[0018] It is understood that the reason why the contrastive loss function of sinusoidal similarity is used as the loss function of the multilayer self-attention network in the early warning method provided in this application is because the inventors of this application have experimentally verified that the multilayer self-attention network trained using the contrastive loss function of sinusoidal similarity has high prediction accuracy. Therefore, in this application embodiment, the well-trained multilayer self-attention network trained based on the contrastive loss function of sinusoidal similarity is beneficial to improving the accuracy of the predicted second environment point cloud data.

[0019] Secondly, embodiments of this application provide a warning device, the device comprising: an acquisition module configured to acquire first environmental point cloud data of a vehicle within a current time window; a first determination module configured to determine a first adjacency matrix of the first environmental point cloud data; a second determination module configured to determine an attention value between any two connected nodes in the first adjacency matrix; wherein the attention value is used to characterize the positional information and semantic association strength between the two connected nodes; a prediction module configured to predict second environmental point cloud data of the vehicle at the next moment based on the attention value; and a triggering module configured to output an alarm signal when it is determined that there is an obstacle in the blind spot of the vehicle based on the first environmental point cloud data and the second environmental point cloud data.

[0020] Thirdly, embodiments of this application provide a vehicle including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement the warning method described in the first aspect.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the early warning method described in the first aspect.

[0022] Fifthly, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the early warning method described in the first aspect. Attached Figure Description

[0023] Figure 1 A schematic diagram illustrating the implementation process of an early warning method provided in an embodiment of this application;

[0024] Figure 2 This is a schematic diagram illustrating the implementation process for determining attention values, provided in an embodiment of this application.

[0025] Figure 3 A schematic diagram illustrating the implementation process of determining a set of representation vectors provided in this application embodiment;

[0026] Figure 4 A schematic diagram of an implementation process for predicting second environmental point cloud data provided in this application embodiment. Figure 1 ;

[0027] Figure 5 A schematic diagram of an implementation process for predicting second environmental point cloud data provided in this application embodiment. Figure 2 ;

[0028] Figure 6 A schematic diagram of an implementation process for predicting second environmental point cloud data provided in this application embodiment. Figure 3 ;

[0029] Figure 7 A schematic diagram illustrating the implementation process of a vehicle blind spot safety prediction method based on point cloud trajectory attention map in complex traffic scenarios, provided in an embodiment of this application;

[0030] Figure 8 A schematic diagram of a graph attention network provided in an embodiment of this application;

[0031] Figure 9 A schematic diagram of a vehicle blind spot safety prediction system based on point cloud trajectory attention map in complex traffic scenarios provided in this application embodiment;

[0032] Figure 10This is a schematic diagram of an early warning device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below in conjunction with the accompanying drawings. The embodiments described below are only some embodiments of this application, not all embodiments. Therefore, the described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In the following description, references to “some embodiments” or “other embodiments” describe a subset of all possible embodiments. However, it is understood that “some embodiments” or “other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0038] When high-risk scenarios occur, if real-time safety prediction of vehicle blind spots can be achieved, both the driver and the vehicle's driver assistance system can receive early warning signals and take corresponding actions, thereby effectively avoiding traffic accidents. Therefore, an accurate and efficient safety prediction method for vehicle blind spots is needed to achieve real-time safety monitoring in high-risk scenarios and minimize the occurrence of accidents.

[0039] In a related technology, a system for compensating for blind spots in vehicle LiDAR obstacle recognition is provided. This system includes: a LiDAR for acquiring preset point cloud data of obstacles within a preset scanning range; multiple ultrasonic radars for acquiring preset point data of obstacles within a preset sensing range; and a shared spatial processing module for performing preset time clustering processing on the acquired preset point cloud data and preset point data according to a preset time clustering method, thereby obtaining obstacle information within the vehicle's preset sensing range and preset scanning range at the same time. This method can compensate for LiDAR blind spots with relatively little data. However, this technology does not consider the interaction of point clouds in spatial information, and the movement of people is often highly correlated with environmental information; treating points in the point cloud as isolated points for prediction is inaccurate.

[0040] Another related technology provides a crowd trajectory prediction method based on a multi-level spatiotemporal graph neural network. This method extracts the motion state features of pedestrians from their historical trajectories in a scene; models the pedestrian movement process as a multi-level dynamic spatiotemporal directed graph, processes it using a multi-level spatiotemporal graph neural network, and extracts the motion interaction features of each pedestrian. During extraction, interaction features at different distance ranges are extracted at different levels, and then the interaction features from different levels are fused to obtain the final interaction features; the future trajectory of pedestrians is predicted based on the motion interaction features and motion state features. This method improves the accuracy of crowd trajectory prediction by making pedestrian interaction modeling more detailed and reasonable through differentiated modeling. However, although this technology combines spatial structure with temporal information, it assumes that the points in the space are homogeneous. Points in the space should be selectively aggregated because the motion of crowds differs significantly from that of vehicles during vehicle movement. If all points in the point cloud are aggregated indiscriminately, oversmoothing will occur due to the inherent limitation of graph neural networks in aggregating high-order neighborhood features, thus interfering with the prediction results.

[0041] In this application embodiment, an early warning method is provided. Figure 1 This is a schematic diagram illustrating the implementation process of an early warning method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes steps 101 to 105:

[0042] Step 101: Obtain the first environmental point cloud data of the vehicle within the current time window;

[0043] Step 102: Determine the first adjacency matrix of the first environmental point cloud data;

[0044] Step 103: Determine the attention value between any two connected nodes in the first adjacency matrix; wherein the attention value is used to characterize the positional information and semantic association strength between any two connected nodes.

[0045] Step 104: Based on the attention value, predict the second environmental point cloud data of the vehicle at the next moment;

[0046] Step 105: If it is determined that there is an obstacle in the blind spot of the vehicle based on the first environmental point cloud data and the second environmental point cloud data, an alarm signal is output.

[0047] It is understood that in the early warning method provided in this application embodiment, considering that there are interactions between points in the first environmental point cloud data, and the degree of interaction also varies, the attention value between any two adjacent nodes in the first adjacency matrix of the determined first environmental point cloud data, that is, the positional information and semantic association strength between any two adjacent nodes, is used to predict the second environmental point cloud data at the next moment. In other words, spatial information is introduced in the prediction process to improve the accuracy of the predicted second environmental point cloud data, thereby improving the safety of the vehicle during driving.

[0048] The following sections will describe further optional implementation methods for each of the above steps, as well as related terms.

[0049] In step 101, the first environmental point cloud data of the vehicle within the current time window is acquired.

[0050] It should be understood that the method of acquiring the first environmental point cloud data is not limited in the embodiments of this application. In some embodiments, the first environmental point cloud data can be acquired by sensors installed on the vehicle. Exemplarily, in one possible implementation, the sensors include, but are not limited to, at least one of the following: millimeter-wave radar, ultrasonic radar, lidar, camera, etc.

[0051] In this embodiment, the length of the current time window is not limited. In some embodiments, the current time window is of fixed duration; in other embodiments, the current time window is of variable duration. In this embodiment, the first environmental point cloud data is also not limited. In some embodiments, the first environmental point cloud data includes the vehicle's surrounding environment data acquired by sensors within the current time window.

[0052] In step 102, the first adjacency matrix of the first environmental point cloud data is determined.

[0053] In some embodiments, the first adjacency matrix is ​​used to represent the spatial relationships or topological structure between points in the first environmental point cloud data. It should be understood that the method for constructing the first adjacency matrix is ​​not limited in the embodiments of this application. In some embodiments, the method for constructing the first adjacency matrix includes, but is not limited to, at least one of the following: distance-based adjacency, feature similarity-based adjacency, etc.

[0054] In some embodiments, determining the first adjacency matrix of the first environmental point cloud data includes: fusing the first environmental point cloud data to obtain an environmental point cloud trajectory; clustering the environmental point cloud trajectory to obtain at least one cluster; performing edge processing on the nodes within the at least one cluster, and performing edge processing on the clusters to obtain a graph structure; and constructing the first adjacency matrix of the graph structure.

[0055] In some embodiments, fusing the first environmental point cloud data to obtain the environmental point cloud trajectory includes: sequentially performing spatiotemporal synchronization and spatial calibration on the first environmental point cloud data to obtain the environmental point cloud trajectory.

[0056] In some other embodiments, fusing the first environmental point cloud data to obtain the environmental point cloud trajectory includes: sequentially performing spatiotemporal synchronization, spatial calibration, and noise filtering on the first environmental point cloud data to obtain the environmental point cloud trajectory.

[0057] In some embodiments, before fusing the first environmental point cloud data, the method further includes: preprocessing the first environmental point cloud data, and using the preprocessed first environmental point cloud data as the first environmental point cloud data; wherein the preprocessing includes: radius filtering and uniform downsampling.

[0058] For example, in one possible implementation, a radius filter is used to remove noise points and isolated points in the first environmental point cloud data, while retaining key point cloud information. First, a suitable radius value r is selected. For each point p in the first environmental point cloud data, all points within its neighborhood with a distance less than r are found. If the number of all points within its neighborhood with a distance less than r is greater than a certain threshold (in this embodiment, the minimum number of points is set to k), then the mean can be calculated using the following formula (1):

[0059]

[0060] Where N is the number of neighborhood points within radius r, p j These are the coordinates of the neighboring points. Finally, the mean value calculated using this formula is used to replace the original point p; based on this, the filtered point cloud data is obtained.

[0061] Then, uniform downsampling is performed on the filtered point cloud data to reduce the number of points in the filtered point cloud data while preserving key spatial information. First, the voxel size δ is determined, and the point cloud space (which can be understood as the filtered point cloud data) is divided into uniform cubic voxels, each with a side length of δ. For each voxel, all points falling within that voxel are found, and its center point is calculated using the following formula (2). Then, the calculated value is used to replace the value within each voxel.

[0062]

[0063] Where N is the number of points falling within a voxel, p i These are the coordinates of these points.

[0064] Next, the point cloud data after uniform downsampling is fused. The point cloud data after uniform downsampling is fused by time synchronization and spatial alignment to ensure data consistency and comparability, thereby obtaining the environmental point cloud trajectory.

[0065] Time synchronization is used to ensure that data from different sensors (which can be understood here as point cloud data after uniform downsampling) are aligned in time to provide consistent spatiotemporal information; the first step is to obtain the sensor's timestamp t. tensor Select a timestamp from a primary sensor as the base time t. ref The time difference Δt for each sensor is calculated using the following formula (3):

[0066] Δt=t ref -t tensor (3)

[0067] For each sensor's data, interpolation or delay processing is performed based on the time deviation to align it to the reference time. Specifically, if the previous time is used as the reference, the point cloud data from the next time is used for interpolation; if the next time is used as the reference, the point cloud data from the previous time is used for delay processing.

[0068] Spatial calibration is used to unify the coordinate systems of different sensors into a single reference coordinate system. First, some known reference points are selected (such as specific calibration objects or ground markers), and the spatial transformation matrix T for each sensor is calculated using the least squares method.

[0069] T = argmin∑ i ||pi m easured -T(p i true )|| 2 (4)

[0070] Where, pi measured It is the measured point, p i true These are known real points. The argmin function is used to find the value of the independent variable that minimizes the objective function. It is an abbreviation for "argument of the minimum". The point cloud data of the sensor (which can be understood as the point cloud data after spatiotemporal synchronization) is spatially calibrated through the transformation matrix T; where the spatial calibration formula is as follows (5):

[0071] p corrected =T·p measured (5)

[0072] Furthermore, filtering techniques (such as Kalman filtering) are applied to reduce noise and uncertainty. Point cloud data generated by various image sensors (i.e., an example of pre-processed point cloud data) are fused through time synchronization and spatial alignment to generate environmental point cloud trajectories. First, the state transition model and observation model are defined using the following formulas (6) and (7):

[0073] x k =Fx k-1 +Bu k +w k (6)

[0074] z k =Hx k +v k (7)

[0075] Where F is the state transition matrix, H is the observation matrix, and w k and v k These represent process noise and observation noise, respectively, x k Let u represent the state estimate vector of the system at time k. k Let B represent the control input vector, and let x represent the control input matrix. k-1 Let z represent the state estimate vector of the system at time k-1. k Let K represent the observation vector. Then, calculate the Kalman gain K using the following formula (8):

[0076] K = P k H T HP k H T +R) -1 (8)

[0077] The state estimate covariance is then updated using the following formulas (9) and (10):

[0078] x k|k =x k|k-1 +K(zk -Hx k|k-1 (9)

[0079] P k|k =(I-KH)P k|k-1 (10)

[0080] Where R represents the observation noise covariance matrix, P k This represents the error covariance matrix.

[0081] Finally, the k-means clustering algorithm is used to cluster the environmental point cloud trajectories into clusters, and the points within a cluster are connected by edges. The nearest points are connected between different clusters to form an original graph structure. The first adjacency matrix of the original graph structure is then constructed.

[0082] In step 103, the attention value between any two connected nodes in the first adjacency matrix is ​​determined; wherein the attention value is used to characterize the positional information and semantic association strength between any two connected nodes.

[0083] In some embodiments, the semantic association strength refers to the degree of association between different nodes based on their respective semantic categories (such as object categories, component labels, etc.).

[0084] In some embodiments, Figure 2 This is a schematic diagram illustrating an implementation process for determining attention values, as provided in an embodiment of this application. Figure 2 As shown, the attention value between any two connected nodes in the first adjacency matrix can be determined through the following steps 201 to 202:

[0085] Step 201: Determine the representation vector set of the first adjacency matrix; wherein the representation vector set includes: the representation vector of any node in the first adjacency matrix;

[0086] Step 202: Determine the attention value between any two connected nodes based on the representation vectors of any two connected nodes.

[0087] It is understood that in the early warning method provided in this application embodiment, the attention value between two connected nodes is not directly determined based on the first adjacency matrix. Instead, the nodes in the first adjacency matrix are represented by representation vectors, and then the attention value between any two connected nodes is determined based on the representation vectors. In this way, graph data is transformed from discrete, unstructured adjacency relationships into continuous, computable mathematical expressions. This allows for compatibility with machine learning or deep learning algorithms and facilitates the learning of semantic information between nodes based on representation vectors, thereby improving the accuracy of determining attention values, increasing the prediction accuracy of environmental data for the vehicle in the next moment, and ultimately improving vehicle safety during driving.

[0088] It should be understood that the method for determining the attention value is not limited in the embodiments of this application. In some embodiments, the method for determining the attention value includes, but is not limited to, at least one of the following: calculating the attention value based on a multilayer perceptron, calculating the attention value based on a graph attention network, and calculating the attention value based on a hybrid attention mechanism.

[0089] For example, in one possible implementation, a graph attention network is used to extract the attention between each node. For any connected pair of nodes, a vector mapping is performed using W, the mapping results are concatenated, and then passed through an activation function to obtain the attention value between the two points; where the attention value e ij The calculation formula is as follows (11).

[0090]

[0091] e ij This represents the attention value between nodes (i,j). represents the vector representations of nodes i and j respectively; W is a learnable parameter of a single vector mapping, LeakyReLU is the activation function, and || represents the concatenation operation.

[0092] In some embodiments, Figure 3 This application provides a schematic diagram of an implementation process for determining a set of representation vectors, as illustrated in the embodiments of this application. Figure 3 As shown, the set of representation vectors for the first adjacency matrix can be determined through the following steps 301 to 302:

[0093] Step 301: Perform a dimension-up operation on the i-th node in the first adjacency matrix to obtain the representation vector of the i-th node; where i is greater than 0 and less than or equal to the number of nodes in the first adjacency matrix;

[0094] Step 302: Based on the representation vectors of the nodes in the first adjacency matrix, obtain the set of representation vectors.

[0095] It is understood that in the early warning method provided in this application embodiment, the first adjacency matrix records the low-dimensional features between nodes. Performing a dimensionality increase operation on the nodes in the first adjacency matrix can capture high-dimensional features, thereby encoding structural information into a dense vector. Furthermore, subtle differences in low-dimensional features are more easily distinguished in the high-dimensional vector. Thus, mapping nodes to a higher-dimensional space is beneficial for capturing finer-grained feature representations between nodes, thereby improving the accuracy of determining attention values, increasing the prediction accuracy of environmental data for the vehicle in the next moment, and ultimately improving vehicle safety during driving.

[0096] It should be understood that the specific implementation method of the dimensionality upgrade operation is not limited in the embodiments of this application. In some embodiments, the dimensionality upgrade operation can be implemented by, but is not limited to, the following methods: linear transformation (e.g., fully connected layer, multilayer perceptron), nonlinear activation function, graph embedding method, etc.

[0097] For example, in one possible implementation, a vector embedding is performed on the first adjacency matrix using an N*128 linear matrix to obtain a representation vector of the first adjacency matrix. Each node in the adjacency matrix is ​​embedded as a 128-dimensional vector, where N is the number of point clouds represented by the first adjacency matrix.

[0098] In step 104, based on the attention value, the second environmental point cloud data of the vehicle at the next moment is predicted.

[0099] In some embodiments, Figure 4 A schematic diagram of an implementation process for predicting second environmental point cloud data provided in this application embodiment. Figure 1 ,like Figure 4 As shown, the second environmental point cloud data of the vehicle at the next moment can be predicted through the following steps 401 to 403:

[0100] Step 401: Perform a first process on the attention value between the i-th node and the j-th neighbor node of the i-th node in the first adjacency matrix; wherein, the first process includes: normalizing the attention value between the i-th node and the j-th neighbor node of the i-th node to obtain a normalized value; wherein, j is greater than 0 and less than or equal to the number of neighbor nodes of the i-th node; if the normalized value is less than a normalization threshold, delete the connection between the i-th node and the j-th neighbor node of the i-th node.

[0101] Step 402: Iteratively execute the first process until all nodes in the first adjacency matrix have executed the first process, and obtain the attention graph structure;

[0102] Step 403: Based on the attention map structure, predict the second environmental point cloud data of the vehicle at the next moment.

[0103] It is understood that in the early warning method provided in this application embodiment, the attention value between any two connected nodes is normalized, and the connection between the two connected nodes is deleted if the normalized value is less than or equal to a normalization threshold. That is, if the attention value between any two connected nodes is lower than a certain threshold, their connection is deleted. This improves the differentiation information between nodes, avoids the problem of excessive homogenization of node features due to too many connections, and allows the model to selectively aggregate neighborhood information during the prediction process, thereby avoiding over-smoothing when aggregating high-order neighborhood features and improving the accuracy of the prediction results.

[0104] It should be understood that the normalization threshold is not limited in the embodiments of this application. In some embodiments, the normalization threshold can be determined based on empirical values. For example, in one possible implementation, the normalization threshold is 0.5.

[0105] For example, in one possible implementation, attention normalization is performed using the following formula (12) to remove edges below a threshold in the graph structure (i.e., the graph structure corresponding to the first adjacency matrix), thus transforming the graph structure into an attention graph structure:

[0106]

[0107] in, Representing the neighboring nodes of node i, with a threshold of 0.5, softmax maps each element of the input vector to a value between 0 and 1 through exponential operations and normalization, and the sum of all output values ​​is 1.

[0108] In some embodiments, Figure 5 A schematic diagram of an implementation process for predicting second environmental point cloud data provided in this application embodiment. Figure 2 ,like Figure 5 As shown, the second environmental point cloud data of the vehicle at the next moment can be predicted through the following steps 501 to 503:

[0109] Step 501: Construct the second adjacency matrix of the attention graph structure;

[0110] Step 502: Fill the normalized value that is greater than or equal to the normalization threshold into the second adjacency matrix to obtain the attention graph adjacency matrix;

[0111] Step 503: Based on the attention map adjacency matrix, predict the second environmental point cloud data of the vehicle at the next moment.

[0112] It is understood that in the early warning method provided in this application embodiment, filling the normalized value that is greater than or equal to the normalization threshold into the second adjacency matrix corresponding to the attention graph structure is beneficial to further improve the differential information between nodes, thereby further avoiding the over-smoothing phenomenon when aggregating high-order neighborhood features; and in the prediction process, it enables the model to focus on important connections, avoid low-quality edges from distracting attention, and enable the model to capture feature information more accurately, thereby improving the accuracy of the prediction results.

[0113] In some embodiments, Figure 6 A schematic diagram of an implementation process for predicting second environmental point cloud data provided in this application embodiment. Figure 3 ,like Figure 6 As shown, the second environmental point cloud data of the vehicle at the next moment can be predicted through the following steps 601 to 603:

[0114] Step 601: Extract environmental features from the attention map adjacency matrix to obtain feature vectors of environmental point cloud data at different times within the current time window;

[0115] Step 602: Aggregate the feature vectors at different times to obtain a feature vector sequence;

[0116] Step 603: Input the feature vector sequence into the trained multilayer self-attention network to predict the second environment point cloud data, and obtain the second environment point cloud data.

[0117] It is understood that in the early warning method provided in this application embodiment, the feature vectors obtained from environmental feature extraction of the adjacency matrix of the attention map at different times are aggregated to obtain a feature vector sequence; this feature vector sequence is then used as the input of a trained multi-layer attention network. In this way, the trained multi-layer attention network can capture the temporal features of minute temporal movements and, combined with spatial information, collaboratively improve the accuracy of the predicted second environmental point cloud data.

[0118] For example, in one possible implementation, a graph neural network is used to perform vector aggregation to extract environmental information; wherein, the formula for calculating the node feature matrix of the (l+1)th layer is as follows (13):

[0119]

[0120] Among them, X (l) X is the feature matrix of the nodes in the l-th layer. (l+1) W is the node feature matrix of the (l+1)th layer. (b) Here, σ represents the training parameters, and σ represents the activation function (such as the linear rectified unit function). This is the adjacency matrix after attention is applied. A two-layer graph neural network model is then constructed based on this matrix to aggregate environmental information.

[0121] The aggregated point cloud vectors (an example of feature vectors) within a time window are packaged together to form a point cloud sequence (an example of a feature vector sequence). The point cloud vectors in the aggregated feature vector sequence are then passed through a multi-layer self-attention layer to extract time-series information. The calculation process is as follows:

[0122] Input the aggregated point cloud vector within a time window Where n is the sequence length and d is the feature dimension (taken as 128); the input X is transformed by three different linear transformations, namely by formula (14), to generate query, key, and value matrices:

[0123] Q = XW Q K = XW K V = XW V (14)

[0124] Among them, W Q W K W V These are all learnable weight parameters, and their function is to perform a linear transformation on the original moments.

[0125] Next, the similarity between the query and the key is calculated using the dot product, resulting in the attention score matrix, AttentionScores.

[0126] Attention Scores = QK T (15)

[0127] Scale these scores to avoid the vanishing gradient problem when the dot product result is large:

[0128]

[0129] Applying the Softmax function to the scaled score matrix yields the attention weight matrix:

[0130]

[0131] The value matrix is ​​weighted and summed using the attention weight matrix to obtain the final self-attention output:

[0132]

[0133] The result is The matrix represents the sequence features after the self-attention mechanism, and the last row vector is taken as the vector prediction for the next time step of the node.

[0134] In some embodiments, the trained multilayer self-attention network is trained based on a contrastive loss function of sinusoidal similarity.

[0135] It is understood that the reason why the contrastive loss function of sinusoidal similarity is used as the loss function of the multilayer self-attention network in the early warning method provided in this application is because the inventors of this application have experimentally verified that the multilayer self-attention network trained using the contrastive loss function of sinusoidal similarity has high prediction accuracy. Therefore, in this application embodiment, the well-trained multilayer self-attention network trained based on the contrastive loss function of sinusoidal similarity is beneficial to improving the accuracy of the predicted second environment point cloud data.

[0136] For example, in one possible implementation, during training, the loss function is calculated for the output matrix:

[0137]

[0138] in, Let j be an indicator function, indicating that j ≠ i 1 If the condition is met, then take 1; otherwise, take 0. , respectively, are the predicted i-node representation vector and the actual i-node representation vector; M is the total number of monitored point cloud nodes, and τ is the temperature parameter, set to 0.05.

[0139] The similarity function sim is calculated using the following formula (20):

[0140]

[0141] Then the loss of this function over the entire training set, i.e. the average loss for each data point, is calculated using the following formula (21):

[0142]

[0143] The gradient descent algorithm is used to minimize the loss function, maximizing the similarity between the predicted and true representation vectors of the same node. Representation vectors between different nodes are spaced as far apart as possible to update the parameters in the multi-layer self-attention network.

[0144] In step 105, if it is determined that there is an obstacle in the blind spot of the vehicle based on the first environmental point cloud data and the second environmental point cloud data, an alarm signal is output.

[0145] In some embodiments, the step of outputting an alarm signal when it is determined that there is an obstacle in the blind spot of the vehicle based on the first environmental point cloud data and the second environmental point cloud data includes: performing a difference operation on the first environmental point cloud data and the second environmental point cloud data to obtain the blind spot obstacle point cloud data of the vehicle; determining the blind spot obstacle based on the blind spot obstacle point cloud data, and outputting an alarm signal.

[0146] In some implementations, determining blind zone obstacles based on the blind zone obstacle point cloud data includes: inputting the blind zone obstacle point cloud data into a deep learning model to obtain the blind zone obstacles.

[0147] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0148] This application provides a method and system for predicting vehicle blind spot safety in complex traffic scenarios based on point cloud trajectory attention maps (i.e., an example of an early warning method) to address high-risk scenarios in complex traffic environments. This method can predict vehicle blind spot safety in complex traffic environments, proactively address potential traffic accidents, accurately and in real-time ensure vehicle blind spot safety in high-risk scenarios, improve the active safety performance of vehicle systems, and reduce the traffic accident rate.

[0149] For example, in one possible implementation, the method includes the following steps:

[0150] Step S1, data acquisition: environmental data is acquired through image sensors to obtain environmental information and obtain multi-sensor initial point cloud data (i.e., first environmental point cloud data);

[0151] Step S2, data processing: preprocess and fuse the first environmental point cloud data to generate environmental point cloud trajectories;

[0152] Step S3, trajectory prediction: The algorithm performs point cloud trajectory clustering to generate the original graph structure, embeds high-order representation vectors, embeds attention graph structure in the spatial domain, and performs temporal prediction, thereby achieving trajectory prediction.

[0153] Step S4, Decision Execution: Preset the decision-making mechanism and execution unit to complete vehicle blind spot prediction.

[0154] Furthermore, in some embodiments, the image sensor described in step S1 generates detailed three-dimensional point cloud data by emitting a light beam and receiving the reflected light signal, measuring the distance and angle of each point. This data not only includes the spatial location of the object but also reflects its shape and size.

[0155] Furthermore, in some embodiments, the data processing method in step S2 includes the following data preprocessing and fusion:

[0156] In step S21, data preprocessing involves uniformly downsampling the collected point cloud data (i.e., the first environmental point cloud data) and using a radius filter to remove isolated point clouds. Uniform downsampling reduces the amount of data and improves processing efficiency, while the radius filter retains key point cloud information and removes noise points.

[0157] For the radius filtering described in step S21, the radius filter is used to remove noise points and isolated points in the first environmental point cloud data, while retaining key point cloud information. First, a suitable radius value r is selected. For each point p in the first environmental point cloud data, all points in its neighborhood with a distance less than r are found. If the number of all points in the neighborhood with a distance less than r is greater than a certain threshold (in this embodiment, the minimum number of points is set to k), then the mean can be calculated using the following formula (1):

[0158]

[0159] Where n is the number of neighborhood points within radius r, p j These are the coordinates of the neighboring points. Finally, the mean value calculated using this formula is used to replace the original point p; based on this, the filtered point cloud data is obtained.

[0160] For the uniform downsampling described in step S21, uniform downsampling is performed on the filtered point cloud data to reduce the number of points in the filtered point cloud data while maintaining key spatial information. First, the voxel size δ is determined, and the point cloud space (which can be understood as the filtered point cloud data) is divided into uniform cubic voxels, with each voxel having a side length of δ. For each voxel, all points falling within that voxel are found, and its center point is calculated using the following formula (2). Then, the calculated value is used to replace the value within each voxel.

[0161]

[0162] Where N is the number of points falling within a voxel, p i These are the coordinates of these points.

[0163] Step S22: Multi-sensor data fusion. The point cloud data after uniform downsampling is fused by time synchronization and spatial alignment to ensure data consistency and comparability, thereby obtaining the environmental point cloud trajectory.

[0164] Regarding the time synchronization described in step S22, time synchronization is used to ensure that data from different sensors (which can be understood here as point cloud data after uniform downsampling) are aligned in time to provide consistent spatiotemporal information; firstly, the timestamp t of the sensor needs to be obtained. tensor Select a timestamp from a primary sensor as the base time t. refThe time difference Δt for each sensor is calculated using the following formula (3):

[0165] Δt=t ref -t tensor (3)

[0166] For each sensor's data, interpolation or delay processing is performed based on the time deviation to align it to the reference time. Specifically, if the previous time is used as the reference, the point cloud data from the next time is used for interpolation; if the next time is used as the reference, the point cloud data from the previous time is used for delay processing.

[0167] For the spatial calibration described in step S22, spatial calibration is used to unify the coordinate systems of different sensors into a single reference coordinate system; firstly, some known reference points (such as specific calibration objects or ground markers) are selected, and the spatial transformation matrix T for each sensor is calculated using the least squares method:

[0168] T = argmin∑ i ||p i measured -T(p i true )|| 2 (4)

[0169] Where, p i masured It is the measured point, p i true These are known real points. The argmin function is used to find the value of the independent variable that minimizes the objective function. It is an abbreviation for "argument of the minimum". The point cloud data of the sensor (which can be understood as the point cloud data after spatiotemporal synchronization) is spatially calibrated through the transformation matrix T; where the spatial calibration formula is as follows (5):

[0170] p corrected =T·p measured (5)

[0171] For the noise filtering described in step S22, filtering techniques (such as Kalman filtering) are applied to reduce noise and uncertainty. Point cloud data generated by various image sensors (i.e., an example of pre-processed point cloud data) are fused through time synchronization and spatial alignment to generate environmental point cloud trajectories. First, the state transition model and observation model are defined using the following formulas (6) and (7):

[0172] x k =Fx k-1 +Bu k +w k (6)

[0173] z k =Hx k +v k (7)

[0174] Where F is the state transition matrix, H is the observation matrix, and w k and v k These represent process noise and observation noise, respectively, x k Let u represent the state estimate vector of the system at time k. k Let B represent the control input vector, and let x represent the control input matrix. k-1 Let z represent the state estimate vector of the system at time k-1. k Let K represent the observation vector. Then, calculate the Kalman gain K using the following formula (8):

[0175] K = P k H T HP k H T +R) -1 (8)

[0176] The state estimate covariance is then updated using the following formulas (9) and (10):

[0177] x k|k =x k|k-1 +H(z k -Hx k|k-1 (9)

[0178] P k|k =(I-KH)P k|k-1 (10)

[0179] Where R represents the observation noise covariance matrix, P k This represents the error covariance matrix.

[0180] Figure 7 This application provides a schematic diagram illustrating the implementation process of a vehicle blind spot safety prediction method based on point cloud trajectory attention maps in complex traffic scenarios. Figure 7 As shown, the method includes the following steps 701 to 710:

[0181] Step 701: Obtain point cloud data at time 1, point cloud data at time 2, and point cloud data at time 3;

[0182] Step 702: Determine the first adjacency matrix;

[0183] Step 703, generate the representation vector;

[0184] Step 704: Input the representation vector into the attention map neural module;

[0185] Step 705: Obtain the eigenvectors at time 1, time 2, and time 3.

[0186] Step 706: Input the feature vectors at different times (i.e., the feature vectors at time 1, time 2, and time 3) into the self-attention module to predict the vector at time 4.

[0187] Step 707, obtain the prediction vector;

[0188] Step 708: Perform high-dimensional embedding inverse operation;

[0189] Step 709 yields the second adjacency matrix;

[0190] Step 710 yields the blind zone prediction result at time 4.

[0191] Furthermore, in some embodiments, the trajectory prediction process described in step S3 includes the following steps S31 to S35:

[0192] The vehicle blind spot prediction method based on point cloud trajectory attention map described in step S3 includes the following steps:

[0193] Step S31: The original graph structure is generated by using a clustering algorithm (e.g., k-means) to cluster the point cloud trajectories into clusters, establishing edges between points within a cluster, and establishing edges between different clusters based on the nearest points, thus forming an original graph structure, which is stored in the form of an adjacency matrix.

[0194] Step S32 represents vector generation. An N*128 linear matrix is ​​used to embed the first adjacency matrix into a vector, resulting in a representation vector for the first adjacency matrix. Each node in the adjacency matrix is ​​embedded as a 128-dimensional vector, where N is the number of point clouds represented by the first adjacency matrix.

[0195] Step S33, Spatial Embedding: A graph attention network is used to model the representation vectors within a time window, obtaining the positional relationships between points in the point cloud within the spatial domain, their interaction methods, and calculating the attention parameters between nodes. This is then combined with a threshold (an example of a normalized threshold) to form an attention graph structure, followed by vector aggregation based on the attention graph structure. In other words, a graph attention network is used to model the point cloud representation vectors within a time window, aggregating environmental information within the spatial domain to achieve spatial embedding. Specific solution:

[0196] Accumulate data on crowd movement in real-world scenarios, and form a dataset from the point cloud records at each moment, then split it into a training set and a test set;

[0197] Use a graph attention network to extract the attention graph structure. Figure 8 This is a schematic diagram of a graph attention network provided in an embodiment of this application. First, the attention between each node is extracted. For any connected pair of nodes, vector mapping is performed using W. The mapping results are then concatenated, and finally, an activation function is applied to obtain the attention value between the two points. In other words, based on... and Get e ij 803; where e ij The formula for calculating 803 is as follows (11):

[0198]

[0199] e ij 803 represents the attention value between nodes (i,j). represents the vector representations of nodes i and j respectively; W is a learnable parameter of a single vector mapping, LeakyReLU is the activation function, and || represents the concatenation operation.

[0200] Attention normalization is performed using the following formula (12), removing edges below a threshold from the graph structure (i.e., the graph structure corresponding to the first adjacency matrix), thus transforming the graph structure into an attention graph structure; that is, e ij 803 is input to softmax to obtain α. ij 804:

[0201]

[0202] in, Representing the neighboring nodes of node i, with a threshold of 0.5, softmax maps each element of the input vector to a value between 0 and 1 through exponential operations and normalization, and the sum of all output values ​​is 1.

[0203] Reconstruct the adjacency matrix based on the attention graph structure, and then add α values ​​that exceed the threshold. ij The adjacency matrix is ​​filled into the attention graph adjacency matrix, and the graph neural network is used to perform vector aggregation to extract environmental information; among them, the calculation formula of the node feature matrix of the (l+1)th layer is as follows (13):

[0204]

[0205] Among them, X (l) X is the feature matrix of the nodes in the l-th layer. (l+1) W is the node feature matrix of the (l+1)th layer. (b) Here, σ represents the training parameters, and σ represents the activation function (such as the linear rectified unit function). This is the adjacency matrix after attention is applied. A two-layer graph neural network model is then constructed based on this matrix to aggregate environmental information.

[0206] Furthermore, in some embodiments, step S34, temporal prediction, uses multiple self-attention layers to extract time series information, realizes point cloud prediction at the next time step, and restores the prediction result to the location point cloud using linear matrix inverse operation.

[0207] In some embodiments, aggregated point cloud vectors (an example of feature vectors) within a time window are packaged together to form a point cloud sequence (an example of a feature vector sequence). The point cloud vectors in the aggregated feature vector sequence are then processed through a multi-layer self-attention layer to extract time series information. The calculation process is as follows:

[0208] Input the aggregated point cloud vector within a time window Where n is the sequence length and d is the feature dimension (taken as 128); the input X is transformed by three different linear transformations, namely by formula (14), to generate query, key, and value matrices:

[0209] Q = XW Q K = XW K V = XW V (14)

[0210] Among them, W Q W K W V These are all learnable weight parameters, and their function is to perform a linear transformation on the original moments.

[0211] Next, the similarity between the query and the key is calculated using the dot product, resulting in the attention score matrix, AttentionScores.

[0212] Attention Scores = QK T (15)

[0213] Scale these scores to avoid the vanishing gradient problem when the dot product result is large:

[0214]

[0215] Applying the Softmax function to the scaled score matrix yields the attention weight matrix:

[0216]

[0217] The value matrix is ​​weighted and summed using the attention weight matrix to obtain the final self-attention output:

[0218]

[0219] The result is The matrix represents the sequence features after the self-attention mechanism, and the last row vector is taken as the vector prediction for the next time step of the node.

[0220] Calculate the loss function for the output matrix:

[0221]

[0222] in, Let j be an indicator function, indicating that j ≠ i 1 If the condition is met, then take 1; otherwise, take 0. , respectively, are the predicted i-node representation vector and the actual i-node representation vector; M is the total number of monitored point cloud nodes, and τ is the temperature parameter, set to 0.05.

[0223] The similarity function sim is calculated using the following formula (20):

[0224]

[0225] Then the loss of this function over the entire training set, i.e. the average loss for each data point, is calculated using the following formula (21):

[0226]

[0227] The gradient descent algorithm is used to minimize the loss function, maximizing the similarity between the predicted and true representation vectors of the same node. Representation vectors between different nodes are spaced as far apart as possible to update the parameters in the graph attention network, graph neural network, and self-attention network.

[0228] After training, a prediction model combining spatial and temporal information is obtained. This model is used to generate the node representation vector of the point cloud at the next time step. The inverse operation of the linear matrix mentioned above is used to restore the point cloud matrix to point cloud coordinates. Trajectory prediction for the next time step is then completed.

[0229] Step S35: The point cloud of the location predicted in step S34 is compared with the point cloud obtained by the image sensor (e.g., the sensor in the radar unit and the vision unit) at the current time within a certain similarity range to obtain the point cloud of obstacles in the vehicle's blind spot.

[0230] Furthermore, the obstacle point cloud obtained in step S35 can be used to identify specific obstacles through a deep learning recognition algorithm, and provide data preparation for downstream tasks.

[0231] Furthermore, in some embodiments, the decision execution in step S4 refers to the output of the prediction result to the alarm unit when a potentially dangerous object / person appears in the vehicle's blind spot, and the alarm unit decides whether to execute a hazard warning based on the prediction result.

[0232] In some embodiments, this application provides a vehicle blind spot safety prediction system based on point cloud trajectory attention maps in complex traffic scenarios. Figure 9 A schematic diagram of a vehicle blind spot safety prediction system based on point cloud trajectory attention map in complex traffic scenarios provided in this application embodiment is shown below. Figure 9 As shown, the functional module 901 of the vehicle blind spot prediction system includes:

[0233] The high-risk scene recognition module 11 uses an image sensor and scene recognition algorithm to identify the current vehicle driving scene and classify the risk level;

[0234] If the high-risk scene recognition module 11 classifies the current vehicle driving scene as high risk level, the blind spot safety prediction module 12 is triggered to identify whether a potentially risky person / object will appear in the blind spot area at the next moment.

[0235] If the blind spot safety prediction module 12 identifies a potentially dangerous person or object within the blind spot area, the alarm module 13 will be triggered. The alarm module 13 will then decide whether to issue a hazard warning based on the blind spot prediction result.

[0236] Furthermore, in order to realize the functions of each module of the above-mentioned vehicle blind spot safety prediction system, the system should include the following hardware unit 902:

[0237] The image unit 21 includes: a millimeter-wave radar 211 for omnidirectional detection of objects at medium and long distances around the vehicle, providing high-resolution environmental information; an ultrasonic radar 212 for detecting nearby obstacles, supplementing the blind spots of the millimeter-wave radar; and a lidar 213 for providing high-precision 3D environmental modeling and capturing fine features of surrounding objects. The millimeter-wave radar is installed near the front and rear bumpers and side mirrors of the vehicle, the ultrasonic radar is installed around the vehicle, and the lidar is installed on the roof or in front of the vehicle.

[0238] Algorithm unit 22 includes: data preprocessing 211, which receives data from millimeter-wave radar, ultrasonic radar, and lidar, and preprocesses the collected point cloud data to remove noise; data fusion 222, which performs multi-sensor data fusion to generate comprehensive environmental perception information; algorithm execution 223, which runs the corresponding algorithm according to controller instructions; and a decision module, which includes three tasks: determining the risk level of the scenario; determining whether a potentially dangerous person / object will appear in the next moment within the blind zone; and determining whether to issue an alarm.

[0239] The alarm unit 23 includes: a horn 231, a display screen 232, a seat 233, and a driver assistance system 234. The control unit combines blind spot prediction results and high-risk scene recognition results to issue warnings to the driver via the horn 231, display screen 232, seat 233, and driver assistance system 234. When necessary, it coordinates with the vehicle's autonomous driving system through the vehicle bus interface to assist the driver in avoiding danger. In some embodiments, the display screen 232 includes a central control display screen.

[0240] Furthermore, the alarm unit 23 includes the following two modules:

[0241] Decision module: Combining blind spot prediction results and high-risk scenario identification results, decides whether to issue an alert;

[0242] Execution module: Alarm unit 23, including horn 231, display screen 232, seat 233, driver assistance 234, HUD, audio, central control display screen and risk warning light. Based on the decision results of the decision module, it issues corresponding alarm prompts to assist the driver in avoiding danger.

[0243] The control unit 903 includes: a processor 31 for executing the computational tasks of the algorithm unit; a memory 32 for storing sensor data, algorithm models, and historical trajectory data; an in-vehicle communication unit 33 with a vehicle bus interface for communicating with the vehicle's control system to obtain information such as the vehicle's speed and direction, thus assisting the prediction algorithm; a wireless communication unit 34 for communicating with external devices, cloud servers, and other vehicles, supporting vehicle-to-everything (V2X) functionality and remote system updates; and a decision and control module that uses reinforcement learning and fuzzy logic algorithms to generate driving decisions and coordinates with the vehicle's autonomous driving system through the vehicle bus interface to execute necessary active protection measures, such as automatic braking or steering correction.

[0244] It is understood that this application summarizes a method and system for predicting vehicle blind spots in complex traffic scenarios based on point cloud trajectory attention maps. This system, based on point cloud trajectory attention map prediction technology, aims to effectively address high-risk traffic environments by accurately predicting potential hazards within vehicle blind spots, thereby achieving preventative management of traffic accidents. It triggers an early warning mechanism before the danger manifests, significantly enhancing the vehicle system's active safety protection capabilities and effectively reducing the incidence of traffic accidents. The blind spot prediction method provided in this application constructs an attention map structure in the point cloud, introducing spatial information into the prediction process. Given the significant differences between pedestrian flow and the movement of vehicles and static objects, this method employs a selective aggregation strategy, allowing point cloud data to intelligently integrate relevant information from neighboring nodes, effectively avoiding the information oversmoothing problem that may occur with traditional methods. This greatly broadens the model's perception and understanding of the surrounding environment, making the prediction results more accurate and comprehensive.

[0245] This application provides a blind spot prediction method (i.e., an example of an early warning method). In this method, an attention map structure is constructed in the point cloud, that is, spatial information is introduced in the prediction process. Since the movement of people is significantly different from the movement of vehicles and objects, the point cloud selectively aggregates the node information of the neighborhood to avoid the problem of overly smooth prediction results that may be caused by homogeneous schemes, so that the model can aggregate environmental information over a wider range.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application, and all such modifications or substitutions should be covered within the protection scope of this application.

[0247] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution. Based on the foregoing embodiments, this application provides an apparatus comprising the included modules and the units included in each module, which can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in implementation, the processor can be an AI acceleration engine (such as an NPU), a graphics processing unit (GPU), a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0248] Figure 10 A schematic diagram of an early warning device provided in an embodiment of this application is shown below. Figure 10 As shown, the early warning device 100 includes: an acquisition module 1001, a first determination module 1002, a second determination module 1003, a prediction module 1004, and a triggering module 1005; wherein,

[0249] Module 1001 is configured to acquire the first environmental point cloud data of the vehicle within the current time window.

[0250] The first determining module 1002 is configured to determine the first adjacency matrix of the first environmental point cloud data;

[0251] The second determining module 1003 is configured to determine the attention value between any two connected nodes in the first adjacency matrix; wherein the attention value is used to characterize the positional information and semantic association strength between any two connected nodes.

[0252] Prediction module 1004 is configured to predict the second environmental point cloud data of the vehicle at the next moment based on the attention value;

[0253] The trigger module 1005 is configured to output an alarm signal when it is determined, based on the first environmental point cloud data and the second environmental point cloud data, that there is an obstacle in the blind spot of the vehicle.

[0254] In some embodiments, the second determining module 1003 is configured to determine a set of representation vectors of the first adjacency matrix; wherein the set of representation vectors includes: the representation vector of any node in the first adjacency matrix; and based on the representation vectors of any two connected nodes, to determine an attention value between the two connected nodes.

[0255] In some embodiments, the second determining module 1003 is configured to perform a dimensionality increase operation on the i-th node in the first adjacency matrix to obtain the representation vector of the i-th node; wherein i is greater than 0 and less than or equal to the number of nodes in the first adjacency matrix; and based on the representation vectors of the nodes in the first adjacency matrix, obtain the set of representation vectors.

[0256] In some embodiments, the prediction module 1004 is configured to perform a first process on the attention value between the i-th node and the j-th neighbor node of the i-th node in the first adjacency matrix; wherein the first process includes: normalizing the attention value between the i-th node and the j-th neighbor node of the i-th node to obtain a normalized value; wherein j is greater than 0 and less than or equal to the number of neighbor nodes of the i-th node; if the normalized value is less than a normalization threshold, deleting the connection between the i-th node and the j-th neighbor node of the i-th node; iteratively executing the first process until all nodes in the first adjacency matrix have performed the first process to obtain an attention graph structure; and predicting the second environmental point cloud data of the vehicle at the next time step based on the attention graph structure.

[0257] In some embodiments, the prediction module 1004 is configured to construct a second adjacency matrix of the attention graph structure; fill the second adjacency matrix with a normalized value greater than or equal to the normalization threshold to obtain an attention graph adjacency matrix; and predict the second environmental point cloud data of the vehicle at the next time step based on the attention graph adjacency matrix.

[0258] In some embodiments, the prediction module 1004 is configured to extract environmental features from the adjacency matrix of the attention map to obtain feature vectors of environmental point cloud data at different times within the current time window; aggregate the feature vectors at different times to obtain a feature vector sequence; and input the feature vector sequence into a trained multilayer self-attention network to predict the second environmental point cloud data to obtain the second environmental point cloud data.

[0259] In some embodiments, the trained multilayer self-attention network is trained based on a contrastive loss function of sinusoidal similarity.

[0260] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0261] It should be noted that the module division in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or have two or more units integrated into one unit. The integrated units can be implemented in hardware, as software functional units, or a combination of software and hardware.

[0262] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause the vehicle to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0263] This application provides a vehicle, which includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps of the method provided in the above embodiments.

[0264] It should be noted that the memory is configured to store processor-executable instructions and applications, and can also cache data to be processed or already processed by the processor and various modules in the vehicle. This can be achieved through flash memory or random access memory (RAM).

[0265] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0266] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0267] It should be noted that the descriptions of the storage medium and vehicle embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and vehicle embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0268] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0269] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0270] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0271] In the several embodiments provided in this application, it should be understood that the disclosed vehicles and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between vehicles or modules can be electrical, mechanical, or other forms.

[0272] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0273] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0274] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0275] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause the vehicle to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0276] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined to obtain new product embodiments without conflict. The features disclosed in the several method or vehicle embodiments provided in this application can be arbitrarily combined to obtain new method embodiments or vehicle embodiments without conflict.

[0277] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An early warning method, characterized in that, The method includes: Acquire the first environmental point cloud data of the vehicle within the current time window; Determine the first adjacency matrix of the first environmental point cloud data; Determine the attention value between any two connected nodes in the first adjacency matrix; wherein the attention value is used to characterize the positional information and semantic association strength between any two connected nodes; Based on the attention value, predict the second environmental point cloud data of the vehicle at the next moment; If an obstacle is determined to exist in the blind spot of the vehicle based on the first environmental point cloud data and the second environmental point cloud data, an alarm signal is output.

2. The early warning method according to claim 1, characterized in that, Determining the attention value between any two connected nodes in the first adjacency matrix includes: Determine the representation vector set of the first adjacency matrix; wherein, the representation vector set includes: the representation vector of any node in the first adjacency matrix; Based on the representation vectors of any two connected nodes, determine the attention value between those two connected nodes.

3. The early warning method according to claim 1 or 2, characterized in that, The step of determining the set of representation vectors for the first adjacency matrix includes: Perform a dimension-up operation on the i-th node in the first adjacency matrix to obtain the representation vector of the i-th node; where i is greater than 0 and less than or equal to the number of nodes in the first adjacency matrix; The set of representation vectors is obtained based on the representation vectors of the nodes in the first adjacency matrix.

4. The early warning method according to claim 2, characterized in that, The step of predicting the second environmental point cloud data of the vehicle at the next moment based on the attention value includes: Perform a first process on the attention value between the i-th node and the j-th neighbor node of the i-th node in the first adjacency matrix; wherein the first process includes: normalizing the attention value between the i-th node and the j-th neighbor node of the i-th node to obtain a normalized value; wherein j is greater than 0 and less than or equal to the number of neighbor nodes of the i-th node; if the normalized value is less than a normalized threshold, delete the connection between the i-th node and the j-th neighbor node of the i-th node. The first process is executed iteratively until all nodes in the first adjacency matrix have completed the first process, resulting in an attention graph structure. Based on the attention map structure, predict the second environmental point cloud data of the vehicle at the next moment.

5. The early warning method according to claim 4, characterized in that, The prediction of the second environmental point cloud data of the vehicle at the next moment based on the attention map structure includes: Construct the second adjacency matrix of the attention graph structure; The normalized value, which is greater than or equal to the normalization threshold, is filled into the second adjacency matrix to obtain the attention graph adjacency matrix; Based on the attention graph adjacency matrix, predict the second environmental point cloud data of the vehicle at the next time step.

6. The early warning method according to claim 5, characterized in that, The step of predicting the second environmental point cloud data of the vehicle at the next time step based on the attention map adjacency matrix includes: Environmental features are extracted from the adjacency matrix of the attention map to obtain feature vectors of environmental point cloud data at different times within the current time window; The feature vectors at different times are aggregated to obtain a feature vector sequence; The feature vector sequence is input into a trained multi-layer self-attention network to predict the second environment point cloud data, thereby obtaining the second environment point cloud data.

7. The early warning method according to claim 6, characterized in that, The trained multi-layer self-attention network is trained based on a contrastive loss function with sinusoidal similarity.

8. An early warning device, characterized in that, The device includes: The acquisition module is configured to acquire the first environmental point cloud data of the vehicle within the current time window; The first determining module is configured to determine the first adjacency matrix of the first environmental point cloud data; The second determining module is configured to determine the attention value between any two connected nodes in the first adjacency matrix; wherein the attention value is used to characterize the positional information and semantic association strength between any two connected nodes. The prediction module is configured to predict the second environmental point cloud data of the vehicle at the next moment based on the attention value; The triggering module is configured to output an alarm signal when it is determined, based on the first environmental point cloud data and the second environmental point cloud data, that there is an obstacle in the blind spot of the vehicle.

9. A vehicle comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method described in any one of claims 1 to 7.

Images