A remote takeover risk early warning system and method based on a behavior prediction model

Abstract

The application discloses a remote takeover risk early warning system and method based on a behavior prediction model, and belongs to the technical field of intelligent driving safety. The system comprises multi-source data sensing, behavior and interaction prediction, risk quantification, early warning decision, and first of all, relevant data is collected and preprocessed through a multi-source data sensing module; then a pre-trained model is used by a behavior and interaction prediction module to predict the future trajectory of a vehicle and the operation behavior of a driver, time-space intersection point information of the vehicle and surrounding objects is generated through a vector space analysis engine; subsequently, a risk quantification module calculates collision time, operation deviation degree and avoidability indexes based on the prediction information, and obtains a takeover risk probability value; finally, an early warning decision module triggers a graded early warning according to the risk probability value, and outputs a takeover strategy suggestion. The application realizes accurate, dynamic and operable early warning of remote takeover risk, and significantly improves the safety and reliability of the intelligent driving remote takeover scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving safety, in particular to a remote takeover risk early warning system and method based on a behavior prediction model. BACKGROUND

[0002] In the process of rapid development of intelligent driving technology, remote takeover as a key link to cope with the limit scene of automatic driving system, its safety and reliability directly determine the landing application boundary of intelligent driving. However, the existing remote takeover risk early warning technology always faces two core bottlenecks, which seriously restricts the early warning accuracy and decision support ability. On the one hand, the existing system is difficult to realize efficient fusion and deep utilization of multi-source heterogeneous data. For the structured data scattered in vehicles, environment, drivers and other different dimensions (such as vehicle dynamics parameters, sensor perception data), the traditional centralized processing mode cannot realize collaborative modeling under the premise of guaranteeing data privacy and security, and lacks standardized quantitative extraction means for massive unstructured data (such as driver physiological state, environmental semantic information), resulting in single dimension of model input data, incomplete information, and difficulty in fully describing the real state of complex driving scene. On the other hand, the existing deduction model lacks the linkage mechanism of micro behavior prediction and macro risk analysis, either relying only on macro statistical model for risk prediction, which cannot accurately capture the dynamic interaction relationship between vehicle trajectory and driver operation, or lacking data-driven spatio-temporal risk quantification method, which is difficult to determine the time node, spatial position and avoidance possibility of risk occurrence, resulting in that the early warning opportunity is either too early to cause resource waste or too late to lose the avoidance window, and the early warning result lacks explainability, which cannot provide accurate operation guidance for remote drivers. These problems make the existing remote takeover risk early warning system difficult to meet the safety needs in complex driving scenarios, and an integrated technical solution is needed to realize multi-source data fusion, high-precision behavior prediction, accurate risk quantification and scientific early warning decision. SUMMARY

[0003] In view of the defects in the prior art, the purpose of the present application is to provide a remote takeover risk early warning system and method based on a behavior prediction model, which realizes accurate, dynamic and operable early warning of remote takeover risk, and significantly improves the safety and reliability of intelligent driving remote takeover scene.

[0004] In order to achieve the above purpose, the application provides the following technical scheme:

[0005] The embodiment of the application provides a remote takeover risk early warning system based on a behavior prediction model, which comprises a multi-source data sensing module, the multi-source data sensing module is used for collecting and preprocessing data from a controlled vehicle, a driving environment and a driving station in real time; a behavior and interaction prediction module, the behavior and interaction prediction module comprises a behavior prediction model based on a double-flow space-time graph neural network and a vector space analysis engine, the behavior and interaction prediction module is used for receiving a processing result of the multi-source data sensing module, and synchronously outputting a vehicle future trajectory prediction sequence, a driver operation behavior prediction sequence and space-time intersection point prediction information between the vehicle and surrounding objects; a risk quantifier module, the risk quantifier module is connected with the behavior and interaction prediction module, the risk quantifier module is used for generating a dynamic takeover risk probability value based on the vehicle future trajectory prediction sequence, the driver operation behavior prediction sequence and the space-time intersection point prediction information, and evaluating a possibility of avoiding a predicted risk through driving operation according to the takeover risk probability value, and generating a risk avoidability evaluation result; and an early warning decision and execution module, the early warning decision and execution module is connected with the risk quantifier module, the early warning decision and execution module is used for triggering a hierarchical early warning signal and generating a takeover strategy suggestion containing specific avoidance operation according to the takeover risk probability value and the risk avoidability evaluation result.

[0006] Further, the multi-source data sensing module comprises a vehicle state sensing unit, the vehicle state sensing unit is used for acquiring vehicle CAN bus data, including but not limited to vehicle speed, acceleration, yaw rate, steering light state; an environment sensing unit, the environment sensing unit is used for acquiring vehicle-mounted sensor data, the vehicle-mounted sensor data including but not limited to laser radar, millimeter wave radar, camera data, the vehicle-mounted sensor data being used for sensing surrounding obstacles, lane lines, traffic signal light states; and a driver sensing unit, the driver sensing unit is used for acquiring driving station data, the driving station data including but not limited to steering wheel angle, accelerator / brake pedal opening, steering frequency and eye tracker and / or physiological sensor data.

[0007] Further, the vector space analysis engine comprises: constructing a dynamic vector space with the vehicle itself as a coordinate origin; constructing a reference vector based on a future predicted motion trajectory of the vehicle itself; converting a predicted motion trajectory of a surrounding moving object of the vehicle into a relative motion vector relative to the vehicle itself; judging whether the vehicle itself and the surrounding moving object of the vehicle will appear in a space-time intersection point within a future preset time based on the reference vector and all relative motion vectors, and obtaining a time and information of the space-time intersection point appearing; and outputting the time and information of the space-time intersection point appearing to the risk quantifier module, which is used for evaluating a possibility of avoiding the space-time intersection point through changing a speed, an acceleration or a direction of the vehicle itself.

[0008] Further, the risk quantifier module comprises: calculating a collision time based on the predicted information of the space-time intersection point; calculating a deviation degree between a safe operation required to avoid the space-time intersection point based on the driver operation behavior prediction sequence; simulating at least one avoidance operation based on the deviation degree and obtaining a change amount of the collision time after the avoidance operation is performed; and fusing the collision time, the deviation degree and the avoidance operation to output the dynamic takeover risk probability value.

[0009] Further, the early warning decision and execution module comprises: a hierarchical early warning unit, which is configured to preset a plurality of risk thresholds, including a first threshold, a second threshold and a third threshold, and the third threshold is greater than the second threshold which is greater than the first threshold; when the takeover risk probability value exceeds the first threshold and is lower than the second threshold, triggering a focus level warning; when the takeover risk probability value exceeds the second threshold and is lower than the third threshold, triggering a preparation level warning; and when the takeover risk probability value exceeds the third threshold, triggering an immediate takeover level warning; and a strategy suggestion unit, which is configured to generate a takeover strategy suggestion including an optimal takeover time and specific avoidance operation parameters based on the risk avoidability evaluation result when the preparation level warning and the immediate takeover level warning are triggered.

[0010] Further, the optimal takeover time is obtained by simulating the system risk change at different delayed takeover time points and selecting a future time point with the minimum risk accumulation as the recommended takeover timing.

[0011] Correspondingly, the embodiment of the application also provides a remote takeover risk early warning method based on a behavior prediction model, which comprises the following steps: S1, collecting and fusing the data of the vehicle, the driving environment and the driving station in real time and generating fused data; S2, inputting the fused data into a behavior prediction model of a pre-trained double-flow space-time graph neural network and a vector space analysis engine to synchronously predict a vehicle future trajectory sequence, a driver future operation sequence and space-time intersection point information of the vehicle and surrounding moving objects, and outputting a prediction result; S3, quantitatively calculating a dynamic takeover risk probability value at the current time based on the prediction result and evaluating the possibility of avoiding the predicted space-time intersection point by operation; and S4, performing hierarchical early warning and outputting a takeover strategy suggestion containing specific avoidance operation based on the dynamic takeover risk probability value and the risk avoidability evaluation result.

[0012] Further, the training and construction method of the behavior prediction model of the dual-flow spatio-temporal graph neural network comprises: S21, constructing a time series dataset containing historical vehicle trajectories, environmental context, driver operation sequences, and corresponding takeover event labels; S22, a vehicle trajectory prediction flow, based on the spatio-temporal graph constructed from historical vehicle states and environmental information, capturing spatial topology and temporal dependence through a spatio-temporal graph convolution network, and outputting a future trajectory sequence of the vehicle; a driver behavior prediction flow, based on the historical driver operation sequence, extracting local features through a time series convolution network, and then capturing long-range dependence through a self-attention mechanism, and outputting a future operation sequence of the driver; S23, based on a joint loss function that minimizes the vehicle trajectory prediction error and the driver behavior prediction error, training the network parameters end-to-end, wherein the expression of the joint loss function is: ; represents the loss of the vehicle trajectory prediction flow, which is calculated by mean square error, and the specific expression is the sum of squares of the Euclidean distances between the predicted trajectory points and the real trajectory points, represents the loss of the driver behavior prediction flow, which is calculated by cross-entropy loss, and the specific expression is the difference between the predicted Caozhi instruction distribution and the real operation quality distribution, represents the contrast learning loss, which is calculated by the interval loss based on triplets, and is used to pull the hidden layer feature vectors of the vehicle trajectory flow and the driver behavior flow at the same time step closer, while pushing the feature vectors at different time steps or in different scenes farther apart, so as to promote the consistency of the understanding of the same scene by the two prediction flows, , , are adjustable parameters of the vehicle trajectory prediction flow loss, the driver behavior prediction flow loss, and the contrast learning loss, respectively.

[0013] Further, the process of quantitatively calculating the dynamic takeover risk probability value at the current time comprises: S31, based on the predicted spatio-temporal intersection information, obtaining the most critical collision time within a preset time, wherein the is defined as the time required for the vehicle to collide with the nearest obstacle; S32, based on the predicted driver operation, calculating the deviation degree from the ideal safe operation model, wherein the is quantified by calculating the cosine similarity or Euclidean distance in the vector space between the predicted operation sequence and the ideal operation sequence; S33, simulating a set of preset evasion operation sets, calculating the improvement degree after executing each evasion operation, to obtain an avoidability index , wherein the is defined as the ratio of the increment after executing the evasion operation to the original ; S34, based on the , and , a dynamic risk function is constructed, and the expression is: ; wherein and are adjustable weight coefficients of are adjustable weight coefficients of are adjustable weight coefficients of indicates a final calculated dynamic takeover risk probability value, and is normalized to the interval by a Sigmoid function.

[0014] Further, the takeover strategy suggestion includes: a suggested deceleration value, a steering angle change amount, and a suggested execution time point of an avoidance operation.

[0015] The beneficial effects of the present application: through the technical fusion of multi-source data fusion, double-flow space-time graph neural network prediction, vector space risk analysis and hierarchical early warning decision, the problems of insufficient data support, low behavior prediction accuracy, inaccurate risk quantification, and non-specific strategy suggestion in the existing remote takeover risk early warning are solved, and the precise, dynamic and operable early warning of the remote takeover risk is realized, which greatly improves the safety and reliability of the intelligent driving remote takeover scene. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a structural schematic diagram of a remote takeover risk early warning system based on a behavior prediction model provided by the embodiments of the present application;

[0017] Figure 2 is a flow schematic diagram of a remote takeover risk early warning method based on a behavior prediction model provided by the embodiments of the present application. DETAILED DESCRIPTION

[0018] The specific embodiments of the embodiments of the present application are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application.

[0019] The terms "system" and "network" in the embodiments of the present application can be used interchangeably. "Multiple" means two or more, and therefore, "multiple" can also be understood as "at least two" in the embodiments of the present application. "And / or", which describes the association relationship of associated objects, means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / ", if not specially stated, generally represents that the associated objects before and after it are in an "or" relationship. In addition, it should be understood that in the description of the embodiments of the present application, "first", "second", and the like are used only for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance, nor indicating or implying sequence.

[0020] Embodiment 1

[0021] The existing takeover risk early warning system is difficult to fully integrate vehicle, environment, and driver multi-source data, and lacks accurate analysis of the spatio-temporal interaction between the vehicle and the surrounding objects, resulting in low risk prediction accuracy and unreasonable early warning timing.

[0022] As shown in Figure 1 The embodiments of the present application provide a remote takeover risk early warning system based on a behavior prediction model, which comprises: a multi-source data perception module, the multi-source data perception module is used for real-time collection and preprocessing of data from a controlled vehicle, a driving environment and a driving station; a behavior and interaction prediction module, the behavior and interaction prediction module comprises a behavior prediction model based on a double-flow spatio-temporal graph neural network and a vector space analysis engine, the behavior and interaction prediction module is used for receiving the processing result of the multi-source data perception module, and synchronously outputting a vehicle future trajectory prediction sequence, a driver operation behavior prediction sequence and spatio-temporal intersection point prediction information between the vehicle and the surrounding objects; a risk quantifier module, the risk quantifier module is connected with the behavior and interaction prediction module, the risk quantifier module is used for generating a dynamic takeover risk probability value based on the vehicle future trajectory prediction sequence, the driver operation behavior prediction sequence and the spatio-temporal intersection point prediction information, and generating a risk avoidability evaluation result according to the possibility of avoiding the predicted risk through driving operation based on the takeover risk probability value; an early warning decision and execution module, the early warning decision and execution module is connected with the risk quantifier module, the early warning decision and execution module is used for triggering a hierarchical early warning signal and generating a takeover strategy suggestion containing specific avoidance operation according to the takeover risk probability value and the risk avoidability evaluation result.

[0023] In another possible embodiment, the multi-source data perception module collects CAN bus data of the controlled vehicle, vehicle-mounted sensor data of the driving environment, and operation and physiological data of the driving station in real time, and performs preprocessing such as cleaning and standardization on the data; the behavior and interaction prediction module receives the preprocessed data, inputs the data into a pre-trained behavior prediction model of a double-flow space-time graph neural network to predict a future trajectory sequence of the vehicle and a driver operation behavior prediction sequence, and simultaneously inputs the data into a vector space analysis engine to generate space-time intersection point prediction information between the vehicle and surrounding objects; the risk quantifier module calculates a collision time, an operation deviation degree, and a collision time change amount after trajectory operation based on the future trajectory of the vehicle, the driver operation behavior, and the space-time intersection point prediction information, substitutes the values into a dynamic risk function to generate a dynamic takeover risk probability value, and evaluates risk avoidability; and the early warning decision and execution module triggers a hierarchical early warning signal according to the takeover risk probability value, and generates a takeover strategy suggestion including specific avoidance operations in combination with the risk avoidability evaluation result.

[0024] By integrating the four modules of multi-source data perception, behavior and interaction prediction, risk quantification, and early warning decision, high-precision behavior prediction driven by multi-source data fusion, accurate space-time risk analysis supported by vector space, dynamic quantification of takeover risk assessment, and hierarchical and accurate early warning decision are achieved, which greatly improves the timeliness, accuracy, and operability of remote takeover risk early warning.

[0025] Existing multi-source data acquisition modules have single functions and cannot comprehensively cover vehicle state, environmental information, and driver behavior data, resulting in a lack of complete data support for subsequent risk prediction.

[0026] In the embodiments of the present application, the multi-source data perception module comprises: a vehicle state perception unit, configured to acquire vehicle CAN bus data, including but not limited to vehicle speed, acceleration, yaw rate, and steering light state; an environment perception unit, configured to acquire vehicle-mounted sensor data, including but not limited to laser radar, millimeter wave radar, and camera data, the vehicle-mounted sensor data being used to perceive surrounding obstacles, lane lines, and traffic signal light states; and a driver perception unit, configured to acquire driving station data, including but not limited to steering wheel angle, accelerator / brake pedal opening, steering frequency, and eye tracker and / or physiological sensor data.

[0027] In another possible embodiment, the vehicle state perception unit continuously collects vehicle dynamics data and state data such as vehicle speed, acceleration, yaw rate, steering light state, etc. through the vehicle CAN bus interface; the environment perception unit collects environmental information such as the position, speed, shape of surrounding obstacles, the orientation of lane lines, and the state of traffic signal lights in real time through vehicle-mounted sensors such as laser radar, millimeter wave radar, camera, etc.; and the driver perception unit continuously collects steering wheel angle, accelerator / brake pedal opening, steering frequency, driver's line of sight direction, and physiological state data through a steering wheel angle sensor, an accelerator / brake pedal opening sensor, an eye tracker, and a physiological sensor (such as a heart rate sensor, etc.).

[0028] By subdividing the vehicle state perception, environment perception, and driver perception units, vehicle CAN bus data, vehicle-mounted sensor data, and driver station data are collected respectively, multi-dimensional and full-scenario data coverage is achieved, complete and accurate input data are provided for subsequent behavior prediction and risk analysis, and the comprehensiveness and timeliness of the data are ensured.

[0029] The prior art lacks a quantitative analysis method for the spatiotemporal interaction relationship between a vehicle and surrounding objects, cannot accurately determine spatiotemporal intersection points and risk avoidance possibilities, and thus leads to a lack of scientific basis for remote takeover decisions.

[0030] In the embodiment of the present application, the vector space analysis engine includes: constructing a dynamic vector space with the vehicle itself as the coordinate origin; constructing a reference vector based on the future predicted motion trajectory of the vehicle itself; converting the predicted motion trajectory of the moving objects around the vehicle into a relative motion vector relative to the vehicle itself; based on the reference vector and all relative motion vectors, determining whether the vehicle itself and the moving objects around the vehicle will have a spatiotemporal intersection point within a future preset time, and obtaining the time and information of the spatiotemporal intersection point; and outputting the time and information of the spatiotemporal intersection point to the risk quantifier module for evaluating the possibility of avoiding the spatiotemporal intersection point by changing the speed, acceleration, or direction of the vehicle itself.

[0031] In another possible embodiment, a dynamic vector space is constructed with the current position of the vehicle as the coordinate origin, which is updated in real time as the vehicle moves; a reference vector is constructed based on the vehicle future trajectory sequence output by the dual-flow spatio-temporal graph neural network; the future trajectory sequence of surrounding moving objects output by the environment perception data and the trajectory prediction model is converted into relative motion vectors relative to the vehicle itself; through vector dot product, cross product and other operations, it is determined whether there is a spatio-temporal intersection point between the vehicle itself and the surrounding moving objects within a preset time (such as 5 seconds) in the future, and if there is, the time and spatial coordinates of the spatio-temporal intersection point are accurately calculated, and then the spatio-temporal intersection point information is output to the risk quantifier module for evaluating the possibility of avoiding the intersection point by adjusting the vehicle speed, acceleration or direction.

[0032] By constructing a dynamic vector space with the vehicle itself as the origin, the motion trajectories of the vehicle and surrounding objects are converted into relative motion vectors, the time, position and avoidance possibility of the spatio-temporal intersection point are accurately determined, and core spatio-temporal analysis support is provided for risk quantification and early warning decision-making, enabling the risk judgment to move from qualitative to quantitative.

[0033] The existing risk quantification method lacks comprehensive consideration of collision time, driver operation deviation and avoidance operation effect, resulting in inaccurate calculation of takeover risk probability value and inability to truly reflect the risk level.

[0034] In an embodiment of the present application, the risk quantifier module includes: calculating the collision time based on the prediction information of the spatio-temporal intersection point; calculating the deviation between the driver operation behavior prediction sequence and the ideal safe operation sequence required to avoid the spatio-temporal intersection point based on the driver operation behavior prediction sequence; simulating at least one avoidance operation based on the deviation and obtaining the change in the collision time after the avoidance operation is performed; and fusing the collision time, the deviation and the avoidance operation to output the dynamic takeover risk probability value.

[0035] In another possible embodiment, the collision time is calculated based on the relative distance and relative speed between the obstacle and the vehicle based on the prediction information of the spatio-temporal intersection point ; the deviation between the driver operation behavior prediction sequence and the ideal safe operation sequence required to avoid the spatio-temporal intersection point is calculated based on the driver operation behavior prediction sequence , which can be calculated using vector cosine similarity; at least one avoidance operation (such as deceleration, lane change, turning, etc.) is simulated based on the deviation, and the change after each avoidance operation is calculated ; the collision time , the deviation and the change after the avoidance operation are substituted into the dynamic risk function (wherein To avoid the change in collision time after the operation, the avoidability index corresponding to the change in collision time after the operation is calculated to obtain a dynamic takeover risk probability value.

[0036] The collision time, the driver operation deviation degree, and the change in collision time after the operation are comprehensively considered to construct a dynamic risk function, thereby realizing accurate quantification of the takeover risk probability, providing a reliable numerical basis for the hierarchical early warning, and enabling the risk assessment to develop from a single dimension to multidimensional fusion.

[0037] If the early warning decision model adopts a single threshold early warning, the risk level cannot be classified for early warning, and there is a lack of specific takeover strategy suggestions, resulting in a lack of pertinence in remote takeover operation.

[0038] In the embodiments of the present application, the early warning decision and execution module comprises a hierarchical early warning unit, which is configured to preset a plurality of risk thresholds, including a first threshold, a second threshold, and a third threshold, and the third threshold is greater than the second threshold, which is greater than the first threshold. When the takeover risk probability value exceeds the first threshold and is lower than the second threshold, a concern level early warning is triggered. When the takeover risk probability value exceeds the second threshold and is lower than the third threshold, a preparation level early warning is triggered. When the takeover risk probability value exceeds the third threshold, an immediate takeover level early warning is triggered. A strategy suggestion unit is configured to generate a takeover strategy suggestion including an optimal takeover time and specific evasive operation parameters based on the risk avoidability evaluation result when the preparation level early warning and the immediate takeover level early warning are triggered.

[0039] In another possible embodiment, the hierarchical early warning unit presets a first threshold, a second threshold, and a third threshold, and the third threshold is greater than the second threshold, which is greater than the first threshold. When the takeover risk probability value exceeds the first threshold and is lower than the second threshold, a concern level threshold (such as displaying a yellow prompt box on the remote driving interface) is triggered. When the takeover risk probability value exceeds the second threshold and is lower than the third threshold, a preparation level early warning (such as displaying an orange prompt box on the remote driving interface accompanied by a prompt sound) is triggered. When the third threshold is exceeded, an immediate takeover level early warning (such as displaying a red prompt box on the remote driving interface and emitting a sharp prompt sound) is triggered. The strategy suggestion unit generates a takeover strategy suggestion including an optimal takeover time and specific evasive operation parameters (such as deceleration value and steering angle change) based on the risk avoidability evaluation result when the preparation level early warning and the immediate takeover level early warning are triggered.

[0040] The hierarchical early warning unit and the strategy suggestion unit are designed to realize hierarchical early warning of the concern level, the preparation level, and the immediate takeover level, and to generate a takeover strategy suggestion including an optimal takeover time and specific evasive operation parameters when the preparation level early warning and the immediate takeover level early warning are triggered, thereby improving the practicality and operability of the early warning and enabling the early warning to develop from a simple prompt to accurate decision support.

[0041] In the embodiments of the present application, the optimal takeover time is determined by simulating the system risk changes at different delayed takeover time points and selecting the future time point with the minimum risk accumulation as the recommended takeover timing.

[0042] In another possible embodiment, different delayed takeover time points (such as 0.5 seconds, 1 second, 1.5 seconds, etc.) are simulated, for each time point, the risk accumulation value (combining factors such as collision time and operation deviation) of the time point is calculated, the risk accumulation values of all time points are compared, and the future time point with the minimum risk accumulation is selected as the optimal takeover time.

[0043] By simulating the system risk changes at different delayed takeover time points and selecting the future time point with the minimum risk accumulation as the recommended takeover timing, the optimal takeover time is accurately determined, and the takeover timing is changed from experience judgment to data-driven optimization selection.

[0044] Embodiment 2

[0045] Please refer to Figure 2 The present application also provides a remote takeover risk early warning method based on a behavior prediction model, which comprises the following steps: S1, collecting and fusing the data of the vehicle, the driving environment and the driving station in real time, and generating fused data; S2, inputting the fused data into the behavior prediction model of the pre-trained double-flow space-time graph neural network and the vector space analysis engine, synchronously predicting the future trajectory sequence of the vehicle, the future operation sequence of the driver and the space-time intersection point information of the vehicle and the surrounding moving objects, and outputting the prediction result; S3, based on the prediction result, quantitatively calculating the dynamic takeover risk probability value at the current time, and evaluating the possibility of avoiding the predicted space-time intersection point through operation; S4, based on the dynamic takeover risk probability value and the risk avoidability evaluation result, performing hierarchical early warning and outputting the takeover strategy suggestion containing specific avoidance operation.

[0046] In another possible embodiment, first, the multi-source data perception module collects the data of the controlled vehicle, the driving environment and the driving station in real time, and fuses these data to generate fused data; then, the fused data is input into the behavior prediction model of the pre-trained double-flow space-time graph neural network and the vector space analysis engine, and the future trajectory sequence of the vehicle, the future operation sequence of the driver and the space-time intersection point information of the vehicle and the surrounding moving objects are synchronously predicted, and the prediction result is output; then, the risk quantifier module quantitatively calculates the dynamic takeover risk probability value at the current time based on the prediction result, and evaluates the possibility of avoiding the predicted space-time intersection point through operation; finally, the early warning decision and execution module performs hierarchical early warning based on the dynamic takeover risk probability value and the risk avoidability evaluation result, and outputs the takeover strategy suggestion containing specific avoidance operation.

[0047] By constructing the complete process of "data collection fusion - behavior and interaction prediction - risk quantitative evaluation - grading early warning decision", the full-process automation and precision processing of remote takeover risk early warning are realized, and a closed-loop system from data input to decision output is formed.

[0048] The existing behavior prediction model has a simple structure and cannot fully capture the spatio-temporal dependence relationship of vehicle trajectory and driver behavior and the interaction relationship between the two, resulting in insufficient prediction accuracy.

[0049] In the embodiment of the present application, the training and construction method of the behavior prediction model of the double-flow spatio-temporal graph neural network comprises: S21, constructing a time series data set containing historical vehicle trajectories, environmental context, driver operation sequences and corresponding takeover event labels; S22, vehicle trajectory prediction flow, based on the spatio-temporal graph constructed based on historical vehicle state and environmental information, capturing spatial topology and time dependence through a spatio-temporal graph convolution network, outputting a future trajectory sequence of the vehicle; driver behavior prediction flow, based on the historical driver operation sequence, extracting local features through a time series convolution network, and then capturing long-range dependence through a self-attention mechanism, outputting a future operation sequence of the driver; S23, based on the joint loss function of minimizing the vehicle trajectory prediction error and the driver behavior prediction error, the network parameters are trained end-to-end, wherein the expression of the joint loss function is: ; represents the loss of the vehicle trajectory prediction flow, which is calculated by mean square error, and the specific expression is the sum of squares of the Euclidean distance between the predicted trajectory points and the real trajectory points, represents the loss of the driver behavior prediction flow, which is calculated by cross-entropy loss, and the specific expression is the difference between the predicted Caozhi instruction distribution and the real operation quality distribution, represents the contrast learning loss, which is calculated by the interval loss based on triplets, used to pull the hidden layer feature vectors of the vehicle trajectory flow and the driver behavior flow at the same time step, while pushing the feature vectors at different time steps or in different scenes, to promote the consistency of the two prediction flows in understanding the same scene, 、 、 are adjustable parameters of the vehicle trajectory prediction flow loss, the driver behavior prediction flow loss and the contrast learning loss, respectively.

[0050] In another possible embodiment, first the historical driving data of the mobile phone is used to construct a time series dataset containing historical vehicle trajectory, environmental context, driver operation sequence and corresponding takeover practice label; wherein the vehicle trajectory prediction stream is based on historical vehicle state and environmental information to construct a spatio-temporal graph, and the spatial topology and time dependence are captured through a spatio-temporal graph convolution network to output a future trajectory sequence of the vehicle; the driver behavior prediction stream is based on the historical driver operation sequence, and local features are extracted through a time series convolution network, and long-range dependence is captured through a self-attention mechanism to output a future operation sequence of the driver; in the encoder stage of the vehicle trajectory prediction stream and the driver behavior prediction stream, a collaborative attention fusion layer is introduced to realize the cross-stream interaction of the vehicle state features and the driver behavior features; based on a joint loss function for minimizing the vehicle trajectory prediction error and the driver behavior prediction error, the joint loss function expression is: wherein is the mean square error loss of the vehicle trajectory prediction stream, is the cross-entropy loss of the driver behavior prediction stream, is the contrastive learning loss, , , is an adjustable parameter corresponding to the loss, and the network parameters are trained end-to-end.

[0051] By designing a behavior prediction model of a double-stream spatio-temporal graph neural network, the model is divided into a vehicle trajectory prediction stream and a driver behavior prediction stream, a collaborative attention fusion layer is introduced to realize cross-stream interaction, and a joint loss function is used for end-to-end training, which greatly improves the prediction accuracy of the vehicle trajectory and the driver behavior, and makes the behavior prediction from single mode and weak interaction to multi-modal and strong interaction.

[0052] The existing dynamic takeover risk probability calculation method lacks comprehensive modeling of collision time, driver operation deviation and avoidance operation effect, resulting in inaccurate risk quantization results.

[0053] In the embodiments of the present application, the process of quantitatively calculating the dynamic takeover risk probability value at the current time includes: S31, based on the predicted spatio-temporal intersection point information, obtaining the most critical collision time within a preset time , wherein the is defined as the time required for the vehicle to collide with the nearest obstacle; S32, based on the predicted driver operation, calculating the deviation degree from the ideal safe operation model , wherein the is quantified by calculating the cosine similarity or Euclidean distance of the predicted operation sequence and the ideal operation sequence in the vector space; S33, a set of preset avoidance operations is simulated, and the improvement degree after executing each avoidance operation is calculated to obtain an avoidability index , the Defined as after performing the evasion operation The increment and the original The ratio; S34, based on the above , as well as Construct a dynamic risk function, the expression of which is: ;in and for Adjustable weighting coefficients for Adjustable weighting coefficients for Adjustable weighting coefficients This represents the final calculated dynamic takeover risk probability value, normalized by the Sigmoid function to... Interval.

[0054] In another possible embodiment, based on the predicted spatiotemporal intersection information, the most critical collision time within a preset time period is obtained. , The estimated time required for the vehicle to launch into the nearest obstacle is calculated; based on the predicted driver actions, the deviation from the ideal safe operating model is calculated. , To quantify the cosine similarity or Euclidean distance between the predicted operation sequence and the ideal operation sequence in the vector space, a set of pre-defined evasion operations (such as operations with different decelerations and different steering angles) is simulated, and the result after executing each evasion operation is calculated. The degree of improvement is used to obtain an avoidability index. , After performing the evasion operation The increment and the original The ratio; , as well as Substituting into the dynamic risk function, the expression is: ,in , , , The adjustable weighting coefficients and the dynamic takeover risk probability value are normalized to the value using the Sigmoid function. Interval.

[0055] By combining collision time, driver operation deviation, and avoidability indicators, a dynamic risk function is constructed to achieve accurate and dynamic quantification of takeover risk probability, enabling risk quantification to move from a single factor to a precise calculation based on the fusion of multiple factors.

[0056] In the embodiments of the present application, the takeover strategy suggestion includes a suggested deceleration value, a steering angle change amount, and a suggested execution time point of the evasion operation.

[0057] In another possible embodiment, when a preparation level or immediate takeover level warning is triggered, the strategy suggestion unit calculates and outputs a suggested deceleration value (such as ), a steering angle change amount (such as turning 5° to the left), and a suggested execution time point of the evasion operation (such as executing after 2 seconds) based on the risk avoidability evaluation result.

[0058] The explicit takeover strategy suggestion includes a suggested deceleration value, a steering angle change amount, and a suggested execution time point of the evasion operation, providing specific and operable evasion guidance for the remote driver, so that the strategy suggestion changes from abstract to specific and executable command.

[0059] In order to better illustrate the technical solutions of the present application, the intelligent driving remote takeover scenario is taken as an example for illustration:

[0060] Firstly, the vehicle state perception unit of the multi-source data perception module continuously collects vehicle data (current speed 100 km / h, acceleration 0, steering light off) through the CAN bus, the environment perception unit collects real-time front environment information (a truck is traveling at a speed of 80 km / h in the same direction, 100 meters away from the vehicle, the lane line is straight, and the traffic signal light is green) through the laser radar, millimeter wave radar, and camera, and the driver perception unit collects the driving station data (steering wheel angle 0°, accelerator pedal opening degree 50%, and eye tracker shows that the driver's line of sight focuses on the front) and fuses the data after cleaning and standardization to generate multi-source fusion data.

[0061] Then, the fusion data is input into the pre-trained dual-flow spatio-temporal graph neural network behavior prediction model and vector space analysis engine. The vehicle trajectory prediction flow of the dual-flow spatio-temporal graph neural network is based on vehicle state and environment information to construct a spatio-temporal graph, and predicts the vehicle's trajectory sequence in the next 5 seconds (maintaining the current lane, and the speed slowly decreases to 95 km / h) through a spatio-temporal graph convolution network; the driver behavior prediction flow is based on the historical operation sequence, and predicts the driver's future operation sequence (maintaining the accelerator opening degree for the next 2 seconds) through time series convolution and self-attention mechanism, and the two are cross-flow interacted through a collaborative attention fusion layer. The vector space analysis engine constructs a dynamic vector space with the vehicle as the origin, converts the predicted trajectory of the front truck into a relative motion vector, and calculates that the vehicle and the truck will have a spatio-temporal intersection point in the next 3 seconds.

[0062] Then, the risk quantifier module calculates the collision time ( = 3) seconds based on the spatio-temporal intersection point information, and calculates the deviation degree ( =0.3), the simulation of the evasion operation (deceleration to 85km / h) calculates the avoidability index =0.2), the dynamic risk function ( ) is calculated (R≈0.56), and the takeover risk probability value is about 0.56 after normalization by the Sigmoid function.

[0063] Finally, the early warning decision and execution module triggers the preparation level warning according to the takeover risk probability value 0.56 (assuming that the second threshold is 0.5 and the third threshold is 0.7), displays an orange prompt box with a prompt sound. At the same time, the strategy suggestion unit simulates the risk accumulation value at different delayed takeover time points, determines the optimal takeover time as 1 second, and outputs the recommended deceleration value -2m / s², the steering angle change amount 0° (maintain lane) and the recommended execution time point of the evasion operation as 1 second.

[0064] The above describes the optional implementation of the embodiments of the present application in detail in combination with the drawings, but the embodiments of the present application are not limited to the specific details in the above implementation, and various simple modifications can be made to the technical solutions of the embodiments of the present application within the technical concept of the embodiments of the present application, and these simple modifications all belong to the protection scope of the embodiments of the present application.

[0065] In addition, it should be noted that each specific technical feature described in the above specific implementation can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the embodiments of the present application will not be described again.

[0066] In addition, any combination of various different embodiments of the present application can also be made, as long as it does not deviate from the idea of the embodiments of the present application, and it should also be considered as disclosed by the embodiments of the present application.

Claims

1. A method for early warning of remote takeover risk based on a behavior prediction model, characterized in that, The method comprises the following steps: S1, collecting and fusing data of a vehicle, a driving environment and a driver's cab in real time, and generating fused data; S2, inputting the fused data into a behavior prediction model of a pre-trained double-flow spatio-temporal graph neural network and a vector space analysis engine, synchronously predicting a future trajectory sequence of the vehicle, a future operation sequence of the driver and spatio-temporal intersection information between the vehicle and surrounding moving objects, and outputting a prediction result; S3, based on the prediction result, quantitatively calculating a dynamic takeover risk probability value at the current time, and evaluating the possibility of avoiding the predicted spatio-temporal intersection through operation; The process of quantitatively calculating the dynamic takeover risk probability value at the current time comprises: S31、based on the predicted space-time intersection point information, obtaining the most critical collision time within a preset time wherein the defined as the time required for the vehicle to collide with the nearest obstacle S32, based on the predicted driver future operation sequence, calculate a degree of deviation from an ideal safe operation model wherein the quantified by calculating the cosine similarity or Euclidean distance in the vector space between the predicted driver future operation sequence and the ideal operation sequence; S33, simulate a set of pre-set avoidance operation set, calculate the improvement degree after executing each avoidance operation , get avoidability index , the defined as the increment after executing avoidance operation , the ratio of the original ; S34, based on the , and , a dynamic risk function is constructed, expressed as: ； wherein and is an adjustable weight coefficient, is an adjustable weight coefficient, is an adjustable weight coefficient, represents the final calculated dynamic takeover risk probability value, and is normalized to the interval by the Sigmoid function; S4, based on the dynamic takeover risk probability value and the risk avoidability evaluation result, performing hierarchical early warning and outputting a takeover strategy suggestion containing specific avoidance operations.

2. The remote takeover risk early warning method based on a behavior prediction model according to claim 1, characterized in that, The training and construction method of the behavior prediction model of the double-flow spatio-temporal graph neural network comprises: S21, constructing a time series data set containing historical vehicle trajectories, environmental context, driver operation sequences and corresponding takeover event labels; S22, a vehicle trajectory prediction flow, based on a spatio-temporal graph constructed based on historical vehicle states and environmental information, capturing spatial topology and time dependence through a spatio-temporal graph convolution network, and outputting a future trajectory sequence of the vehicle; a driver behavior prediction flow, based on a historical driver operation sequence, extracting local features through a time series convolution network, and then capturing long-range dependence through a self-attention mechanism, and outputting a future operation sequence of the driver; S23, based on a joint loss function for minimizing vehicle trajectory prediction error and driver behavior prediction error, performing end-to-end training on network parameters, wherein the expression of the joint loss function is: ； represents the loss of vehicle trajectory prediction flow, calculated by mean square error, specifically expressed as the sum of squares of the Euclidean distance between the predicted trajectory points and the real trajectory points, represents the loss of driver behavior prediction flow, calculated by cross-entropy loss, specifically expressed as the difference between the predicted Caozhi instruction distribution and the real operation quality distribution, represents the contrastive learning loss, calculated by the triplet-based margin loss, used to pull the hidden layer feature vectors of the vehicle trajectory flow and the driver behavior flow at the same time step closer, and to push the feature vectors at different time steps or in different scenes away, so as to promote the consistency of the two prediction flows in understanding the same scene, are adjustable parameters of the vehicle trajectory prediction flow loss, the driver behavior prediction flow loss and the contrastive learning loss respectively.​​ 3. The method of claim 1, wherein the method further comprises: The takeover strategy suggestion comprises a suggested deceleration value, a steering angle change amount and a suggested execution time point of the avoidance operation.

4. A remote takeover risk early warning system based on a behavior prediction model, applied to a remote takeover risk early warning method based on a behavior prediction model according to any one of claims 1-3, characterized in that, It comprises: A multi-source data perception module for collecting and preprocessing data from a controlled vehicle, a driving environment and a driver's cab in real time; A behavior and interaction prediction module comprising a behavior prediction model based on a double-flow spatio-temporal graph neural network and a vector space analysis engine, the behavior and interaction prediction module being configured to receive the processing result of the multi-source data perception module and synchronously output a future trajectory prediction sequence of the vehicle, a driver operation behavior prediction sequence and spatio-temporal intersection prediction information between the vehicle and surrounding objects; A risk quantifier module connected with the behavior and interaction prediction module, the risk quantifier module being configured to generate a dynamic takeover risk probability value based on the future trajectory prediction sequence of the vehicle, the driver operation behavior prediction sequence and the spatio-temporal intersection prediction information, and to generate a risk avoidability evaluation result according to the possibility of avoiding the predicted risk through driving operation; The early warning decision and execution module is connected to the risk quantifier module. The early warning decision and execution module is used to trigger a graded early warning signal based on the takeover risk probability value and the risk avoidability assessment result, and generate takeover strategy suggestions that include specific avoidance operations.

5. The remote takeover risk early warning system based on behavior prediction model according to claim 4, characterized in that, The multi-source data sensing module includes: The vehicle status sensing unit is used to acquire vehicle CAN bus data, including but not limited to vehicle speed, acceleration, yaw rate, and turn signal status. An environmental perception unit is used to acquire vehicle sensor data, including but not limited to LiDAR, millimeter-wave radar, and camera data. The vehicle sensor data is used to perceive surrounding obstacles, lane lines, and traffic light status. The driver perception unit is used to acquire dashboard data, which includes, but is not limited to, steering wheel angle, accelerator / brake pedal opening, operation frequency, and eye tracker and / or physiological sensor data.

6. The remote takeover risk early warning system based on behavior prediction model according to claim 4, characterized in that, The vector space analysis engine includes: A dynamic vector space is constructed with the vehicle itself as the origin of the coordinate system. A baseline vector is constructed based on the vehicle's predicted future trajectory. The predicted motion trajectory of moving objects around the vehicle is converted into a relative motion vector relative to the vehicle itself. Based on the reference vector and all relative motion vectors, it is determined whether the vehicle itself and the moving objects around the vehicle will have a spatiotemporal intersection point within a preset time in the future, and the time and information of the spatiotemporal intersection point are obtained. The time and information of the occurrence of the spatiotemporal intersection are output to the risk quantifier module to assess the possibility of avoiding the spatiotemporal intersection by changing the vehicle's own speed, acceleration, or direction.

7. The remote takeover risk early warning system based on behavior prediction model according to claim 4, characterized in that, The risk quantifier module includes: The collision time is calculated based on the predicted information of the spatiotemporal intersection point; Based on the driver's operational behavior prediction sequence, the deviation between the safe operation required to avoid the spatiotemporal intersection is calculated. Simulate at least one avoidance operation based on the deviation, and obtain the change in the collision time after performing the avoidance operation; The dynamic takeover risk probability value is output by fusing the collision time, the deviation, and the avoidance operation.

8. The remote takeover risk early warning system based on behavior prediction model according to claim 4, characterized in that, The early warning decision and execution module includes: A tiered early warning unit is used to preset multiple risk thresholds, including a first threshold, a second threshold, and a third threshold, wherein the third threshold is greater than the second threshold and the first threshold. When the takeover risk probability value exceeds the first threshold but is lower than the second threshold, a concern-level early warning is triggered. When the takeover risk probability value exceeds the second threshold but is lower than the third threshold, a preparation-level early warning is triggered. When the takeover risk probability value exceeds the third threshold, an immediate takeover-level early warning is triggered. The strategy suggestion unit is used to generate takeover strategy suggestions, including optimal takeover time and specific avoidance operation parameters, based on the risk avoidability assessment results when the preparation-level warning and the immediate takeover-level warning are triggered.

9. The remote takeover risk early warning system based on behavior prediction model according to claim 8, characterized in that, The optimal takeover time is determined by simulating the changes in system risk at different delayed takeover times and selecting the future time point with the minimum risk accumulation as the recommended takeover opportunity.

Images