Multi-terminal collaborative driver risk cognitive ability monitoring system and method

Abstract

The invention discloses a multi-terminal collaborative driver risk cognitive ability monitoring system and method. The system comprises: a vehicle-mounted terminal comprising a vehicle-mounted information acquisition assembly; the roadside end comprises a roadside information acquisition assembly; the cloud comprises a scene matching module, a state recognition module, a cognitive monitoring module and an information storage module for storing cloud information; the vehicle-mounted information acquisition assembly, the roadside information acquisition assembly and the information storage module are respectively in communication connection with the scene matching module, and the vehicle-mounted information acquisition assembly is in communication connection with the state recognition module; the scene matching module is in communication connection with the cognitive monitoring module; the state recognition module is in communication connection with the cognitive monitoring module; the cognitive monitoring module is in communication connection with the information storage module; according to the invention, a cooperative working mechanism of the vehicle-mounted terminal, the roadside terminal and the cloud terminal is constructed, and the risk cognitive ability of the driver in the current risk scene is monitored by matching the current driving risk scene and combining the identified current driver state.

Description

Technical Field

[0001] The present invention relates to the field of intelligent connected vehicles, and in particular to a multi-terminal collaborative monitoring system and method for the driver's risk perception ability. Background Art

[0002] With the rapid development of autonomous driving technology, the automation level of most current autonomous vehicles is still in the L2 to L3 level, that is, the "human-machine co-driving" stage. In this stage, the autonomous driving system is the main driving operation entity, and the driver needs to take over the vehicle in a timely manner when the system requests to handle emergencies.

[0003] Stable and reliable takeover intervention is the key to ensuring the safety of human-machine co-driving, and the driver's "sufficient" risk perception ability is the basis for ensuring "sufficient" takeover performance; research shows that the driver's risk perception ability is affected by various factors such as the driver's own attributes, the design of the autonomous driving system, and the characteristics of the traffic environment. Integrating human-machine-environment big data is a necessary means to achieve accurate monitoring of the driver's risk perception ability, but the computing power of the current in-vehicle computing unit is difficult to meet the needs of large-scale data computing; at the same time, the safe completion of the driver's takeover task requires at least 7 seconds of takeover time budget, and the existing autonomous vehicles mainly rely on in-vehicle sensors to perceive the surrounding traffic conditions, making it difficult to identify long-time domain driving risks. With the development and application of wireless communication technologies such as 5G, the intelligent connection between the in-vehicle terminal, the roadside terminal, and the cloud has become possible. The powerful computing resources of the cloud provide an effective way to solve the above problems.

[0004] In summary, those skilled in the art urgently need a multi-terminal collaborative monitoring system and method for the driver's risk perception ability to solve the deficiencies in the prior art. Summary of the Invention

[0005] In view of the above deficiencies of the prior art, the present invention provides a multi-terminal collaborative monitoring system and method for the driver's risk perception ability to solve the problems of the lack of a multi-terminal collaborative mechanism, limited long-time domain driving risk identification ability, and insufficient computing power of the in-vehicle computing unit in the prior art.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a multi-terminal collaborative monitoring system for the driver's risk perception ability, including:

[0008] An in-vehicle terminal, including an in-vehicle information collection component for collecting in-vehicle terminal information;

[0009] A roadside terminal, including a roadside information collection component for collecting roadside terminal information;

[0010] The cloud, including a scene matching module, a state recognition module, a cognitive monitoring module, and an information storage module for storing cloud information;

[0011] The vehicle - side information includes the self - vehicle movement information, the vehicle - surrounding environment information, and the driver's physiological information; the cloud information includes the map information, the risk scenario library information, and the driver attribute information;

[0012] The vehicle - mounted information acquisition component, the roadside information acquisition component, and the information storage module are respectively communicatively connected to the scene matching module, and the vehicle - mounted information acquisition component is communicatively connected to the state recognition module;

[0013] The scene matching module is communicatively connected to the cognitive monitoring module, and is used to perform scene matching on the self - vehicle movement information, the vehicle - surrounding environment information, and the roadside - end information with the map information and the risk scenario library information to obtain the matching result of the current driving risk scenario, and send the matching result to the cognitive monitoring module;

[0014] The state recognition module is communicatively connected to the cognitive monitoring module, and is used to identify the driver's physiological information to obtain the state recognition result of the driver, and send the state recognition result to the cognitive monitoring module;

[0015] The cognitive monitoring module is communicatively connected to the information storage module, and is used to evaluate the driver's risk perception ability based on the matching result, the state recognition result, and the driver attribute information to obtain the evaluation result of the driver's risk perception ability, so as to monitor the driver's risk perception ability.

[0016] In an optional embodiment, the vehicle - side further includes a human - machine co - driving control module;

[0017] The human - machine co - driving control module is respectively communicatively connected to the scene matching module and the cognitive monitoring module, and is used to obtain the matching result of the current driving risk scenario and the evaluation result of the driver's risk perception ability, and evaluate the driver's takeover ability based on the matching result and the evaluation result of the cognitive ability to obtain the evaluation result of the driver's takeover ability; it is also used to perform vehicle control right allocation according to the evaluation result of the driver's takeover ability.

[0018] In an optional embodiment, the human - machine co - driving control module is communicatively connected to the vehicle - mounted information acquisition component, and is used to obtain the self - vehicle movement information, and calculate the time required for the current driving to the collision and the severity of the collision based on the matching result and the self - vehicle movement information, so as to perform driving risk warning.

[0019] In an optional embodiment, the vehicle - side further includes a vehicle - mounted information processing module;

[0020] The input end of the vehicle-mounted information processing module is communicatively connected to the output end of the vehicle-mounted information collection component. The output end of the vehicle-mounted information processing module is respectively communicatively connected to the input ends of the scenario matching module and the state recognition module, so as to perform data preprocessing on the vehicle-mounted information and send the preprocessed vehicle-mounted information to the scenario matching module and the state recognition module respectively;

[0021] The vehicle-mounted information collection component includes a first information collection unit for collecting the movement information of the vehicle itself, a second information collection unit for collecting the vehicle surrounding environment information, and a third information collection unit for collecting the physiological information of the driver;

[0022] The physiological information of the driver includes the facial image sequence information of the driver, the electroencephalogram signal sequence information of the driver, and the eye movement signal sequence information of the driver;

[0023] The output ends of the first information collection unit, the second information collection unit, and the third information collection unit are respectively communicatively connected to the input end of the vehicle-mounted information processing module.

[0024] In an optional implementation manner, the roadside end further includes a roadside information processing module;

[0025] The input end of the roadside information processing module is communicatively connected to the output end of the roadside information collection component, and the output end of the roadside information processing module is communicatively connected to the input end of the scenario matching module, so as to perform data preprocessing on the roadside information and send the preprocessed roadside information to the scenario matching module;

[0026] The roadside information includes the traffic state and road network information within the current driving area of the vehicle itself.

[0027] In an optional implementation manner, the scenario matching module includes a pre-trained neural network model;

[0028] The state recognition module includes a state recognition model; the cognitive monitoring module includes a cognitive evaluation model; both the state recognition model and the cognitive evaluation model are pre-trained based on a large language model; the large language model is built using an encoder-decoder architecture;

[0029] The human-machine co-driving control module includes a pre-constructed fuzzy rule model.

[0030] In a second aspect, the present invention provides a method for monitoring a driver's risk perception ability based on risk scenario matching under multi-terminal collaboration, which is implemented based on the above-mentioned risk perception ability monitoring system, and includes the following steps:

[0031] Collect vehicle-mounted information based on the vehicle-mounted information collection component;

[0032] The vehicle - side information includes the self - vehicle motion information, the vehicle - surrounding environment information, and the driver's physiological information;

[0033] Collect road - side information based on the road - side information collection component;

[0034] The scene matching module obtains the self - vehicle motion information, the vehicle - surrounding environment information, and the road - side information, and obtains the map information and the risk scenario library information from the information storage module;

[0035] The scene matching module performs scene matching on the self - vehicle motion information, the vehicle - surrounding environment information, and the road - side information, with the map information and the risk scenario library information, based on a pre - trained neural network model, to obtain the matching result of the current driving risk scenario;

[0036] The state recognition model obtains the driver's physiological information and performs state recognition based on a pre - trained large - language model according to the driver's physiological information to obtain the state recognition result of the driver;

[0037] The cognitive monitoring module obtains the matching result of the current driving risk scenario and the state recognition result of the driver, and obtains the driver attribute information from the information storage module;

[0038] The cognitive monitoring module evaluates the driver's risk perception ability based on a pre - trained large - language model according to the matching result of the current driving risk scenario, the state recognition result of the driver, and the driver attribute information, to obtain the evaluation result of the risk perception ability, so as to monitor the driver's risk perception ability.

[0039] In an optional implementation manner, the method further includes: The human - machine co - driving control module obtains the matching result of the current driving risk scenario and the evaluation result of the risk perception ability;

[0040] The human - machine co - driving control module evaluates the driver's takeover ability based on a pre - constructed fuzzy rule model according to the matching result of the current driving risk scenario and the evaluation result of the risk perception ability, to obtain the evaluation result of the driver's takeover ability;

[0041] Perform vehicle control right allocation according to the evaluation result of the driver's takeover ability to achieve human - machine collaborative control.

[0042] In an optional implementation manner, the scene matching module performs scene matching on the self - vehicle motion information, the vehicle - surrounding environment information, and the road - side information, with the map information and the risk scenario library information, based on a pre - trained neural network model, to obtain the matching result of the current driving risk scenario, specifically including the following steps:

[0043] The scene matching module performs multi - modal feature extraction on the self - vehicle motion information, the vehicle - surrounding environment information, and the road - side information;

[0044] Based on a neural network, a predefined risk scenario constructed from map information and a risk scenario library is matched with multi-modal features to obtain a matching result of the current driving risk scenario;

[0045] The multi-modal features include pedestrian-related features, vehicle-related features, road-related features, and environment-related features;

[0046] The function expression of the matching result of the current driving risk scenario is as follows:

[0047]

[0048] In the formula, is the matching result of the ego vehicle driving risk scenario at time t; is the pedestrian-related feature; is the vehicle-related feature; is the road-related feature; is the environment-related feature; is the mapping mechanism between the multi-modal features and the predefined risk scenario.

[0049] In an optional embodiment, the function expression of the cognitive evaluation model is:

[0050]

[0051] Where, is the facial image sequence of the driver; is the electroencephalogram signal sequence of the driver; is the eye movement signal sequence of the driver; DS is the driver state recognition result at time t; DA is the driver attribute information; is the monitoring result of the driver's risk perception ability at time t; P(.) is the conditional probability distribution; l i is the driver risk perception level; is the encoder; is the decoder; is the high-order feature vector generated by the encoder.

[0052] The beneficial effects brought by the embodiments provided by the present invention include:

[0053] The present invention constructs a collaborative working mechanism for in-vehicle, roadside, and cloud ends, enabling the integration of multi-faceted information such as human-vehicle-environment. It dynamically matches the risk scenarios of the current driving through a scenario matching module, and combines the current driver state identified by the state recognition module to evaluate and monitor the driver's risk perception ability in the current risk scenario, so as to achieve long-time-domain identification of driving risks and reliable monitoring of the driver's risk perception ability, effectively improving the safety of human-machine co-driving and reducing road traffic accidents caused by untimely risk warnings, insufficient driver takeover time and ability.

[0054] The present invention obtains and integrates in-vehicle end information and roadside end information, breaks through the physical detection limitations of traditional in-vehicle sensors, obtains dynamic traffic environment data within the ultra-long sight range, and provides high-coverage and high-real-time data support for long-time-domain driving risk identification and risk warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0056] Figure 1 FIG. shows a flowchart of a risk perception ability monitoring system in an embodiment of the present specification;

[0057] Figure 2 FIG. shows another flowchart of a risk perception ability monitoring system in an embodiment of the present specification;

[0058] Figure 3 FIG. shows a flowchart of a risk perception ability monitoring method in an embodiment of the present specification. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The following will describe in detail the features and exemplary embodiments of various aspects of the present invention. In the following detailed description, many specific details are presented in order to provide a comprehensive understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by showing examples of the present invention.

[0060] In the description of the present invention, it should be noted that, unless otherwise clearly specified and defined, the terms "connected" and "coupled" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or a communication connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication between two components inside. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] Embodiment 1

[0062] As Figure 1-2 shown, this embodiment provides a multi-terminal collaborative monitoring system for drivers' risk perception ability, including:

[0063] An in-vehicle terminal, including an in-vehicle information collection component for collecting in-vehicle terminal information;

[0064] A roadside terminal, including a roadside information collection component for collecting roadside terminal information;

[0065] A cloud, including a scene matching module, a state recognition module, a cognition monitoring module, and an information storage module;

[0066] Exemplarily, the in-vehicle information collection component, the roadside information collection component, and the information storage module are respectively communicatively connected to the scene matching module, and the in-vehicle information collection component is communicatively connected to the state recognition module;

[0067] Specifically, the information storage module is used to store cloud information and continuously iterate and update it. The cloud information includes map information, risk scene library information, and driver attribute information; the in-vehicle terminal information includes self-vehicle movement information, vehicle surrounding environment information, and driver physiological information;

[0068] Exemplarily, the scene matching module is communicatively connected to the cognition monitoring module, and is used to perform scene matching on the self-vehicle movement information, vehicle surrounding environment information, and roadside terminal information with the map information and the risk scene library information to obtain the matching result of the current driving risk scene, and send the matching result to the cognition monitoring module; wherein, the scene matching module includes a pre-trained neural network model;

[0069] Specifically, the scene matching module extracts features from the self-vehicle movement information, vehicle surrounding environment information, and roadside terminal information to obtain multi-modal features, and inputs the multi-modal features into the pre-trained neural network model, so that the predefined risk scenes constructed by the map information and the risk scene library information are matched with the multi-modal features in terms of similarity, and finally the risk scene features with the highest similarity are selected as the matching result of the current driving risk scene;

[0070] Among them, the self-vehicle motion information includes the position, speed, acceleration, and steering angle information of the self-vehicle; the vehicle surrounding environment information includes image sequence information, three-dimensional point cloud information, and wireless communication information;

[0071] Exemplarily, the state recognition module is communicatively connected to the cognitive monitoring module, and is used to recognize the physiological information of the driver to obtain the state recognition result of the driver, and send the state recognition result to the cognitive monitoring module; among them, the state recognition module includes a state recognition model, and the state recognition model is pre-trained based on a large language model, and the state recognition model is built using an encoder-decoder architecture;

[0072] Specifically, the state recognition module inputs the driver's physiological information into the state recognition model constructed based on the large language model. The state recognition model outputs the state recognition result of the driver, that is, the current state of the driver, according to the input information and the classification rules of the predefined driver states; among them, the driver's physiological information includes the facial image sequence information of the driver, the electroencephalogram signal sequence information of the driver, and the eye movement signal sequence information of the driver.

[0073] In some embodiments, the state recognition module acquires the facial images, electroencephalogram signals, and eye movement signals of the driver within a period of time, and judges whether the driver is in a fatigued state and / or whether the driver is in a distracted state and / or in what emotional state according to the classification rules of the predefined driver states, and outputs the current state of the driver as the state recognition result of the driver.

[0074] Exemplarily, the cognitive monitoring module is communicatively connected to the information storage module, and is used to evaluate the driver's risk cognitive ability based on the matching result, the state recognition result, and the driver attribute information, and obtain the evaluation result of the driver's risk cognitive ability to realize the monitoring of the driver's risk cognitive ability. Among them, the cognitive monitoring module includes a cognitive evaluation model, and the cognitive evaluation model is pre-trained based on a large language model, and the cognitive evaluation model is built using an encoder-decoder architecture;

[0075] Specifically, the cognitive monitoring module inputs the matching result of the current driving risk scenario, the state recognition result of the driver, and the driver attribute information into the cognitive evaluation model constructed based on the large language model. The cognitive evaluation model outputs the current risk cognitive ability evaluation result of the driver according to the input information and the classification rules of the predefined driver cognitive ability.

[0076] Among them, the driver attribute information includes but is not limited to driving experience, driving style, and acceptance of autonomous driving.

[0077] Exemplarily, the in-vehicle terminal further includes a human-machine co-driving control module;

[0078] Specifically, the human-machine co-driving control module is communicatively connected to the scenario matching module and the cognitive monitoring module respectively, receives the matching result of the current driving risk scenario and the evaluation result of the risk perception ability, evaluates the driver's takeover ability based on the evaluation result of the risk perception ability and the matching result of the current driving risk scenario, and allocates the vehicle control right according to the evaluation result of the takeover ability;

[0079] It should be noted that the human-machine co-driving control module allocates different proportions of vehicle control rights to the driver and the assisted driving system according to the evaluation result of the driver's takeover ability, so that the driver and the assisted driving system can complete the driving operation together.

[0080] In this embodiment, the human-machine co-driving control module includes a pre-built fuzzy rule model;

[0081] Specifically, the fuzzy rule module evaluates the driver's takeover ability according to the input matching result of the current driving risk scenario and the evaluation result of the risk perception ability according to the preset fuzzy rule table to obtain the evaluation result of the driver's takeover ability, as shown in Table 1:

[0082]

[0083] Table 1 is the driver takeover ability evaluation form;

[0084] Among them, the matching result of the current driving risk scenario includes the current scenario risk level, and the current scenario risk level includes safety SA, low risk LR, medium risk MR, medium-high risk MHR, and high risk HR;

[0085] In some embodiments, the human-machine co-driving control module is communicatively connected to the vehicle-mounted information collection component;

[0086] The human-machine co-driving control module is respectively connected to the scenario matching module to obtain the self-vehicle movement information, and based on the matching result and the self-vehicle movement information, calculates the time required for the current driving to the collision and the severity of the collision to issue a driving risk warning.

[0087] Specifically, the matching result of the current driving risk scenario is a specific risk scenario. The human-machine co-driving control module estimates the collision parameters at the time of collision according to the risk scenario of the current driving, and calculates the time required for the current driving to the collision and the severity of the collision based on the collision parameters and the self-vehicle current speed in the self-vehicle movement information;

[0088] Among them, the collision parameters include collision point position, collision speed at the time of vehicle collision, collision angle, collision occurrence position and other collision parameters;

[0089] In this embodiment, the human-machine co-driving control module calculates the Euclidean distance from the current position of the host vehicle to the collision point based on the collision point position and in combination with the map information in the risk scenario, and then calculates the time required for the current driving to the collision using the Euclidean distance from the current position of the host vehicle to the collision point and the current speed of the host vehicle in the host vehicle motion information; the human-machine co-driving control module estimates the collision severity by inputting the collision parameters as collision boundary conditions into the dynamic simulation algorithm.

[0090] Among them, the calculation formula for the time required for the current driving to the collision is:

[0091]

[0092] In the formula, TTC is the time required for the current driving to the collision; D represents the distance between the vehicle and the collision point when braking measures are taken; V represents the speed of the vehicle when braking measures are taken.

[0093] In this embodiment, the human-machine co-driving control module includes a predefined warning scheme; the human-machine co-driving control module uses the predefined warning scheme to generate a graded warning signal according to the time required for the current driving to the collision and the collision severity, so as to issue alarms of different urgency levels, as shown in Table 2.

[0094]

[0095] Table 2 is the output variable table of the warning level.

[0096] Among them, the graded warning signal includes a safety signal S, a low-risk warning signal L, a medium-risk warning signal M, a medium-high-risk warning signal MH, and a high-risk warning signal H.

[0097] In some embodiments, the vehicle-mounted terminal further includes a vehicle-mounted information processing module.

[0098] The input end of the vehicle-mounted information processing module is communicatively connected to the output end of the vehicle-mounted information collection component, and the output end of the vehicle-mounted information processing module is communicatively connected to the input ends of the scene matching module and the state recognition module respectively, for preprocessing the vehicle-mounted terminal information and sending the preprocessed vehicle-mounted terminal information to the scene matching module and the state recognition module respectively.

[0099] In some embodiments, the roadside terminal further includes a roadside information processing module.

[0100] The input end of the roadside information processing module is communicatively connected to the output end of the roadside information collection component, and the output end of the roadside information processing module is communicatively connected to the input end of the scene matching module, for preprocessing the roadside terminal information and sending the preprocessed roadside terminal information to the scene matching module.

[0101] Among them, the roadside information includes the traffic status and road network information within the current driving area of the vehicle; the traffic status within the current driving area of the vehicle includes the movement status of other vehicles and traffic flow within the detectable range of non-vehicle-mounted sensors; the road network information includes road segments, road segment topological relationships, and intersections.

[0102] In this embodiment, the types of information covered by the vehicle-mounted information and roadside information include, but are not limited to, image information, radar point cloud information, and communication information.

[0103] Among them, the preprocessing steps of the vehicle-mounted information processing module and the roadside information processing module for image information include: image grayscale conversion, image geometric transformation, and image enhancement; the preprocessing steps of the vehicle-mounted information processing module and the roadside information processing module for radar point cloud information include: data screening, filtering and segmentation, point cloud data sampling, point cloud filtering, and point cloud feature extraction.

[0104] In some embodiments, the scene matching module, the state recognition module, and the cognitive monitoring module all include multiple high-performance computing clusters to efficiently process a large amount of data.

[0105] In some embodiments, the vehicle-mounted information acquisition component includes a first information acquisition unit, a second information acquisition unit, and a third information acquisition unit.

[0106] Specifically, the output end of the first information acquisition unit is communicatively connected to the input end of the vehicle-mounted information processing module for acquiring the movement information of the vehicle itself; the output end of the second information acquisition unit is communicatively connected to the input end of the vehicle-mounted information processing module for acquiring the vehicle surrounding environment information within the detectable range around the vehicle; the output end of the third information acquisition unit is communicatively connected to the input end of the vehicle-mounted information processing module for acquiring the physiological information of the driver.

[0107] In some embodiments, the roadside information acquisition component includes, but is not limited to, a high-definition camera, a detection radar, and a communication device installed on the roadside; among them, the roadside includes street lamp poles, traffic light poles, and traffic sign poles, etc.

[0108] The first information acquisition unit includes, but is not limited to, a GPS, a speed sensor, an acceleration sensor, and a steering angle sensor.

[0109] The second information acquisition unit includes, but is not limited to, several high-definition cameras installed outside the vehicle body, a 192-line lidar, a millimeter-wave radar, and a communication device.

[0110] The third information acquisition unit includes, but is not limited to, a high-definition camera installed at the inside rearview mirror, a high-definition camera installed at the top inside of the left A-pillar, a non-embedded electroencephalogram acquisition device, and a wearable eye movement acquisition device.

[0111] Embodiment 2

[0112] Such asFigure 3 As shown in Figure 3 , this embodiment provides a method for monitoring a driver's risk perception ability based on risk scenario matching under multi-terminal collaboration, including the following steps:

[0113] S1: Collect in-vehicle information based on the in-vehicle information collection component;

[0114] The in-vehicle information includes self-vehicle movement information, vehicle surrounding environment information, and driver physiological information;

[0115] S2: Collect roadside information based on the roadside information collection component;

[0116] S3: The scenario matching module obtains the self-vehicle movement information, vehicle surrounding environment information, and roadside information, and obtains the map information and risk scenario library information from the information storage module;

[0117] S4: The scenario matching module performs scenario matching on the self-vehicle movement information, vehicle surrounding environment information, and roadside information with the map information and risk scenario library information based on a pre-trained neural network model to obtain the matching result of the current driving risk scenario;

[0118] S5: The state recognition model obtains the driver physiological information, and performs state recognition based on a pre-trained large language model according to the driver physiological information to obtain the driver's state recognition result;

[0119] S6: The cognitive monitoring module obtains the matching result of the current driving risk scenario and the driver's state recognition result, and obtains the driver attribute information from the information storage module;

[0120] S7: The cognitive monitoring module evaluates the driver's risk perception ability based on a pre-trained large language model according to the matching result of the current driving risk scenario, the driver's state recognition result, and the driver attribute information, and obtains the evaluation result of the risk perception ability to realize the monitoring of the driver's risk perception ability.

[0121] In some embodiments, the method further includes: S8: The human-machine co-driving control module obtains the matching result of the current driving risk scenario and the evaluation result of the risk perception ability;

[0122] S9: The human-machine co-driving control module evaluates the driver's takeover ability based on a pre-constructed fuzzy rule model according to the matching result of the current driving risk scenario and the evaluation result of the risk perception ability to obtain the evaluation result of the driver's takeover ability;

[0123] S10: Perform vehicle control right allocation according to the evaluation result of the driver's takeover ability to realize human-machine collaborative control.

[0124] In some embodiments, the specific steps of S4 include the following:

[0125] The scene matching module performs multi-modal feature extraction on the ego-vehicle motion information, vehicle surrounding environment information, and roadside unit information;

[0126] Based on a neural network, a predefined risk scenario constructed from map information and risk scenario library information is matched with the multi-modal features to obtain the matching result of the current driving risk scenario;

[0127] The multi-modal features include pedestrian-related features, vehicle-related features, road-related features, and environment-related features;

[0128] The function expression of the matching result of the current driving risk scenario is as follows:

[0129]

[0130] In the formula, is the matching result of the ego-vehicle driving risk scenario at time t; is the pedestrian-related feature; is the vehicle-related feature; is the road-related feature; is the environment-related feature; is the mapping mechanism between the multi-modal features and the predefined risk scenario, that is, the corresponding relationship between the multi-modal features and the risk scenario.

[0131] In some embodiments, the function expression of the cognitive evaluation model is:

[0132]

[0133] Among them, is the facial image sequence of the driver; is the electroencephalogram signal sequence of the driver; is the eye movement signal sequence of the driver; (t-N+1:t) is the continuous time from t-N+1 to t, including the start time t-N+1 and the end time t, covering a total of N time points; DS is the driver state recognition result at time t; DA is the driver attribute information; is the monitoring result of the driver's risk perception ability at time t; P(.) is the conditional probability distribution; l i is the driver risk perception level; is the encoder; is the decoder; is the high-order feature vector generated by the encoder.

[0134] In summary, in this embodiment, a multi-terminal collaborative working mechanism is constructed through the vehicle-mounted terminal, roadside terminal, and cloud. Information on people, vehicles, the environment, and other aspects in the data acquired by the vehicle-mounted terminal and roadside terminal is fused. The risk scenarios of the current driving are dynamically matched through the scenario matching module, and combined with the current driver state identified by the state recognition module, the risk perception ability of the driver in the current risk scenario is evaluated and monitored, so as to realize the long-time domain identification of driving risks and the reliable monitoring of the driver's risk perception ability, effectively improve the safety of human-machine co-driving, and reduce road traffic accidents caused by untimely risk warnings, insufficient driver takeover time and ability.

[0135] In this embodiment, by acquiring and fusing the vehicle-mounted terminal information and roadside terminal information, the physical detection limitations of traditional vehicle-mounted sensors are broken through, and the dynamic data of the traffic environment within the ultra-line-of-sight range is acquired, providing high-coverage and high-real-time data support for long-time domain driving risk identification and risk warning. At the same time, with the help of the high-performance computing power of the cloud, the present invention can efficiently process a large amount of data, solving the problem of low efficiency in processing large-scale data by the vehicle-mounted terminal, resulting in data processing delay.

[0136] The above are only the preferred embodiments of the present invention. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention.

Claims

1. A multi-terminal collaborative monitoring system for drivers' risk perception ability, characterized in that, Including: The vehicle-mounted terminal, including a vehicle-mounted information collection component for collecting vehicle-mounted terminal information; The roadside terminal, including a roadside information collection component for collecting roadside terminal information; The cloud, including a scene matching module, a state recognition module, a cognitive monitoring module, and an information storage module for storing cloud information; The vehicle-mounted terminal information includes self-vehicle movement information, vehicle surrounding environment information, and driver physiological information; the cloud information includes map information, risk scene library information, and driver attribute information; The vehicle-mounted information collection component, the roadside information collection component, and the information storage module are respectively communicatively connected to the scene matching module, and the vehicle-mounted information collection component is communicatively connected to the state recognition module; The scene matching module is communicatively connected to the cognitive monitoring module, and is used to perform scene matching on the self-vehicle movement information, vehicle surrounding environment information, and roadside terminal information with the map information and the risk scene library information, so as to obtain the matching result of the current driving risk scene, and send the matching result to the cognitive monitoring module; The state recognition module is communicatively connected to the cognitive monitoring module, and is used to recognize the driver physiological information to obtain the state recognition result of the driver, and send the state recognition result to the cognitive monitoring module; The cognitive monitoring module is communicatively connected to the information storage module, and is used to evaluate the driver's risk perception ability based on the matching result, state recognition result, and driver attribute information, and obtain the evaluation result of the driver's risk perception ability, so as to monitor the driver's risk perception ability.

2. The system according to claim 1, characterized in that, The vehicle-mounted terminal further includes a human-machine co-driving control module; The human-machine co-driving control module is respectively communicatively connected to the scene matching module and the cognitive monitoring module, and is used to obtain the matching result of the current driving risk scene and the evaluation result of the driver's risk perception ability, and evaluate the driver's takeover ability based on the matching result and the evaluation result of the cognitive ability, so as to obtain the evaluation result of the driver's takeover ability; It is also used to allocate vehicle control rights according to the evaluation result of the driver's takeover ability.

3. The system according to claim 2, wherein The human-machine co-driving control module is communicatively connected to the vehicle-mounted information collection component, and is used to obtain the self-vehicle movement information, and calculate the time required for the current driving to the collision and the severity of the collision based on the matching result and the self-vehicle movement information, so as to issue a driving risk warning.

4. The system according to claim 1, wherein The vehicle-mounted terminal further includes a vehicle-mounted information processing module; The input end of the vehicle-mounted information processing module is communicatively connected to the output end of the vehicle-mounted information collection component, and the output end of the vehicle-mounted information processing module is respectively communicatively connected to the input ends of the scene matching module and the state recognition module, and is used to perform data preprocessing on the vehicle-mounted terminal information, and send the preprocessed vehicle-mounted terminal information to the scene matching module and the state recognition module respectively; The vehicle-mounted information collection component includes a first information collection unit for collecting self-vehicle movement information, a second information collection unit for collecting vehicle surrounding environment information, and a third information collection unit for collecting driver physiological information; The driver physiological information includes the driver's facial image sequence information, the driver's electroencephalogram signal sequence information, and the driver's eye movement signal sequence information; The output ends of the first information collection unit, the second information collection unit, and the third information collection unit are respectively communicatively connected to the input end of the vehicle-mounted information processing module.

5. The system according to claim 1, wherein The roadside unit further includes a roadside information processing module; The input end of the roadside information processing module is communicatively connected to the output end of the roadside information collection component, and the output end of the roadside information processing module is communicatively connected to the input end of the scenario matching module, for performing data preprocessing on the roadside unit information and sending the preprocessed roadside unit information to the scenario matching module; The roadside unit information includes traffic conditions and road network information within the current driving area of the host vehicle.

6. The system according to claim 1, wherein The scenario matching module includes a pre-trained neural network model; The state recognition module includes a state recognition model; the cognitive monitoring module includes a cognitive evaluation model; both the state recognition model and the cognitive evaluation model are pre-trained based on a large language model; the large language model is built using an encoder-decoder architecture; The human-machine co-driving control module includes a pre-constructed fuzzy rule model.

7. A monitoring method for a driver's risk perception ability based on risk scenario matching under multi-terminal collaboration, characterized in that, Implemented based on the risk cognitive ability monitoring system according to any one of claims 1 to 6, including the following steps: Collect vehicle-mounted information based on the vehicle-mounted information collection component; The vehicle-mounted information includes host vehicle movement information, vehicle surrounding environment information, and driver physiological information; Collect roadside unit information based on the roadside information collection component; The scenario matching module obtains the host vehicle movement information, vehicle surrounding environment information, and roadside unit information, and obtains map information and risk scenario library information from the information storage module; The scenario matching module performs scenario matching on the host vehicle movement information, vehicle surrounding environment information, and roadside unit information, and the map information and risk scenario library information based on the pre-trained neural network model to obtain a matching result of the current driving risk scenario; The state recognition model obtains the driver physiological information, and performs state recognition based on the pre-trained large language model according to the driver physiological information to obtain a state recognition result of the driver; The cognitive monitoring module obtains the matching result of the current driving risk scenario and the state recognition result of the driver, and obtains driver attribute information from the information storage module; The cognitive monitoring module evaluates the driver's risk cognitive ability based on the pre-trained large language model according to the matching result of the current driving risk scenario, the state recognition result of the driver, and the driver attribute information, to obtain an evaluation result of the risk cognitive ability, so as to monitor the driver's risk cognitive ability.

8. The method according to claim 7, wherein The method further includes: the human-machine co-driving control module obtains the matching result of the current driving risk scenario and the evaluation result of the risk cognitive ability; The human-machine co-driving control module evaluates the driver's takeover ability based on the pre-constructed fuzzy rule model according to the matching result of the current driving risk scenario and the evaluation result of the risk cognitive ability, to obtain an evaluation result of the driver's takeover ability; Perform vehicle control right allocation according to the evaluation result of the driver's takeover ability to achieve human-machine collaborative control.

9. The method according to claim 7, characterized in that Based on a pre-trained neural network model, the scenario matching module matches the ego-vehicle motion information, the vehicle surrounding environment information, and the roadside unit information with the map information and the risk scenario library information to obtain the matching result of the current driving risk scenario, which specifically includes the following steps: The scenario matching module performs multi-modal feature extraction on the ego-vehicle motion information, the vehicle surrounding environment information, and the roadside unit information; Based on the neural network, a predefined risk scenario constructed from the map information and the risk scenario library information is matched with the multi-modal features to obtain the matching result of the current driving risk scenario; The multi-modal features include pedestrian-related features, vehicle-related features, road-related features, and environment-related features; The function expression of the matching result of the current driving risk scenario is as follows: ； Wherein, is the matching result of the own vehicle driving risk scenario at time t; are pedestrian-related features; are vehicle-related features; are road-related features; are environment-related features; is the mapping mechanism between multi-modal features and predefined risk scenarios.

10. The method according to claim 7, wherein The function expression of the cognitive evaluation model is: ； wherein, is the facial image sequence of the driver; is the electroencephalogram signal sequence of the driver; is the eye movement signal sequence of the driver; DS is the driver state recognition result at time t; DA is the driver attribute information; is the monitoring result of the driver's risk perception ability at time t; P(.) is the conditional probability distribution; l i is the driver risk perception level; is the encoder; is the decoder; is the high-order feature vector generated by the encoder.

Images