Multimodal large model driving risk judgment method, system, medium and program product

Abstract

A multimodal large-scale model driving risk judgment method, system, medium and program product, in which the risk level of collision risk is determined; when it is the first level, the first preset alarm mode is turned on; when it is the second level, real-time video is obtained; the real-time video and vehicle bus data are input into the multimodal first-order large model to obtain the first result; the first result and the vehicle external environment video are input into the multimodal second-order large model to obtain the second result; it is determined whether the corrected risk probability is greater than the preset risk probability; if it is greater, it is determined whether it is in the preset risk type table; if not, the first preset alarm mode is turned on; if it is, the vehicle is controlled to decelerate at a preset acceleration and the second preset alarm mode is turned on; if it is not greater than, the first preset alarm mode is turned on; when it is the third level, the vehicle is controlled to drive according to the preset avoidance plan. The present application improves the accuracy of driving risk judgment, thereby improving driving safety.

Description

Technical Field

[0001] The present application belongs to the field of position control of land vehicles, and in particular, relates to a multi-modal large-model driving risk judgment method, system, medium and program product. Background Art

[0002] In the field of driving risk assessment, the development of technology has been continuously improving. Traditional risk assessment methods rely on data collected by various sensors installed on the vehicle, and use these data to make simple threshold judgments or rule-based judgments. With the development of technology, especially the rise of machine learning and artificial intelligence technology, more complex judgment models have been introduced into this field, making risk prediction and assessment more accurate.

[0003] In related technologies, when the judgment model determines that there is a risk of collision based on the collected data, it will actively reduce the vehicle's speed, or directly take over the steering wheel control to control the vehicle to avoid collision, or directly brake the vehicle to ensure driving safety.

[0004] However, such direct intervention in vehicle control may occur in an instant. If the vehicle speed is suddenly reduced or the driving direction is suddenly changed, the occupants may be frightened and the driver may make actions that affect driving safety, such as swerving, braking suddenly or accidentally stepping on the accelerator, which reduces the driving safety of the car. Or the driver could have controlled the vehicle to avoid a collision, but the steering wheel has been controlled by the system, making it difficult for the driver to control the vehicle's driving direction, thus reducing the driving safety of the car. Summary of the invention

[0005] The present application provides a multimodal large-model driving risk judgment method, system, medium and program product, which are used to combine the driver's real-time status to improve the accuracy of driving risk judgment, assist the driver in avoiding risks, and thus improve driving safety.

[0006] In a first aspect, the present application provides a multi-modal large model driving risk judgment method, in which, when it is determined that a vehicle has a collision risk, a risk level of the collision risk is determined, the risk level includes a first level, a second level and a third level, and the collision probability of the first level, the second level and the third level increases in sequence;

[0007] When the risk level is determined to be the first level, controlling the vehicle computer system of the vehicle to start a first preset alarm mode;

[0008] If the risk level is determined to be the second level, a real-time video of the driver is obtained through a preset camera in the vehicle;

[0009] Inputting real-time video and vehicle bus data into a multimodal first-order large model to obtain a first result, the first result including a driver risk type and a driver risk probability;

[0010] Inputting the first result and the vehicle exterior environment video sent by the vehicle exterior camera into the multimodal second-order large model to obtain a second result, wherein the second result includes a modified risk type and a modified risk probability;

[0011] Determine whether the revised risk probability is greater than the preset risk probability;

[0012] If it is greater than the preset risk probability, determine whether the revised risk type is in the preset risk type table;

[0013] If it is not in the preset risk type table, executing the step of controlling the vehicle computer system to start the first preset alarm mode;

[0014] If it is in the preset risk type table, the vehicle is controlled to decelerate at a preset acceleration and the vehicle system is controlled to activate a second preset alarm mode, and the number of functions activated in the second preset alarm mode is greater than the number of functions activated in the first preset alarm mode;

[0015] If the risk probability is not greater than the preset risk probability, executing the step of controlling the vehicle's vehicle computer system to start a first preset alarm mode;

[0016] When the risk level is determined to be the third level, the vehicle is controlled to travel according to a preset avoidance plan.

[0017] By adopting the above technical solution, the collision risk is divided into three levels. Different response strategies are adopted according to the degree of risk, which can respond to different situations more accurately and avoid overreaction or underreaction. At the second level of risk, the analysis of multimodal large models is introduced. This method combines the driver's status in the car and the external environment information to assess the risk more comprehensively. Through the two-stage large model analysis, the system can more accurately judge the risk type and probability, so as to take more targeted measures. At the highest risk level, the driver may not have time to make evasive actions. The system will automatically execute the preset avoidance plan to increase the success rate of the vehicle in avoiding collision risks. This technical solution combines the real-time status of the driver, improves the accuracy of driving risk judgment, assists the driver in avoiding risks, and thus improves driving safety.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, when it is determined that the vehicle has a collision risk, determining a risk level of the collision risk specifically includes:

[0019] Obtain real-time monitoring data from collision sensors outside the vehicle, and calculate the real-time shortest distance between each potential collision object and the vehicle based on the real-time monitoring data;

[0020] When the real-time shortest distance is not greater than the preset warning distance, it is determined that the vehicle is at risk of collision;

[0021] Determine the risk level of the potential collision object corresponding to the real-time shortest distance not greater than the first preset collision distance as the third level;

[0022] Determine the risk level of the potential collision object corresponding to the real-time shortest distance greater than the first preset collision distance and not greater than the second preset collision distance as the second level;

[0023] The risk level of the potential collision object corresponding to the real-time shortest distance that is greater than the second preset collision distance and not greater than the preset warning distance is determined as the first level.

[0024] By adopting the above technical solution, the system monitors the collision sensor data outside the vehicle in real time and calculates the shortest distance between the potential collision object and the vehicle, which can improve the probability and efficiency of discovering potential dangers. The risk level is corresponded to the specific distance range, such as judging the situation not greater than the first preset collision distance as the highest risk level. This clear classification standard makes the risk assessment more objective and consistent. This distance-based grading method also takes into account the difference in reaction time under different speeds and road conditions, and the preset distance threshold can be adjusted according to actual conditions. It can improve the accuracy of identifying potential dangers and take corresponding measures according to the urgency of the danger, thereby improving safety while reducing the impact of excessive intervention on normal driving.

[0025] In combination with some embodiments of the first aspect, in some embodiments, after determining the risk level of the potential collision object corresponding to the real-time shortest distance greater than the second preset collision distance and not greater than the preset warning distance as the first level, the method further includes:

[0026] When the real-time shortest distance is greater than the preset warning distance, obtain the real-time monitoring video of each potential collision object;

[0027] The direction of the turn signal is the direction close to the vehicle based on the real-time monitoring video to determine whether there is a potential collision object;

[0028] If it exists, the risk level of the potential collision object is determined to be the first level;

[0029] If not, determine whether there is a potential collision object moving in the direction approaching the vehicle based on the real-time monitoring video;

[0030] If it exists, the risk level of the potential collision object is determined to be the first level;

[0031] If not, it is determined that the vehicle does not have a collision risk.

[0032] By adopting the above technical solution, the system continues to monitor and analyze potential collision objects that are beyond the preset warning distance, and can detect and issue warnings in time when they approach the vehicle, which can reduce the risk of potential collision objects that exceed the preset warning distance. By analyzing the turn signal status and driving direction of potential collision objects, the system can predict their future movement trends and identify potential risks in advance. When it is confirmed that all potential collision objects do not pose a threat, the system determines that there is no collision risk, reducing the probability of collision.

[0033] In conjunction with some embodiments of the first aspect, in some embodiments, real-time video and vehicle bus data are input into a multimodal first-order large model to obtain a first result, which specifically includes:

[0034] Obtain historical vehicle bus data, historical driver behavior videos, and historical vehicle risk records;

[0035] Using historical vehicle bus data, historical driver behavior videos, and historical vehicle risk records as training sets, the preset artificial intelligence big model is trained to obtain a multimodal first-order big model;

[0036] The real-time video and vehicle bus data are input into the multimodal first-order large model, and a first result is output according to a preset first template. The first result includes the driver risk type and the driver risk probability.

[0037] By adopting the above technical solutions, the system uses historical vehicle bus data, historical driver behavior videos and historical vehicle risk records as training sets, which improves the adaptability of the large model to risk judgment and makes the output of the large model more consistent with the driver's driving habits. By training the preset artificial intelligence large model, the system can continuously optimize and update its risk judgment ability to adapt to different driving scenarios and risk patterns. By inputting real-time video and vehicle bus data into the model at the same time, a comprehensive analysis of the driver's status and vehicle status is achieved. This multimodal analysis method can capture risk factors that may be overlooked by a single data source, thereby improving the accuracy of assessing driving risks.

[0038] In combination with some embodiments of the first aspect, in some embodiments, the first result and the vehicle exterior environment video sent by the vehicle exterior camera are input into a multimodal second-order large model to obtain a second result, which specifically includes:

[0039] Obtain historical vehicle exterior environment videos;

[0040] Using historical vehicle exterior environment videos and historical vehicle risk records as training sets, the preset artificial intelligence large model is trained to obtain a multimodal second-order large model;

[0041] The first result and the video of the outside vehicle environment sent by the outside vehicle camera are input into the multimodal second-order large model, and the second result is output according to the preset second template. The second result includes a modified risk type and a modified risk probability.

[0042] By adopting the above technical solutions, the system uses historical vehicle external environment videos and historical vehicle risk records as training sets, enabling the model to learn various complex road environments and corresponding risk patterns. By training the preset artificial intelligence large model, the system can continuously optimize its understanding of the external environment and risk assessment capabilities, adapt to different road conditions and weather conditions, and improve the accuracy of risk type and risk probability estimation.

[0043] In combination with some embodiments of the first aspect, in some embodiments, when it is determined that the risk level is the third level, controlling the vehicle to travel according to a preset avoidance plan specifically includes:

[0044] The default avoidance plan is:

[0045] Determine whether there is a potential collision object in the opposite direction of the vehicle's travel based on real-time monitoring video;

[0046] If there is no potential collision object, the vehicle is controlled to slow down and the collision risk is kept at the second level;

[0047] If there is a potential collision object, determine whether there is a potential collision object on the left or right side of the vehicle based on the real-time monitoring video;

[0048] If there is no potential collision object, the vehicle is controlled to change to any lane perpendicular to the driving direction;

[0049] If there is a potential collision object, the vehicle is controlled to change its direction so that the driver is not within the range of the frontal collision.

[0050] By adopting the above technical solution, the system judges the surrounding environment of the vehicle based on the real-time monitoring video, determines whether there are potential collision objects, and provides data reference for the avoidance direction of subsequent vehicles. If there are no potential collision objects, the system will give priority to deceleration and maintain the second-level risk state, which improves safety while reducing the probability of new risks caused by overreaction. If the deceleration is not enough to avoid the risk, the system will consider changing lanes to the left or right, which provides more avoidance options. In extreme cases, the system will control the vehicle to change the direction of travel so that the driver can avoid the frontal collision range. Although this measure may bring certain risks, it can protect the driver's safety to the greatest extent in an emergency. Improved response capabilities in high-risk situations and improved driver safety.

[0051] In conjunction with some embodiments of the first aspect, in some embodiments, controlling the vehicle to change the driving direction so that the driver is not within the frontal collision range specifically includes:

[0052] When it is determined that there is a potential collision object on the left or right side of the vehicle and no potential collision object on the other side, the frontal collision range is calculated so that the driver is not within the frontal collision range. The frontal collision range is the contact area when the vehicle collides with the potential collision object in front.

[0053] By adopting the above technical solution, the system will judge the situation on the left and right sides of the vehicle. Only when there is a potential collision object on one side but not on the other side will the system control the vehicle to change its driving direction so that the driver is not within the frontal collision range, thereby improving the safety of evasive action. By calculating the frontal collision range, the system can determine the area that needs to be avoided. It can better protect the driver and reduce the secondary risks that may be caused by evasive actions.

[0054] In a second aspect, an embodiment of the present application provides a multimodal large-model driving risk judgment system, which includes: one or more processors and a memory; the memory is coupled to one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call computer instructions to enable the system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0055] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a system, causes the system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0056] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a system, the system executes the method described in any possible implementation manner in the first aspect.

[0057] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0058] 1. The present application provides a multimodal large model driving risk judgment method, which divides the collision risk into three levels, and adopts different response strategies according to the degree of risk, so as to respond to different situations more accurately and avoid overreaction or underreaction. At the second level of risk, the analysis of the multimodal large model is introduced. This method combines the driver's status in the car and the environmental information outside the car to assess the risk more comprehensively. Through the two-stage large model analysis, the system can more accurately judge the risk type and probability, so as to take more targeted measures. At the highest risk level, the driver may not have time to make evasive operations, and the system will automatically execute the preset avoidance plan to increase the success rate of the vehicle in avoiding collision risks. This technical solution combines the real-time status of the driver, improves the accuracy of driving risk judgment, assists the driver in avoiding risks, and thus improves driving safety.

[0059] 2. The present application provides a multimodal large-model driving risk judgment method. The system monitors the collision sensor data outside the vehicle in real time and calculates the shortest distance between the potential collision object and the vehicle, which can improve the probability and efficiency of discovering potential dangers. The risk level is corresponded to a specific distance range. For example, the situation that is not greater than the first preset collision distance is judged as the highest risk level. This clear classification standard makes the risk assessment more objective and consistent. This distance-based grading method also takes into account the difference in reaction time under different speeds and road conditions. The preset distance threshold can be adjusted according to actual conditions. It can improve the accuracy of identifying potential dangers and take corresponding measures according to the urgency of the danger, thereby improving safety while reducing the impact of excessive intervention on normal driving.

[0060] 3. The present application provides a multimodal large-model driving risk judgment method. The system judges the vehicle's surrounding environment based on real-time monitoring video to determine whether there are potential collision objects, and provides data reference for the avoidance direction of subsequent vehicles. If there are no potential collision objects, the system will give priority to slowing down and maintaining the second-level risk state. While improving safety, it also reduces the probability of new risks caused by overreaction. If the deceleration is not enough to avoid the risk, the system will consider changing lanes to the left or right, which provides more avoidance options. In extreme cases, the system will control the vehicle to change the direction of travel so that the driver can avoid the frontal collision range. Although this measure may bring certain risks, it can protect the driver's safety to the greatest extent in an emergency. Improved response capabilities in high-risk situations and improved driver safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flow chart of a multi-modal large model driving risk judgment method in an embodiment of the present application.

[0062] Figure 2It is a flow chart of a collision risk judgment method in an embodiment of the present application.

[0063] Figure 3 It is a schematic diagram of the physical device structure of a multi-modal large model driving risk judgment system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0065] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0066] Combine the following Figure 1 , a multi-modal large model driving risk judgment method in an embodiment of the present application is described:

[0067] See also Figure 1 , which is a flow chart of a multi-modal large-model driving risk judgment method in an embodiment of the present application.

[0068] S101. When it is determined that the vehicle has a collision risk, determine a risk level of the collision risk;

[0069] When it is determined that the vehicle has a collision risk, a risk level of the collision risk is determined, and the risk levels include a first level, a second level, and a third level, and the collision probability of the first level, the second level, and the third level increases in sequence. How to determine that the vehicle has a collision risk and the specific classification rules of the risk level will be described below in conjunction with another embodiment, and will not be repeated here.

[0070] S102, when it is determined that the risk level is the first level, controlling the vehicle computer system of the vehicle to start a first preset alarm mode;

[0071] When the system determines that the risk level is level 1, which is the lowest level of risk, the system will control the vehicle's on-board system to turn on the first preset alarm mode. This mode is to alert the driver to potential risks, but no emergency measures are required.

[0072] The first preset alarm mode may include multiple reminder methods to ensure that the driver can be aware of potential risks in a timely manner. For example, the system may display a warning message on the vehicle's central control screen to remind the driver that there may be obstacles or other risk factors ahead. At the same time, the system may emit a slight warning sound through the vehicle's speakers to attract the driver's attention, which is not limited here.

[0073] In order to avoid excessive interference to the driver, the reminder intensity of the first preset alarm mode should be moderate. The system can dynamically adjust the frequency and intensity of warnings based on factors such as vehicle speed and road type. For example, on highways, the system may increase the frequency of warnings; while on urban roads, the system may reduce the intensity of warnings to avoid affecting the driver's attention in complex traffic environments.

[0074] In addition, the system can integrate some auxiliary functions in the first preset warning mode. For example, the system can automatically turn on the headlights or fog lights to improve the visibility of the vehicle; or automatically adjust the cruising speed of the vehicle to keep a safer distance from the vehicle in front. These auxiliary functions can improve driving safety without interfering with the driver.

[0075] The system can also record each time the first preset alarm mode is triggered, including information such as time, location, and triggering reason. This data can be used for subsequent risk analysis and system optimization to make the alarm mode more intelligent and personalized. For example, if a driver often triggers the alarm on a specific road section, the system can issue a reminder in advance the next time the driver approaches the road section.

[0076] S103. When the risk level is determined to be the second level, obtaining a real-time video of the driver through a preset camera in the vehicle;

[0077] When the system determines that the risk level is the second level, that is, the medium level of risk, the system will start a more in-depth monitoring mechanism. The core of this step is to obtain real-time video of the driver through the preset camera inside the vehicle for a more detailed risk assessment.

[0078] The preset cameras in the vehicle are usually installed in appropriate locations in the cockpit, such as near the steering wheel column or rearview mirror, to ensure that the driver's facial expressions, eye state, head posture and other key information can be clearly captured. The system can use high-resolution, wide dynamic range cameras to adapt to shooting needs under different lighting conditions.

[0079] After acquiring real-time video, the system can analyze the driver's status in real time. For example, the system can detect whether the driver has symptoms of fatigue, such as frequent blinking, yawning, or head drooping. The system can also identify whether the driver has distracted behavior, such as not looking at the road ahead for a long time, frequently checking the phone, etc. This information is crucial for evaluating the driver's reaction ability and alertness.

[0080] To protect the driver's privacy, the system can take strict data processing measures. For example, real-time video can be processed and analyzed locally instead of being transmitted to the cloud. The system can save only necessary feature data without storing the original video. In addition, the system can set up data encryption and access control mechanisms to ensure that only authorized personnel can access this sensitive information.

[0081] The system can also combine the driver's real-time status with the vehicle's operating data for comprehensive analysis. For example, if driver fatigue is detected and the vehicle makes frequent small steering corrections, the system may determine that there is a higher risk, thereby triggering a stronger warning or taking more active intervention measures.

[0082] In addition, the system can build personalized behavior models based on the characteristics of different drivers. By collecting and analyzing data over a long period of time, the system can learn the normal behavior patterns of each driver and thus more accurately identify abnormal conditions. This personalized analysis can improve the accuracy and reliability of the system, reduce false alarms, and increase the detection rate of truly dangerous situations.

[0083] S104. When the risk level is determined to be the third level, controlling the vehicle to travel according to a preset avoidance plan;

[0084] When the risk level is determined to be the third level, the vehicle is controlled to drive according to the preset avoidance plan, which is:

[0085] Determine whether there is a potential collision object in the opposite direction of the vehicle's travel based on real-time monitoring video;

[0086] If there is no potential collision object, the vehicle is controlled to slow down and the collision risk is kept at the second level;

[0087] If there is a potential collision object, determine whether there is a potential collision object on the left or right side of the vehicle based on the real-time monitoring video;

[0088] If there is no potential collision object, the vehicle is controlled to change to any lane perpendicular to the driving direction;

[0089] If there is a potential collision object, then the frontal collision range is calculated so that the driver is not within the frontal collision range, provided that it is determined that there is a potential collision object on the left or right side of the vehicle and there is no potential collision object on the other side. The frontal collision range is the contact area when the vehicle collides with the potential collision object in front.

[0090] When the system determines that the risk level reaches the highest third level, it means that the vehicle is facing an imminent risk of collision. In this case, the system will immediately initiate a preset avoidance plan to minimize the risk of collision or mitigate the consequences of a collision.

[0091] The execution of the preset avoidance plan is a dynamic decision-making process. First, the system analyzes the environment around the vehicle based on real-time surveillance video. The system prioritizes whether there are potential collision objects in the opposite direction of the vehicle's travel. If there are no obstacles in the opposite direction, the system will control the vehicle to slow down and reduce the risk level to the second level. This approach can effectively avoid the risk of head-on collision and buy time for subsequent decision-making and operations.

[0092] If there is a potential collision object in the opposite direction, the system will further analyze the situation on the left and right sides of the vehicle. If there is no potential collision object on one side, the system will control the vehicle to change lanes in that direction. This lane change operation requires the system to accurately control the steering angle and speed to ensure that the lane change is completed safely without causing new risks.

[0093] In the most complex situation, when there are potential collision objects all around the vehicle, the system will adopt a more complex avoidance strategy. The system will calculate the frontal collision range, that is, the area where the vehicle may come into contact with the potential collision object in front. Based on this calculation, the system will control the vehicle to make fine adjustments to keep the driver's position away from the frontal collision range as much as possible. This fine adjustment may include slight steering, deceleration, or a combination of both, with the aim of protecting the driver's safety to the greatest extent possible.

[0094] While executing the evasive maneuver, the system will also initiate a series of auxiliary measures. For example, the system can automatically tighten the seat belts and pre-inflate the airbags to better protect the occupants in the event of a collision. The system can also automatically turn on the hazard warning lights and send warning signals to surrounding vehicles.

[0095] In addition, the system can optimize the avoidance plan according to the specific model and performance characteristics of the vehicle. For example, for an SUV with a higher center of gravity, the system will be more cautious when performing steering operations to avoid the risk of rollover; while for a sports car with better handling performance, the system may adopt a more flexible avoidance strategy.

[0096] The system can also continuously evaluate the risk situation during the execution of the avoidance plan. If it finds that the initial avoidance plan is not effective in reducing the risk, the system can adjust the strategy in real time, such as switching from lane change to emergency braking.

[0097] S105, obtaining historical vehicle bus data, historical driver behavior videos, and historical vehicle risk records;

[0098] This step is followed by step S103 , that is, this step and subsequent steps are executed only when step S103 is executed.

[0099] In this step, the system begins to collect and organize data for training the multimodal first-order large model. This data includes three main parts: historical vehicle bus data, historical driver behavior videos, and historical vehicle risk records. The comprehensiveness and diversity of this data are crucial for training a high-quality model.

[0100] Historical vehicle bus data contains various parameter information during vehicle operation, such as vehicle speed, acceleration, steering angle, braking force, etc. The system can continuously collect this data through the vehicle's CAN bus or other data interfaces. To ensure the representativeness of the data, the system should collect data at different times and under different road conditions, including data under normal driving and abnormal conditions.

[0101] Historical driver behavior videos are recorded by in-car cameras. These video data can capture detailed information such as the driver's facial expressions, eye movements, head posture, etc. When collecting these videos, the system needs to consider privacy protection issues, and can use methods such as real-time anonymization or saving only key feature points.

[0102] Historical vehicle risk records include various risk events that the vehicle has encountered, such as emergency braking, emergency avoidance, minor collisions, etc. These records include not only the event itself, but also the contextual information before and after the event, such as the road conditions and weather conditions at the time.

[0103] S106, using historical vehicle bus data, historical driver behavior videos, and historical vehicle risk records as training sets, training a preset artificial intelligence large model to obtain a multimodal first-order large model;

[0104] In this step, the system uses the data collected in the previous step to train the preset artificial intelligence model, and finally obtains the multimodal first-order large model. This process involves complex machine learning algorithms and technologies, aiming to create a powerful model that can comprehensively analyze multiple data sources.

[0105] First, the system needs to preprocess the collected data. This includes steps such as data cleaning, standardization, and feature extraction. For vehicle bus data, the system may need to perform time series analysis to extract meaningful statistical features. For driver behavior videos, the system may use computer vision technology to extract key visual features such as eye movement tracks, head posture, etc. For risk records, the system may need to perform text classification and event classification.

[0106] The preset AI model is usually a complex neural network structure, which may contain multiple sub-networks to process different types of input data. For example, a convolutional neural network (CNN) can be used to process video data, a recurrent neural network (RNN) or a long short-term memory network (LSTM) can be used to process time series data, and a fully connected layer can be used to integrate various features.

[0107] During the training process, the system can use a multi-task learning method to optimize multiple objective functions at the same time, such as risk prediction accuracy, driver status recognition accuracy, etc. This method enables the model to share underlying features when processing multiple tasks and improve the generalization ability of the model.

[0108] In order to improve training efficiency and model performance, the system can use advanced optimization algorithms, such as Adam optimizer or dynamic learning rate adjustment strategy. At the same time, the system can use techniques, such as batch normalization, dropout, etc., to improve the stability and generalization ability of the model.

[0109] During the training process, the system should implement a strict cross-validation mechanism to prevent overfitting. K-fold cross-validation or holdout method can be used to evaluate the performance of the model. At the same time, the system can adopt an early stopping strategy to stop training when the performance of the validation set no longer improves.

[0110] After training is complete, the system needs to conduct a comprehensive evaluation and testing of the model. This includes using an independent test set to evaluate the generalization ability of the model, as well as real-car testing in a simulated environment or a controlled real environment. The system can also use interpretability techniques, such as SHAP values ​​or LIME, to understand the model's decision-making process and ensure that the model's decisions meet logical and safety standards.

[0111] S107, inputting the real-time video and vehicle bus data into the multimodal first-order large model, and outputting a first result according to a preset first template;

[0112] The system inputs real-time video and vehicle bus data into a multimodal first-order large model, and outputs a first result according to a preset first template. The first result includes a driver risk type and a driver risk probability.

[0113] In this step, the system inputs the real-time collected data into the multimodal first-order large model trained in the previous step, and outputs the results according to the preset first template. This process involves multiple links such as real-time data processing, model reasoning, and result formatting.

[0114] First, the system needs to collect and preprocess input data in real time. For real-time video, the system may need to perform image preprocessing, such as cropping, scaling, color adjustment, etc., to meet the input requirements of the model. At the same time, the system may need to use target detection or posture estimation algorithms to extract the key features of the driver. For vehicle bus data, the system needs to read and normalize it in real time, and may also need to slide the time window to capture short-term data change trends.

[0115] After data preprocessing is completed, the system inputs this data into a multimodal first-order large model. The reasoning process of the model needs to be completed in a very short time to ensure that the system can respond to potential risks in a timely manner. To this end, the system may need to adopt model compression techniques such as knowledge distillation or quantization to reduce the computational complexity of the model and increase the reasoning speed. At the same time, the system can use GPUs or dedicated AI chips to accelerate the calculation process.

[0116] The original results output by the model need to be formatted according to the preset first template. This template defines the structure and content of the output results, which usually includes driver risk type and driver risk probability. Driver risk types may include multiple categories such as fatigue driving, distracted driving, and aggressive driving. Each risk type will correspond to a risk probability, indicating the possibility that the current state belongs to this risk type. The preset first template can be a pre-stored template, which is not limited here.

[0117] S108, inputting the first result and the vehicle exterior environment video sent by the vehicle exterior camera into the multimodal second-order large model to obtain a second result;

[0118] The system inputs the first result and the video of the exterior environment sent by the exterior camera into the multimodal second-order large model to obtain a second result. The system uses the large model training method of step S107 to obtain the historical video of the exterior environment of the vehicle. The historical video of the exterior environment and the historical vehicle risk records are used as training sets to train the preset artificial intelligence large model to obtain the multimodal second-order large model. The system inputs the first result and the video of the exterior environment sent by the exterior camera into the multimodal second-order large model, and outputs the second result according to the preset second template. The second result includes a corrected risk type and a corrected risk probability.

[0119] The system combines the first result obtained in the previous step with the environmental video captured by the external camera and inputs it into the multimodal second-order large model to obtain a more comprehensive risk assessment result. This process involves the fusion of multi-source data, the reasoning of complex models and the comprehensive analysis of results.

[0120] First, the system needs to preprocess the video of the vehicle's external environment. This may include operations such as image enhancement, object detection, and semantic segmentation. For example, the system can use object detection algorithms to identify key elements in the video, such as other vehicles, pedestrians, and traffic signs. At the same time, the system can use semantic segmentation algorithms to understand information such as road structure and lane lines. These preprocessing operations can extract visual features that are important for risk assessment.

[0121] The system combines the preprocessed video data of the vehicle's exterior environment with the results of the first step (including driver risk type and risk probability) as the input to the multimodal second-order large model. The training process of this model is similar to that of the first-order model, and it is also trained using historical data. The difference is that the training data of this model also includes historical videos of the vehicle's exterior environment, allowing the model to learn the complex relationship between the internal state (driver state) and the external environment.

[0122] The architecture of the multimodal second-order large model may be more complex and may contain multiple sub-networks to process different types of input data. For example, a 3D convolutional neural network can be used to process temporal video data, and an attention mechanism can be used to integrate the output of the first-order model and video features. This complex architecture enables the model to capture more detailed and comprehensive risk features.

[0123] The output of the model, the second result, usually includes the corrected risk type and the corrected risk probability. This result is a correction and supplement to the first-order model result, taking into account the impact of external environmental factors on the risk. For example, even if the driver is in good condition, if there is a potential danger in the external environment (such as icy road ahead, poor visibility, etc.), the system may give a higher risk assessment.

[0124] The system can design a dynamic weighting mechanism to adjust the weights of internal state and external environment in risk assessment according to different scenarios. For example, on a highway, the external environment may receive a higher weight; while on a complex urban road, the driver's state may be more important.

[0125] In addition, the system can introduce predictive analysis in this step. Based on the current driver status and external environment, the system can predict the risk change trend in the short term in the future and provide more information for subsequent decision-making.

[0126] The system formats the second result according to the preset second template. This template may include more detailed information, such as specific environmental risk factors, risk level change trends, etc. These formatted results will provide a basis for the next step of risk judgment and decision-making. Of course, the preset second format is different from the preset first format.

[0127] S109, determining whether the revised risk probability is greater than the preset risk probability;

[0128] The system needs to evaluate the modified risk probability obtained in the previous step to determine whether it exceeds the preset risk probability. The preset risk probability can be a manually set value and is not limited here.

[0129] S110, determining whether the revised risk type is in the preset risk type table;

[0130] If it is greater than the preset risk probability, determine whether the modified risk type is in the preset risk type table.

[0131] If it is not greater than the preset risk probability, the step of controlling the vehicle's on-board system to turn on the first preset alarm mode is executed.

[0132] The system needs to compare the modified risk type obtained in the previous step with the pre-defined risk type table to determine whether the currently identified risk belongs to the system preset risk type that requires special attention or processing.

[0133] S111, controlling the vehicle to decelerate at a preset acceleration and controlling the vehicle computer system to start a second preset alarm mode.

[0134] If it is in the preset risk type table, the vehicle is controlled to decelerate at the preset acceleration and the vehicle system is controlled to turn on the second preset alarm mode. The number of functions turned on in the second preset alarm mode is greater than the number of functions turned on in the first preset alarm mode.

[0135] If it is not in the preset risk type table, the step of controlling the vehicle's on-board system to start the first preset alarm mode is executed.

[0136] In the above embodiment, the collision risk is divided into three levels, and different response strategies are adopted according to the degree of risk, which can respond to different situations more accurately and avoid overreaction or underreaction. At the second level of risk, the analysis of the multimodal large model is introduced. This method combines the driver's status in the car and the external environment information, and can more comprehensively assess the risk. Through the two-stage large model analysis, the system can more accurately judge the risk type and probability, so as to take more targeted measures. At the highest risk level, the driver may not have time to make evasive operations, and the system will automatically execute the preset avoidance plan to increase the success rate of the vehicle in avoiding collision risks. This technical solution combines the real-time status of the driver, improves the accuracy of driving risk judgment, assists the driver in avoiding risks, and thus improves driving safety.

[0137] In the first step of the above embodiment, the system determines the risk level of the collision risk when determining that the vehicle has a collision risk. The following describes how to implement this step using another embodiment. Figure 2 , a collision risk judgment method in an embodiment of the present application is described:

[0138] See also Figure 2 , which is a flow chart of a collision risk judgment method in an embodiment of the present application.

[0139] S201, obtaining real-time monitoring data of a collision sensor outside the vehicle, and calculating the real-time shortest distance between each potential collision object and the vehicle based on the real-time monitoring data;

[0140] The system obtains real-time monitoring data through collision sensors installed on the outside of the vehicle. These sensors may include but are not limited to ultrasonic sensors, radar sensors, lidar sensors, etc. The system continuously receives data from these sensors, including the location, speed, direction and other information of surrounding objects. Based on this data, the system uses an algorithm to calculate the real-time shortest distance between each potential collision object and the vehicle.

[0141] The system can use a variety of algorithms to calculate the shortest distance, such as the nearest neighbor algorithm, spherical coordinate system conversion algorithm, etc. In order to improve the calculation efficiency and accuracy, the system can use parallel computing methods to process data from multiple sensors at the same time. In addition, the system can use technologies such as Kalman filtering to fuse and optimize sensor data to reduce noise and improve the accuracy of distance calculation.

[0142] The system can also predict the location of potential collision objects based on the vehicle's speed and direction, thereby calculating the shortest distance in the future. This predictive calculation can provide the system with a longer reaction time and improve the accuracy of collision risk judgment.

[0143] S202: When the real-time shortest distance is not greater than the preset warning distance, it is determined that the vehicle has a collision risk;

[0144] The system compares the real-time shortest distance calculated in S201 with the pre-set warning distance. The warning distance is a threshold used to determine whether there is a potential collision risk. When the real-time shortest distance between any potential collision object and the vehicle is less than or equal to the warning distance, the system will determine that there is a collision risk.

[0145] The setting of the warning distance can take into account multiple factors, such as the current speed of the vehicle, road conditions, weather conditions, etc. The system can dynamically adjust the warning distance, such as increasing the warning distance at high speeds and reducing the warning distance at low speeds. In addition, the system can set different warning distances for different types of potential collision objects (such as pedestrians, bicycles, other vehicles, etc.).

[0146] The system can also introduce a time factor and calculate a time-to-distance ratio. For example, if the distance between a potential collision object and the vehicle is decreasing rapidly, the system may determine that there is a risk of collision even if the current distance is greater than the warning distance. This approach can improve the system's responsiveness to fast-approaching objects.

[0147] S203, determining the risk level of the potential collision object corresponding to the real-time shortest distance not greater than the first preset collision distance as the third level; determining the risk level of the potential collision object corresponding to the real-time shortest distance greater than the first preset collision distance and not greater than the second preset collision distance as the second level; determining the risk level of the potential collision object corresponding to the real-time shortest distance greater than the second preset collision distance and not greater than the preset warning distance as the first level;

[0148] The system divides the collision risk into three levels according to the real-time shortest distance between the potential collision object and the vehicle. This grading method allows the system to take different countermeasures for different degrees of risk.

[0149] The first preset collision distance and the second preset collision distance are two key thresholds used to classify risk levels. The system can dynamically adjust these thresholds based on factors such as vehicle performance and road conditions. For example, on slippery roads, the system can reduce these thresholds to provide a greater safety margin.

[0150] The system can also adjust the risk level by considering factors such as the type, size, and speed of the potential collision object. For example, for a potential collision object at the same distance, if it is a large vehicle, the system may increase its risk level; if it is a stationary object, the system may reduce its risk level.

[0151] S204, when the real-time shortest distance is greater than the preset warning distance, obtaining a real-time monitoring video of each potential collision object;

[0152] The system further analyzes potential collision objects that are farther than the preset warning distance. Although these objects do not currently pose a direct collision risk, the system still needs to continuously monitor their status to prevent possible risks.

[0153] The system obtains real-time monitoring videos of these potential collision objects through the on-board cameras. These cameras can be ordinary visible light cameras, infrared cameras, panoramic cameras, etc. The system can use image recognition algorithms to analyze the video stream in real time to identify and track each potential collision object.

[0154] The system can use multi-camera fusion technology to combine video data from different angles to build a three-dimensional model of potential collision objects. This method can more accurately determine the location, size and movement trajectory of potential collision objects.

[0155] In addition, the system can use deep learning algorithms, such as convolutional neural networks (CNN) or recurrent neural networks (RNN), to conduct a deeper analysis of video content. These algorithms can help the system identify the types of potential collision objects (such as vehicles, pedestrians, animals, etc.), predict their possible behavior patterns, and thus better assess potential risks.

[0156] S205, judging whether there is a potential collision object according to the real-time monitoring video, and the direction of the turn signal is close to the vehicle;

[0157] If so, the risk level of the potential collision object is determined to be the first level.

[0158] The system analyzes the real-time surveillance video, paying special attention to the turn signal status of potential collision objects. This step is mainly aimed at other vehicles, because turn signals usually indicate that the vehicle is about to change direction.

[0159] The system can use computer vision algorithms to identify and analyze turn signal signals in the video. This may include color recognition algorithms (detecting yellow or orange flashes), flash pattern analysis algorithms, etc. The system must not only identify the on state of the turn signal, but also determine whether the direction indicated by the turn signal is toward the vehicle.

[0160] In order to improve the accuracy of recognition, the system can combine multiple frames for analysis to reduce the possibility of misjudgment of a single frame. In addition, the system can use contextual information, such as lane line position, vehicle relative position, etc., to assist in determining steering intentions.

[0161] If the system determines that there is a potential collision object with its turn signal pointing in the direction close to the vehicle, even if the object is currently far away, the system will raise its risk level to level 1. This preventive judgment can help the system identify potential risks earlier and reserve more time for subsequent risk avoidance operations.

[0162] S206, judging whether there is a potential collision object moving in a direction close to the vehicle according to the real-time monitoring video;

[0163] If not, it is determined based on the real-time monitoring video whether there is a potential collision object traveling in a direction approaching the vehicle.

[0164] The system analyzes real-time surveillance video to determine the direction of travel of potential collision objects. This applies not only to other vehicles, but also to moving objects such as pedestrians and bicycles.

[0165] The system can use optical flow algorithms to analyze the motion trajectory of objects in the video. By comparing the pixel changes between consecutive frames, the system can calculate the motion vector of potential collision objects. The system can also use target tracking algorithms, such as Kalman filters or particle filters, to predict the future position and trajectory of potential collision objects.

[0166] To improve the accuracy of the judgment, the system can combine multiple sensor data. For example, combining radar data can more accurately measure the speed and direction of potential collision objects. The system can also use map data and GPS information, taking into account the road layout and the current position of the vehicle, to more accurately determine whether a potential collision object is approaching.

[0167] In addition, the system can use machine learning methods, such as recurrent neural networks (RNN) or long short-term memory networks (LSTM), to analyze the historical trajectories of potential collision objects and predict their possible future movement directions. This method can help the system better understand complex traffic scenarios and improve the accuracy of judgment.

[0168] S207, determining the risk level of the potential collision object as the first level;

[0169] If it exists, the risk level of the potential collision object will be determined as the first level. If the system determines that the potential collision object is indeed moving towards the vehicle, even if the object is currently far away, the system will raise its risk level to the first level.

[0170] S208: Determine whether there is no collision risk for the vehicle.

[0171] If not, it is determined that the vehicle does not have a collision risk. If the system does not find a potential collision risk through all the previous analyses, it will conclude that the vehicle currently does not have a collision risk. However, this does not mean that the system can stop monitoring.

[0172] The system will continue to conduct risk assessments, repeating the previous steps regularly. The assessment frequency can be adjusted dynamically based on factors such as vehicle speed and traffic density. For example, at high speeds or in complex road conditions, the system can increase the assessment frequency.

[0173] Even if there is no current risk of collision, the system will continue to monitor the surrounding environment. This includes continuing to collect and analyze sensor data and keep track of potential collision objects. The system can maintain a "watch list" to record objects that do not currently pose a risk but require continued attention.

[0174] The system can also use this “risk-free” time window to perform self-diagnosis and calibration, for example, checking the working status of each sensor, updating algorithm parameters, or synchronizing the latest risk assessment model with the cloud database.

[0175] In the above embodiment, the system monitors the collision sensor data outside the vehicle in real time and calculates the shortest distance between the potential collision object and the vehicle, which can improve the probability and efficiency of discovering potential dangers. The risk level is corresponded to a specific distance range, such as determining the situation that is not greater than the first preset collision distance as the highest risk level. This clear classification standard makes the risk assessment more objective and consistent. This distance-based grading method also takes into account the difference in reaction time under different speeds and road conditions, and the preset distance threshold can be adjusted according to actual conditions. It can improve the accuracy of identifying potential dangers and take corresponding measures according to the urgency of the danger, thereby improving safety while reducing the impact of excessive intervention on normal driving.

[0176] The following describes the system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , which is a schematic diagram of the physical device structure of a multi-modal large-model driving risk judgment system provided in an embodiment of the present application.

[0177] It should be noted that Figure 3 The structure of the system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0178] like Figure 3As shown, the system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in RAM 303. CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0179] The following components are connected to the I / O interface 305: an input section 306 including a camera, an infrared sensor, etc.; an output section 307 including a liquid crystal display (LCD) and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LaN (Local area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0180] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0181] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.

[0182] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0183] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the system described in the above embodiment; or may exist independently without being assembled into the system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of a system, the system implements the method provided in the above embodiment.

[0184] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0185] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0186] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0187] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A driving risk judgment method based on a hybrid multimodal large model, characterized in that: include: In the case where it is determined that the vehicle has a collision risk, a risk level of the collision risk is determined, the risk level includes a first level, a second level and a third level, and the collision probabilities of the first level, the second level and the third level increase in sequence, specifically including: Acquire real-time monitoring data of a collision sensor outside the vehicle, and calculate the real-time shortest distance between each potential collision object and the vehicle based on the real-time monitoring data; When the real-time shortest distance is not greater than the preset warning distance, determining that the vehicle is at risk of collision; Determining the risk level of the potential collision object corresponding to the real-time shortest distance not greater than the first preset collision distance as the third level; Determine the risk level of the potential collision object corresponding to the real-time shortest distance greater than the first preset collision distance and not greater than the second preset collision distance as the second level; Determine the risk level of the potential collision object corresponding to the real-time shortest distance greater than the second preset collision distance and not greater than the preset warning distance as the first level; When determining that the risk level is the first level, controlling the vehicle system of the vehicle to start a first preset alarm mode; When it is determined that the risk level is the second level, obtaining a real-time video of the driver through a preset camera in the vehicle; Inputting the real-time video and vehicle bus data into a multimodal first-order large model to obtain a first result, wherein the first result includes a driver risk type and a driver risk probability; Inputting the first result and the vehicle exterior environment video sent by the vehicle exterior camera into the multimodal second-order large model to obtain a second result, wherein the second result includes a modified risk type and a modified risk probability; Determining whether the modified risk probability is greater than the preset risk probability; If it is greater than the preset risk probability, determining whether the modified risk type is in the preset risk type table; If it is not in the preset risk type table, executing the step of controlling the vehicle system of the vehicle to start the first preset alarm mode; If in the preset risk type table, the vehicle is controlled to decelerate at a preset acceleration and the vehicle system is controlled to activate a second preset alarm mode, wherein the number of activated functions in the second preset alarm mode is greater than the number of activated functions in the first preset alarm mode; If the probability is not greater than the preset risk probability, executing the step of controlling the vehicle system to start a first preset alarm mode; When it is determined that the risk level is the third level, the vehicle is controlled to travel according to a preset avoidance plan.

2. The method according to claim 1, characterized in that After determining the risk level of the potential collision object corresponding to the real-time shortest distance greater than the second preset collision distance and not greater than the preset warning distance as the first level, the method further includes: When the real-time shortest distance is greater than the preset warning distance, obtaining a real-time monitoring video of each potential collision object; Determining whether there is a potential collision object based on the real-time monitoring video is in the direction of the turn signal close to the vehicle; If so, determining the risk level of the potential collision object to be the first level; If not, judging whether there is a potential collision object moving in a direction close to the vehicle based on the real-time monitoring video; If so, determining the risk level of the potential collision object to be the first level; If not, it is determined that the vehicle does not have the collision risk.

3. The method according to claim 1, characterized in that The step of inputting the real-time video and vehicle bus data into a multimodal first-order large model to obtain a first result specifically includes: Obtain historical vehicle bus data, historical driver behavior videos, and historical vehicle risk records; Using the historical vehicle bus data, the historical driver behavior video and the historical vehicle risk record as a training set, training the preset artificial intelligence large model to obtain the multimodal first-order large model; The real-time video and vehicle bus data are input into the multimodal first-order large model, and a first result is output according to a preset first template, wherein the first result includes a driver risk type and a driver risk probability.

4. The method according to claim 1, characterized in that: The step of inputting the first result and the vehicle exterior environment video sent by the vehicle exterior camera into the multimodal second-order large model to obtain the second result specifically includes: Obtaining historical vehicle exterior environment videos of the vehicle; The historical vehicle exterior environment video and the historical vehicle risk records are used as training sets to train the preset artificial intelligence large model to obtain the multimodal second-order large model; The first result and the video of the outside environment sent by the outside camera are input into the multimodal second-order large model, and a second result is output according to a preset second template, wherein the second result includes a modified risk type and a modified risk probability.

5. The method according to claim 2, characterized in that: When determining that the risk level is the third level, controlling the vehicle to travel according to a preset avoidance plan specifically includes: When it is determined that the risk level is the third level, the vehicle is controlled to travel according to a preset avoidance plan, and the preset avoidance plan is: Determine whether there is a potential collision object in the opposite direction of the vehicle's travel direction according to the real-time monitoring video; If the potential collision object does not exist, controlling the vehicle to decelerate and maintaining the collision risk at the second level; If the potential collision object exists, determining whether there is a potential collision object in the left or right direction of the vehicle according to the real-time monitoring video; If the potential collision object does not exist, controlling the vehicle to change to any lane perpendicular to the driving direction; If the potential collision object exists, the vehicle is controlled to change the driving direction so that the driver is not within the frontal collision range.

6. The method according to claim 5, characterized in that The controlling the vehicle to change the driving direction so that the driver is not within the frontal collision range specifically includes: When it is determined that the potential collision object exists on the left or right side of the vehicle and does not exist on the other side, the frontal collision range is calculated so that the driver is not within the frontal collision range. The frontal collision range is the contact area when the vehicle collides with the potential collision object in front.

7. A driving risk judgment system based on a hybrid multimodal large model, characterized in that: The system comprises: One or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the system to execute the method as described in any one of claims 1-6.

8. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a system, the system is caused to execute the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program product is run on a system, the system is caused to execute the method according to any one of claims 1 to 6.

Images