Man-machine collaborative automobile field hidden danger identification and auxiliary decision method and electronic equipment

Abstract

This invention discloses a human-machine collaborative method and electronic device for identifying and assisting in decision-making regarding on-site vehicle hazards. The method includes: acquiring a multimodal dataset of the vehicle; determining whether a collision has occurred based on the multimodal dataset; when a collision is confirmed, acquiring driving behavior data and determining the user's operational intent based on the driving behavior data; determining the collision time period, collision location, collision severity, collision action type, and number of collisions for each collision based on the multimodal dataset; determining the collision target type based on ultra-wideband radar data, video data, and acoustic data; determining collision event information; determining the collision accident type and collision liability information based on the collision event information; and generating an accident handling plan based on the collision severity, collision accident type, and collision liability information. This application can record the complete and real situation of the entire collision event from multiple dimensions to generate a suitable accident handling plan.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a human-machine collaborative method and electronic device for identifying on-site vehicle hazards and assisting in decision-making. Background Technology

[0002] The timeliness and accuracy of traffic accident handling directly affect the safety of drivers and passengers, the control of property damage, and the efficiency of restoring traffic order. Currently, vehicle accident identification and decision support have become core research directions in the field of vehicle safety. This not only requires the rapid identification of collision events but also the provision of accurate data support for subsequent rescue, liability determination, and insurance claims.

[0003] Existing methods for identifying potential hazards at vehicle sites mainly rely on one or more onboard sensors to collect collision-related data, and then combine machine learning, deep learning, and other algorithms to achieve preliminary identification and judgment of a single collision event.

[0004] However, traffic accidents are often complex. For example, a car may collide with multiple objects in succession, hitting a person first and then another vehicle. Existing methods have a single dimension of information collection, which can only identify whether a collision occurred, but cannot fully reconstruct the real scene of the entire collision accident, affecting the rationality of accident liability determination and handling plans. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides a human-machine collaborative method and electronic device for identifying and assisting in decision-making regarding on-site vehicle hazards. This application determines collision event information based on the collision time period, collision location, collision severity, collision action type, number of collisions, collision target type, and user's operational intent. It can record the complete and realistic situation of the entire collision event from multiple dimensions to generate appropriate accident handling plans. Furthermore, by determining whether a collision has occurred based on the sequence of abrupt change points, the time interval between abrupt change points, and multiple abrupt change point time series, it can determine whether a collision has occurred based on real physical collision laws and the order of sensor data collection, rather than initially inputting multimodal data into an artificial intelligence model that typically cannot interpret judgment rules, thus improving the accuracy of determining whether a collision has occurred.

[0006] To address the above problems, the present invention provides the following technical solution: In a first aspect, embodiments of this application provide a human-machine collaborative method for identifying and assisting decision-making regarding on-site hazards in automobiles, comprising: acquiring a multimodal dataset of automobiles, wherein the multimodal dataset includes video data, acoustic data, inertial measurement data, and ultra-wideband radar data; Determine whether a car collision has occurred based on the aforementioned multimodal dataset; Determining whether a car collision has occurred based on the multimodal dataset includes: Time series change point detection is performed on each type of data in the multimodal dataset to obtain multiple change point time series, including video data change point time series, acoustic data change point time series, inertial measurement data change point time series, and ultra-wideband radar data change point time series. Obtain the detection delay time information for each type of sensor; Based on the detection delay time information, determine the sequence of abrupt change points and the time interval between abrupt change points for multiple types of data when a car collision occurs; Whether a car collision has occurred is determined based on the sequence of mutation points, the time interval between mutation points, and the time series of multiple mutation points. When it is determined that the car has collided, the user's driving behavior data is acquired, and the user's operating intention is determined based on the driving behavior data; Based on the multimodal dataset, the collision time period, collision location, collision severity, collision action type, and number of collisions for each vehicle collision are determined. The collision action type includes active collisions and passive collisions. The collision target type is determined based on the ultra-wideband radar data, the video data, and the acoustic data. The collision target type includes people, animals, and inanimate objects. The collision event information is determined based on the collision occurrence time period, collision location, collision severity, collision action type, collision frequency, collision target type, and user's operation intention. Based on the collision event information, the collision accident type and collision liability information are determined; An accident handling plan is generated based on the severity of the collision, the type of the collision accident, and the collision liability information.

[0007] Optionally, determining whether a collision has occurred based on the sequence of mutation points, the time interval between mutation points, and the time series of the plurality of mutation points includes: The length of the detection time window is determined based on the time interval between mutation points in the aforementioned multiple types of data. The starting point of the detection time window is determined based on the time point corresponding to each mutation point in the multiple mutation point time series, and the detection time window is determined based on the starting point of the detection time window and the length of the detection time window. A sliding window approach is used to determine whether a collision has occurred based on multiple determined detection time windows and the sequence of abrupt change points, and to determine the time period during which the collision occurred.

[0008] Optionally, when it is determined that the vehicle has collided, acquiring the user's driving behavior data and determining the user's operational intent based on the driving behavior data includes: When it is determined that a collision has occurred, the user's driving behavior data is acquired and the time of the collision is determined. Multiple intent time windows are determined based on the collision occurrence time and multiple preset durations; Extract the first behavior feature vector from the data segment corresponding to each intent time window in the driving behavior data. The first behavior feature vector includes basic behavior feature parameters and rate of change feature parameters. The first row of feature vectors of each data segment of the intent time window is concatenated to obtain the second row of feature vectors. The second behavioral feature vector is input into the trained operation intent determination model to obtain the user's operation intent.

[0009] Optionally, determining the collision time period, collision location, collision severity, collision action type, and number of collisions for each vehicle collision based on the multimodal dataset includes: The driving scenario type and weather type are determined based on the video data in the multimodal dataset; Based on the driving scenario type and the weather type, determine the fusion feature vector of all class data in the multimodal dataset; Based on the fused feature vector, the collision time period, collision location, collision severity, collision action type, and number of collisions for each vehicle collision are determined.

[0010] Optionally, determining the fusion feature vector of all classes in the multimodal dataset based on the driving scenario type and the weather type includes: Extract the first feature vector of each data class in the multimodal dataset; Determine the fusion weight for each type of data corresponding to the driving scenario type and the weather type in the preset weight matrix; The fusion feature vector is determined based on the fusion weight of each type of data and the first feature vector.

[0011] Optionally, determining the collision accident type and collision liability information based on the collision event information includes: Based on the collision target type in the collision event information, a first-level category of the collision accident type is determined. The first-level category includes accidents between motor vehicles, accidents between motor vehicles and pedestrians, accidents between motor vehicles and animals, and accidents between motor vehicles and fixed objects. The second-level category of the collision accident type is determined based on the collision occurrence time period, collision location, collision severity, and collision number in the collision event information. The second-level category includes rear-end collision, frontal collision, side collision, scrape, reversing accident, and chain collision. The collision liability information is determined based on the collision action type in the collision event information and the user's operational intent.

[0012] Optionally, determining the collision responsibility information based on the collision action type in the collision event information and the user's operational intent includes: Traffic rules are determined based on the video data within the data segment during the time period in which the collision occurred. Determine whether the user has committed any wrongdoing based on the traffic rules and the user's operational intent; When a user commits an error, the pre-trained Bayesian causal graph is obtained, and the type of error is determined. The causal contribution of all the said wrongful behaviors to the occurrence of the collision is determined based on the Bayesian causal graph and the type of each said wrongful behavior. The collision liability information is determined based on the causal contribution.

[0013] Optionally, generating an accident handling plan based on the collision severity, the collision accident type, and the collision liability information includes: A property damage risk score is determined based on the severity of the collision. A personal injury risk score is determined based on the type of collision accident and the severity of the collision. A liability risk score is determined based on the collision liability information; The accident handling plan is generated based on the preset action space, the property loss risk score, the personal injury risk score, and the liability risk score.

[0014] Optionally, when it is determined that the car has not been involved in a collision, the collision probability of the car is determined based on the multimodal dataset; A collision warning is generated based on the collision probability of the vehicle.

[0015] Secondly, embodiments of this application provide an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the human-machine collaborative vehicle on-site hazard identification and auxiliary decision-making method as described in the first aspect.

[0016] This application provides a human-machine collaborative method and electronic device for identifying and assisting in decision-making regarding on-site vehicle hazards. This application determines collision event information based on the collision time period, collision location, collision severity, collision action type, number of collisions, collision target type, and user's operational intent. It can record the complete and realistic situation of the entire collision event from multiple dimensions to generate appropriate accident handling plans. Furthermore, by determining whether a collision has occurred based on the sequence of abrupt change points, the time interval between abrupt change points, and multiple abrupt change point time series, it can determine whether a collision has occurred based on real physical collision laws and the order of sensor data collection, rather than initially inputting multimodal data into an artificial intelligence model that typically cannot interpret the judgment rules, thus improving the accuracy of determining whether a collision has occurred. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards, as provided in this application embodiment.

[0018] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0019] Figure 3 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "multiple" means two or more, unless otherwise explicitly specified.

[0022] This application provides a human-machine collaborative method and electronic device for identifying and assisting in decision-making regarding on-site vehicle hazards. By determining collision event information based on the collision time period, collision location, collision severity, collision action type, number of collisions, collision target type, and user's operational intent, it can record the complete and realistic situation of the entire collision event from multiple dimensions to generate appropriate accident handling plans. Furthermore, by determining whether a collision has occurred based on the sequence of abrupt change points, the time interval between abrupt change points, and multiple abrupt change point time series, it can judge whether a collision has occurred based on the actual physical collision laws and the order of sensor data collection, rather than initially inputting multimodal data into an artificial intelligence model that typically cannot interpret the judgment rules, thus improving the accuracy of determining whether a collision has occurred.

[0023] The on-site hazards of automobiles in this application include situations where a collision has already occurred and situations where a collision is about to occur.

[0024] All parameters used in the calculations in this application have been normalized. For example, the Min-Max normalization algorithm is used to normalize all parameters used in the calculations.

[0025] This application relates to the manufacturing industry of new energy vehicle devices and parts, and also to the field of intelligent driving of new energy vehicles.

[0026] The following section will describe in detail, with reference to the accompanying drawings, the human-machine collaborative method for identifying on-site vehicle hazards and assisting in decision-making in this application.

[0027] Please see Figure 1 , Figure 1 This is a flowchart illustrating the human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards, as provided in this application embodiment. Figure 1 As shown, the human-machine collaborative method for identifying on-site vehicle hazards and assisting in decision-making includes steps S100 to S800.

[0028] Step S100: Obtain the multimodal dataset of the car.

[0029] The multimodal dataset includes video data, acoustic data, inertial measurement data, and ultra-wideband radar data.

[0030] Optionally, the multimodal dataset also includes infrared video data and collision sensor data.

[0031] Step S200: Determine whether a car collision has occurred based on the multimodal dataset.

[0032] Optionally, step S200 includes steps S210 to S250.

[0033] Step S210: Perform time series change point detection on each class of data in the multimodal dataset to obtain multiple time series of change points.

[0034] Among them, multiple time series of abrupt change points include time series of abrupt change points in video data, time series of abrupt change points in acoustic data, time series of abrupt change points in inertial measurement data, and time series of abrupt change points in ultra-wideband radar data.

[0035] Optionally, based on the noise characteristics of different types of data, time series change point detection is performed on the data according to the preset processing algorithm for each type of data to obtain multiple change point time series.

[0036] Optionally, a sliding window approach is used to extract multiple normalized time series features for each type of data within a sliding time window. Based on the weight matrix corresponding to that type of data, the multiple normalized time series features are weighted and fused to obtain a change point detection feature index. When the index exceeds a preset change point detection threshold, the middle time of the sliding time window is determined to be a change point.

[0037] Optionally, the duration of the sliding time window is a first preset duration.

[0038] For example, the first preset duration is 1 millisecond (ms), 5 ms, or 10 ms, etc.

[0039] For example, multiple time-series features of acoustic data, inertial measurement data, and ultra-wideband radar data include peak value, slope, variance, pulse width, etc.

[0040] For example, multiple time-series features of video data include the size of the moving target in the image and its speed of motion.

[0041] For example, for video data, an adaptive median filtering algorithm is used. First, each frame of the image is converted to grayscale. Then, based on the grayscale differences in the neighborhood of each pixel, the size of the filtering window is dynamically adjusted to suppress isolated noise points while preserving edge details of collision areas, preventing the loss of collision features due to filtering. Next, optical flow is used to identify moving targets in the image sequence. When the size of a moving target in the image sequence is detected to increase at a rate exceeding a preset rate within a first preset time period, the midpoint of that time period is determined as a mutation point in the video data. This process is repeated to obtain a time series of mutation points in the video data.

[0042] Step S220: Obtain the detection delay time information for each type of sensor.

[0043] The detection delay time information includes the time interval between the time when the data of each type of sensor shows a sudden change point when a real collision event occurs and the manually calibrated collision time.

[0044] Optionally, a mapping relationship between sensor type and detection delay time can be established in advance by analyzing a large amount of sensor response data under different scenarios to obtain detection delay time information. In this way, the detection delay of each modality data can be determined.

[0045] Optionally, a pre-trained video data recognition model is used to identify driving scene types and weather types based on video data. Then, the delay correction coefficients for each type of sensor matching the current driving scene type and weather type are obtained from the database, along with the factory-calibrated base delay time for each type of sensor. The base delay time is multiplied by the delay correction coefficient to obtain the corrected detection delay time, ultimately yielding the detection delay time information for each type of sensor. This method dynamically corrects the sensor detection delay time, reducing the impact of environmental and scene factors on the delay time, making the delay time more closely match the actual detection scenario, and facilitating the accurate determination of the sequence and time interval of abrupt change points.

[0046] Step S230: Determine the sequence of abrupt change points and the time interval between abrupt change points for multiple types of data when a car collision occurs based on the detection delay time information.

[0047] Specifically, the order of mutation points is determined by ranking the time interval between the time of mutation point in the data of each type of sensor when a real collision event occurs and the manually calibrated collision time, from shortest to longest, and the time interval between mutation points is also determined.

[0048] Step S240: Determine whether a car collision has occurred based on the sequence of mutation points, the time interval between mutation points, and the time series of multiple mutation points.

[0049] Optionally, step S240 includes steps S241 to S243.

[0050] Step S241: Determine the detection time window length based on the time interval between mutation points in multiple types of data.

[0051] Optionally, the average of the time intervals between mutation points corresponding to the various data types can be used as the detection time window length.

[0052] Optionally, the longest mutation point time interval among the mutation point time intervals corresponding to the various data types can be determined as the detection time window length.

[0053] Optionally, based on the order of mutation points in the multi-class data, the first mean and standard deviation of the time interval between all pairs of sequentially adjacent mutation points are calculated, and the detection time window length is set to the sum of the second mean and standard deviation of multiple first means. This approach ensures that the detection time window length fully covers all associated mutation points, preventing false positives.

[0054] Step S242: Determine the starting point of the detection time window based on the time point corresponding to each mutation point in the time series of multiple mutation points, and determine the detection time window based on the starting point and length of the detection time window.

[0055] Optionally, the starting point of the detection time window is determined by the time point corresponding to each mutation point in chronological order, and the detection time window is determined based on the starting point and length of the detection time window.

[0056] Optionally, when multiple mutation points occur within a detection time window, the starting point of the next detection time window is determined based on the time point corresponding to the mutation point that is closest to the end time point of the detection time window outside the detection time window, and the next detection time window is determined based on the starting point and the length of the next detection time window.

[0057] Step S243: Using a sliding window approach, determine whether a car has collided based on multiple determined detection time windows and the sequence of abrupt change points, and determine the time period during which the car collision occurred.

[0058] The sequence of abrupt change points reflects the physical propagation patterns during a collision. For example, generally, inertial measurement data reflects vehicle vibration, ultra-wideband radar data reflects changes in target distance, video data reflects visual changes, and acoustic data reflects the propagation of collision sound. The abrupt change point in ultra-wideband radar data should precede that in video data, which in turn should precede that inertial measurement data, which should also precede that in acoustic data.

[0059] In this way, it is possible to verify whether the actual sequence of mutation points is consistent with the theoretical sequence.

[0060] Optionally, the matching degree between the actual sequence of mutation points within the detection time window and the predetermined sequence of mutation points is calculated. When the matching degree is greater than the preset matching degree, it is determined whether the car has collided based on the actual time interval of mutation points within the detection time window, and the time period of the car collision is determined.

[0061] Specifically, a first string is generated based on the actual sequence of mutation points, and a second string is generated based on a predetermined sequence of mutation points. Each character represents a data type, and the order of characters in the strings indicates the order in which mutation points appear. Then, an edit distance algorithm is used to calculate the edit distance between the first and second strings, and the matching degree is determined based on this edit distance. Edit distance refers to the minimum number of character substitution, insertion, or deletion operations required between two strings. The smaller the edit distance, the higher the matching degree.

[0062] Optionally, the matching degree is determined based on the edit distance and a preset mapping relationship between edit distance and matching degree. For example, when the edit distance is 0, the matching degree is determined to be 100%. When the edit distance is 1, the matching degree is determined to be 50%. When the edit distance is greater than 1 and less than or equal to 2, the matching degree is determined to be 30%. When the edit distance is greater than 2, the matching degree is determined to be 0%.

[0063] Optionally, if the matching degree is not greater than a preset matching degree threshold, it is determined that the car has not collided.

[0064] Optionally, when the matching degree is greater than the preset matching degree, the collision status of the car is determined based on the actual time interval of the abrupt change point within the detection time window, and the collision time period of the car is determined.

[0065] Optionally, the driving scenario type is determined based on the video data. Then, based on the preset correspondence between the driving scenario type and the time interval of the abrupt change point, the time interval range of the abrupt change point sequence for adjacent abrupt change points is determined. If the time intervals of the abrupt change points in the modal data of all pairs of adjacent abrupt change point sequences are within the corresponding time interval range, then it is finally determined that a car collision has occurred. The start time of the detection time window is taken as the start time of the collision occurrence time period, and the end time is taken as the end time of the collision occurrence time period. Otherwise, it is determined that no car collision has occurred.

[0066] For example, driving scenario types include urban roads, highways, and rural roads.

[0067] The above methods can improve the accuracy of collision detection and avoid misjudgments.

[0068] Step S300: When it is determined that a car collision has occurred, the user's driving behavior data is acquired, and the user's operating intention is determined based on the driving behavior data.

[0069] Optionally, driving behavior data includes data such as steering wheel angle, accelerator pedal opening, brake pedal travel, and gear changes.

[0070] Optionally, step S300 includes steps S310 to S350.

[0071] Step S310: When it is determined that a car collision has occurred, obtain the user's driving behavior data and determine the time of the collision.

[0072] Optionally, the collision occurrence time is determined based on the midpoint of the collision occurrence time period determined in step S243.

[0073] Step S320: Determine multiple intent time windows based on the collision occurrence time and multiple preset durations.

[0074] Optionally, the multiple intent time windows include a long-term intent window, a short-term intent window, and a critical intent window. The long-term intent window is used to identify the user's basic driving intent. The short-term intent window is used to identify the user's active risk-avoidance intent. The critical intent window is used to identify the user's instinctive reaction intent. In this way, it is possible to accurately capture different types of operational intents, avoiding the limitations of a single window in distinguishing different intents, and providing a reasonable time range for subsequent feature extraction.

[0075] For example, multiple intent time windows are determined based on the collision occurrence time.

[0076] Optionally, the long-term intent window is determined as [tc] [t1, tc + t1], determine the short-term intent window as [tc t2, tc+t2], determine the critical intent window as [tc [t3, tc + t3], where tc represents the collision occurrence time, t1 represents the first preset time interval, t2 represents the second preset time interval, and t3 represents the third preset time interval. t1 is greater than t2, and t2 is greater than t3.

[0077] Step S330: Extract the first behavioral feature vector of the data segment corresponding to each intention time window in the driving behavior data.

[0078] The first row of the feature vector includes basic behavioral feature parameters and rate of change feature parameters.

[0079] For example, basic behavioral characteristic parameters include throttle opening, brake pedal travel, steering wheel angle, vehicle speed, gear, turn signal status indicator, and seat belt status indicator.

[0080] For example, the rate of change characteristic parameters include the rate of change of throttle opening, the rate of change of brake pedal travel, the rate of change of steering wheel angle, and the rate of change of vehicle speed.

[0081] Step S340: Concatenate the first row of feature vectors of each intent time window's data segment to obtain the second row of feature vectors.

[0082] Step S350: Input the second row feature vector into the trained operation intention determination model to obtain the user's operation intention.

[0083] For example, the operational intent determines the model as a neural network model, a long short-term memory network model, or a Transformer model, etc.

[0084] Optionally, the driving scenario type and weather type are determined based on the video data, as described above. Then, based on the driving scenario type, weather type, and the correspondence information between the preset operation intent and the driving scenario type and weather type, the target's first operation intent determination model is selected from multiple trained first operation intent determination models. The second row is input as a feature vector into the target's first operation intent determination model to obtain multiple first operation intents and a first confidence level corresponding to each first operation intent. If all first confidence levels are lower than a preset first confidence level threshold, the second row is input as a feature vector into a trained general second operation intent determination model to obtain multiple second operation intents and a second confidence level corresponding to each second operation intent. Finally, the final operation intent is determined based on the multiple first operation intents and multiple second operation intents. When at least one first confidence level is not lower than the first confidence level threshold, the first operation intent corresponding to the highest first confidence level is determined as the final operation intent.

[0085] In this way, the targeting and accuracy of intent judgment under different driving scenarios and weather conditions can be improved, avoiding the limitations of a single general model that can be adapted to all scenarios. Furthermore, through confidence verification and multi-model assistance, the probability of misjudgment can be further reduced, ensuring the reliability of operation intent judgment.

[0086] Optionally, a dedicated first operational intent determination model can be pre-trained for each type of driving scenario.

[0087] The formula for determining the final operational intent is: , in, This indicates the first operational intent corresponding to the highest first confidence level. Indicates the first The first operational intent Indicates the first The second operational intent, and The types of operational intentions are the same. Indicates the first The first confidence level corresponding to the first operational intent. Indicates the first The second confidence level corresponds to the second operational intent. This represents the preset first confidence threshold. This indicates the preset first operation intent fusion weight. This indicates the preset second operational intent fusion weight.

[0088] For example, when the vehicle in front suddenly brakes and the driver of this vehicle slams on the brakes to avoid an emergency, the above method can determine that the intention of the operation is emergency braking, and accurately distinguishing between emergency braking and normal braking provides a key basis for determining liability.

[0089] For example, when the driver of this vehicle looks down at his mobile phone and fails to notice the stationary vehicle in front, resulting in a rear-end collision, the above method can determine that the driver's intention is not intentional, accurately identify the lack of intention caused by the driver's negligence, and avoid misjudging the passive collision as an active collision in the future.

[0090] Step S400: Determine the time period, location, severity, type of action, and number of collisions for each vehicle collision based on the multimodal dataset.

[0091] The collision action types include active collision and passive collision.

[0092] Optionally, when the above method is used to determine that a car collision has occurred, the number of collisions is recorded and the time period of each collision is determined.

[0093] Optionally, step S400 includes steps S410 to S430.

[0094] Step S410: Determine the driving scenario type and weather type based on the video data in the multimodal dataset.

[0095] Optionally, a pre-trained video data recognition model can be used to identify driving scenario types and weather types based on video data.

[0096] Optionally, the driving scene type recognition model can be a neural network model or a Transformer model, etc.

[0097] For example, driving scenario types include highways, urban roads, rural roads, tunnels, underground garages, mountain roads, bridges, and parking lots.

[0098] For example, weather types include sunny, cloudy, light rain, moderate rain, heavy rain, rainstorm, foggy, snowy, and dust storm.

[0099] For example, the weather type recognition model can be a neural network model or a Transformer model, etc.

[0100] Optionally, in the video data recognition model, semantic segmentation is performed on video frames to extract features such as road type and surrounding environment from the image. Combined with features such as brightness, contrast, and blur of the video frames, the driving scene type and weather type are identified.

[0101] Optionally, the video data recognition model also identifies the driving scene type and weather type based on environmental noise characteristics in the acoustic data. This approach improves recognition accuracy, especially for scenarios with poor video frame quality.

[0102] Step S420: Determine the fusion feature vector of all classes in the multimodal dataset based on the driving scenario type and weather type.

[0103] Optionally, step S420 includes steps S421 to S423.

[0104] Step S421: Extract the first feature vector of each class of data in the multimodal dataset.

[0105] Step S422: Determine the fusion weight of each type of data corresponding to the driving scenario type and weather type in the preset weight matrix.

[0106] The preset weight matrix is ​​trained based on historical accident data and reflects the reliability of each modality of data under different environments.

[0107] Step S423: Determine the fusion feature vector based on the fusion weights and the first feature vector for each type of data.

[0108] Optionally, the first feature vector is weighted based on the fusion weights of various types of data to obtain the second feature vector, and then all the second feature vectors are concatenated to obtain the fused feature vector.

[0109] For example, in urban road and sunny weather scenarios, the fusion weight corresponding to video data is higher than that corresponding to ultra-wideband radar data and acoustic data. In foggy weather scenarios, the fusion weight of ultra-wideband radar data is the highest.

[0110] Step S430: Determine the collision time period, collision location, collision severity, collision action type, and number of collisions for each vehicle collision based on the fused feature vector.

[0111] Optionally, the comprehensive collision energy curve is calculated based on the fused feature vector, and the DBSCAN density clustering algorithm is used to perform spatiotemporal clustering of the energy peak of the comprehensive collision energy curve. Each cluster corresponds to an independent collision event, and the number of clusters is the number of collisions.

[0112] Optionally, the earliest time corresponding to all energy peaks in each cluster is determined as the collision start time of the collision event corresponding to that cluster, and the latest time corresponding to all energy peaks in each cluster is determined as the collision end time of the collision event corresponding to that cluster. The collision occurrence time period of the collision event is determined based on the collision start time and collision end time. For example, the single-mode energy curve is calculated based on the second feature vector corresponding to each type of data in the fused feature vector. Then, each single-mode energy curve is normalized to obtain the normalized single-mode energy curve. Finally, the normalized single-mode energy curves are weighted and fused to obtain the comprehensive collision energy curve.

[0113] Furthermore, based on the detection delay time information of each type of sensor, multiple single-mode energy curves are time-aligned, and then each single-mode energy curve is normalized to obtain a normalized single-mode energy curve. Finally, the normalized single-mode energy curves are weighted and fused to obtain a comprehensive collision energy curve.

[0114] Optionally, since the second eigenvector has been weighted, each normalized single-mode energy curve has the same weight.

[0115] Optionally, the single-mode energy curve corresponding to the inertial measurement data is calculated as follows: , in, Represents a time variable. This represents the single-mode energy curve corresponding to the inertial measurement data. It represents translational energy.

[0116] Optionally, ,in, This indicates the preset vehicle curb weight. This represents the triaxial composite acceleration determined based on the second eigenvector of the inertial measurement data. Indicates that the car was obtained from the car. The speed of time, This represents the time variable for integration.

[0117] Optionally, in collision and impact scenarios, to facilitate real-time calculations using acceleration data measured by the inertial measurement unit, this application adopts an engineering equivalent form: .

[0118] Represents rotational energy. , This represents the preset average moment of inertia of the car about its center of mass. This represents the triaxial composite angular velocity determined based on the second eigenvector of the inertial measurement data. This represents the preset first calculation coefficient. This represents the preset second calculation coefficient.

[0119] Optionally, the energy spectrum is obtained by calculating the squared amplitude of each frequency component based on the second eigenvector of the acoustic data. Then, the single-mode energy curve corresponding to the acoustic data is determined based on the energy spectrum.

[0120] Optionally, the energy curve of the reflected radar signal in the ultra-wideband radar data is calculated and determined as the single-mode energy curve corresponding to the ultra-wideband radar data.

[0121] Optionally, the single-mode energy curve corresponding to the video data is determined based on the gradient energy at multiple time points in the second feature vector of the video data. The gradient energy is the sum of squares of the image gradient magnitudes.

[0122] Optionally, the time difference of arrival algorithm is used to determine the direction of the collision sound source based on the second feature vector corresponding to the acoustic data, and the direction of the collision sound source is verified based on the target angle measurement result in the second feature vector corresponding to the ultra-wideband radar data. If the direction of the collision sound source is located in the target angle direction, the collision position is further determined based on the position of the moving target in the second feature vector corresponding to the video data.

[0123] For example, the collision location includes the front, rear, left, and right sides of the vehicle.

[0124] Optionally, the total energy of all collisions is calculated based on the comprehensive collision energy curve, and the collision severity corresponding to the total energy of the collisions is determined based on the total energy of all collisions, multiple preset total energy value ranges, and the correspondence between multiple total energy value ranges and collision severity.

[0125] Optionally, the collision action type can be determined based on the fused feature vector, the user's driving behavior data, and the user's operational intent.

[0126] Specifically, the speed and direction of the vehicle before and after the collision are determined based on the user's driving behavior data and the time period of the collision. The relative speed between the vehicle and the collision target before and after the collision is determined based on the speed of the moving target in the second feature vector corresponding to the video data and the speed of the vehicle before and after the collision. The angle between the collision force vector and the direction of the vehicle's movement is determined based on the second feature vector corresponding to the inertial measurement data. Combined with the user's operating intention, the collision action type is determined to be an active collision or a passive collision.

[0127] For example, when the relative speed between the vehicle and the collision target before and after the collision is greater than a preset relative speed threshold, and the angle between the collision force vector and the direction of the vehicle's movement is less than 45 degrees, and the user's intention is to brake urgently, the collision action type is determined to be an active collision.

[0128] For example, when the angle between the collision force vector and the direction of the vehicle's movement is greater than 90 degrees, and the user's intention is no intention to operate, the collision action type is determined to be a passive collision.

[0129] Step S500: Determine the type of collision target based on ultra-wideband radar data, video data, and acoustic data.

[0130] The collision target types include people, animals, and inanimate objects. Inanimate objects are further divided into vehicles and other objects.

[0131] Optionally, biometric identification is performed based on ultra-wideband radar data to determine whether the collision target is a biological or non-biological object. When the collision target is determined to be a biological object, visual features such as the outline, texture, color, and motion trajectory of the collision target are extracted from video data, and sound features of the collision target are extracted from acoustic data. A pre-trained first convolutional neural network is then used to determine whether the collision target is a human or an animal based on the visual and sound features of the collision target.

[0132] Optionally, when the collision target is determined to be a non-biological object, visual features such as the outline, texture, color and motion trajectory of the collision target are extracted from the video data, and sound features of the collision target are extracted from the acoustic data. A pre-trained second convolutional neural network is then used to determine whether the collision target type is a car or other object based on the visual and sound features of the collision target.

[0133] Step S600: Determine collision event information based on the collision occurrence time period, collision location, collision severity, collision action type, collision frequency, collision target type, and user's operation intent.

[0134] Optionally, the collision event information includes the time period of the collision, the location of the collision, the severity of the collision, the type of collision action, the number of collisions, the type of collision target, and the user's operational intent.

[0135] Step S700: Determine the collision accident type and collision liability information based on the collision event information.

[0136] Optionally, step S700 includes steps S710 to S730.

[0137] Step S710: Determine the first-level category of the collision accident type based on the collision target type in the collision event information.

[0138] The first category includes accidents between motor vehicles, accidents between motor vehicles and pedestrians, accidents between motor vehicles and animals, and accidents between motor vehicles and fixed objects.

[0139] Optionally, when a collision event involves multiple collision targets (e.g., hitting a pedestrian first and then a guardrail), the primary collision target (i.e., the first collision target) is used as the primary category determination criterion, while supplementing the description information of the collision event type with information on other collision targets. This approach ensures the completeness of the collision event type and avoids category confusion caused by multi-target collisions.

[0140] Optionally, if a person is present among the multiple collision targets, the first-level category of the collision accident type is determined to be a motor vehicle and pedestrian accident.

[0141] Step S720: Determine the second-level category of the collision accident type based on the collision occurrence time, collision location, collision severity, and number of collisions in the collision event information.

[0142] The second category includes rear-end collisions, frontal collisions, side collisions, scrapes, reversing accidents, and chain-reaction collisions.

[0143] Optionally, a pre-trained third convolutional neural network model is used to determine the second-level category of the collision accident type based on the collision occurrence time, collision location, collision severity, and collision frequency in the collision event information.

[0144] Optionally, a second-level category of the collision accident type is also determined based on the type of driving scenario.

[0145] For example, in an urban road scenario, when the collision location is the side of the vehicle, the time of the collision is determined to be a peak period, the number of collisions is a single incident, and the severity of the collision is moderate, the collision accident type is determined to be a side collision accident.

[0146] For example, in a highway scenario, when the collision location is the rear of the vehicle, the collision action type is passive collision, and the number of collisions is single, the collision accident type is determined to be a rear-end collision.

[0147] Step S730: Determine the collision responsibility information based on the collision action type and the user's operation intent in the collision event information.

[0148] Optionally, step S730 includes steps S731 to S735.

[0149] Step S731: Determine traffic rules based on the data segments of the video data within the time period of the collision.

[0150] Optionally, semantic segmentation is performed on the video data within the time period of the collision to extract information such as traffic signs, road markings, and the status of traffic participants from the video data. Combined with a pre-set traffic rule database, keyword matching and semantic reasoning are used to determine the applicable traffic rules corresponding to the collision scenario.

[0151] Optionally, traffic rules include rules for running red lights at intersections, rules for liability in rear-end collisions on highways, and rules for yielding to pedestrians.

[0152] Step S732: Determine whether the user has committed any wrongdoing based on traffic rules and the user's operational intent.

[0153] Optionally, a rule base for determining violations of operational intentions and traffic rules can be pre-established. The user's operational intentions are compared with the applicable traffic rules to determine whether the user's operation violates traffic rules. For example, if the user's operational intention is no operation and the applicable traffic rule is "emergency braking is required in an emergency," the user is determined to have committed a wrongful act.

[0154] Step S733: When a user commits an error, obtain the pre-trained Bayesian causal graph and determine the type of error.

[0155] Optionally, the types of wrongful conduct include speeding, running a red light, failure to yield, and operational errors.

[0156] The Bayesian causal graph is pre-trained based on a large number of collision accident samples.

[0157] Step S734: Determine the causal contribution of all wrongdoings to the occurrence of the collision based on the Bayesian causal graph and the type of each wrongdoing.

[0158] Optionally, the causal contribution is the probability that all wrongdoings lead to a collision, and the Bayesian causal graph quantifies the causal relationship probability between various wrongdoings and the occurrence of a collision.

[0159] Optionally, the formula for calculating the causal contribution is: ,in, This represents the probability of a collision occurring. This represents all from the 1st to the 1st. One wrongful act, This indicates a collision event.

[0160] Step S735: Determine collision liability information based on causal contribution.

[0161] Optionally, the collision liability information can be divided into five levels of collision liability: full liability, primary liability, equal liability, secondary liability, and no liability.

[0162] Optionally, collision liability information can be determined based on the causal contribution rate, the pre-defined causal contribution rate, and the correspondence between the collision liability level.

[0163] Step S800: Generate an accident handling plan based on the severity of the collision, the type of the collision accident, and the collision liability information.

[0164] Optionally, step S800 includes steps S810 to S840.

[0165] Step S810: Determine the property damage risk score based on the severity of the collision.

[0166] Optionally, the property damage risk score is obtained by multiplying the severity of the collision by a preset first risk calculation coefficient and adding the number of collisions by a preset second risk calculation coefficient.

[0167] Optionally, the property loss risk score can be set to a value between 0 and 100. If the calculated property loss risk score is higher than 100, the final property loss risk score will be set to 100.

[0168] Step S820: Determine the personal injury risk score based on the type and severity of the collision.

[0169] Optionally, a preset third risk calculation coefficient corresponding to the collision accident type is determined based on the collision accident type and the preset correspondence information between the collision accident type and the accident type weight. The third risk calculation coefficient is added to the collision severity and multiplied by the preset fourth risk calculation coefficient to obtain the personal injury risk score.

[0170] Optionally, the numerical range of the personnel injury risk score is [0, 100]. If the calculated personnel injury risk score is higher than 100, the final personnel injury risk score is set to 100.

[0171] Step S830: Determine the liability risk score based on the collision liability information.

[0172] Optionally, the corresponding liability risk score can be determined based on the collision liability level in the collision liability information and the preset correspondence between the collision liability level and the liability risk score.

[0173] Optionally, the liability risk score can be set to a value between 0 and 100. If the calculated liability risk score is higher than 100, the final liability risk score will be set to 100.

[0174] Step S840: Generate an accident handling plan based on the preset action space, property damage risk score, personal injury risk score, and liability risk score.

[0175] Optionally, the preset action space includes actions such as personnel rescue, accident scene protection, alarm reporting, insurance claims, liability negotiation, and vehicle repair.

[0176] Optionally, a comprehensive risk score is obtained by weighting the property damage risk score, personal injury risk score, and liability risk score according to preset weighting weights, and a corresponding action is matched according to the comprehensive risk score and the type of collision accident using preset matching rules.

[0177] Optionally, corresponding actions are matched for different responsible parties. For example, the party fully responsible must prioritize actions such as reporting to the police, filing an insurance claim, and seeking compensation, while the party without responsibility must cooperate with the on-site investigation and provide relevant evidence.

[0178] Optionally, targeted actions can be added based on the number of collisions and the type of collision target. For example, in cases of multiple collisions, the damage to each collision target needs to be determined based on video data, and in cases of accidents involving motor vehicles and animals, the relevant departments need to be contacted to handle the animal remains.

[0179] In this way, the method can be applied to generate handling solutions for various types of collision accidents, especially for scenarios with different severity levels, different liability divisions, and different accident types. It can directly provide precise handling guidance for drivers, passengers, traffic police, and insurance companies.

[0180] Optionally, a multi-agent proximal policy optimization algorithm can be used to construct an accident handling solution generation model. In this way, the handling strategy can be dynamically adjusted in complex and ever-changing accident scenarios, balancing multiple objectives such as personnel safety, property loss, handling efficiency, and liability compliance.

[0181] Optionally, a reinforcement learning environment can be predefined. The reinforcement learning environment includes a state space and an action space.

[0182] Optionally, the state space includes the time period of the collision, the location of the collision, the severity of the collision, the type of collision action, the number of collisions, the type of collision target, and the user's operational intent.

[0183] Optionally, the action space includes safety protection actions, personal rescue actions, evidence collection actions, and traffic control actions. Reward points and reward types are pre-set for each action. Reward types include reducing personal injury risk, reducing property damage risk, and reducing liability risk.

[0184] For example, safety protection actions include turning on hazard lights, placing warning triangles, and evacuating personnel.

[0185] For example, rescue operations include actions such as calling 120, providing on-site first aid, and escorting the injured.

[0186] For example, evidence preservation includes actions such as taking photos, recording videos, and retaining witness information.

[0187] For example, traffic management includes actions such as dialing 122, moving vehicles, and directing traffic.

[0188] Optionally, an initial state vector is generated based on the collision severity, collision accident type, and collision responsibility information in the reinforcement learning environment. The initial state vector is then input into a pre-trained accident handling scheme generation model to obtain the accident handling scheme.

[0189] For example, the multi-objective optimization function in the accident handling solution generation model is: , in, Represents a multi-objective optimization function. Indicates the first The objective function of each objective.

[0190] Optionally, ,in, Indicates the importance sampling ratio. Indicates the first Advantage function for each objective. This indicates the preset clipping factor.

[0191] Optionally, the advantage function for the first objective is 100 - property loss risk score + reward points for all actions in the current action plan whose reward type is reducing property loss risk. The advantage function for the second objective is 100 - the calculation function of the personal injury risk score + reward points for all actions in the current action plan whose reward type is reducing personal injury risk. The advantage function for the third objective is 100 - liability risk score + reward points for all actions in the current action plan whose reward type is reducing personal liability risk. In the accident handling plan generation model, the goal is to maximize the value of the multi-objective optimization function to ultimately generate the accident handling plan.

[0192] Optionally, the method further includes steps S910 to S920.

[0193] Step S910: When it is determined that the car has not collided, determine the collision probability of the car based on the multimodal dataset.

[0194] Optionally, referring to the above, when it is determined that the car has not collided, the collision probability of the car is determined based on the fused feature vector.

[0195] Step S920: Generate a collision warning based on the vehicle's collision probability.

[0196] For example, in a pre-trained collision warning model, the probability of a car's collision is determined based on a multimodal dataset, and warning thresholds and collision confirmation thresholds are set. When the overall risk score exceeds the warning threshold, a collision warning is generated. When the overall risk score continues to rise and exceeds the collision confirmation threshold, a collision is confirmed.

[0197] Optionally, the collision warning model can be a neural network model or a Transformer model, etc.

[0198] Optionally, to ensure the model's generalization ability under different vehicle types and road conditions, a training strategy combining transfer learning and online incremental learning is adopted to train the above model.

[0199] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 2 As shown, the electronic device 400 includes: one or more processors 410 and a memory 420. Figure 2 Take a processor 410 as an example.

[0200] Alternatively, the processor 410 and the memory 420 can be connected via a bus or other means. Figure 2 Taking the example of a connection between China and Israel via a bus.

[0201] Optionally, the processor 410 is configured to acquire a multimodal dataset of the vehicle, including video data, acoustic data, inertial measurement data, and ultra-wideband radar data; determine whether a collision has occurred based on the multimodal dataset; when a collision is determined, acquire the user's driving behavior data and determine the user's operational intent based on the driving behavior data; determine the collision time period, collision location, collision severity, collision action type, and number of collisions for each collision based on the multimodal dataset, where the collision action type includes active collisions and passive collisions; determine the collision target type based on the ultra-wideband radar data, video data, and acoustic data, where the collision target type includes people, animals, and inanimate objects; determine collision event information based on the collision time period, collision location, collision severity, collision action type, number of collisions, collision target type, and user's operational intent; determine the collision accident type and collision liability information based on the collision event information; and generate an accident handling plan based on the collision severity, collision accident type, and collision liability information.

[0202] Optionally, the memory 420, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules of the human-machine collaborative vehicle scene hazard identification and auxiliary decision-making method in the embodiments of this application. The processor 410 executes various functional applications and data processing of the electronic device 400 by running the non-volatile software programs, instructions, and modules stored in the memory 420, thereby realizing the human-machine collaborative vehicle scene hazard identification and auxiliary decision-making method of the above method embodiments.

[0203] Optionally, the memory 420 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device 400, etc. Furthermore, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. Optionally, the memory 420 may include memory remotely located relative to the processor 410, and these remote memories may be connected to the controller via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0204] Optionally, one or more modules are stored in the memory 420. When executed by one or more processors 410, they perform the human-machine collaborative vehicle scene hazard identification and auxiliary decision-making method in any of the above method embodiments, for example, performing the above-described... Figure 1 The method steps S100 to S800.

[0205] Please refer to Figure 3 , Figure 3 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 500 stores program code 510, which can be called by a processor to execute the human-machine collaborative vehicle on-site hazard identification and auxiliary decision-making method described in the above method embodiments.

[0206] The computer-readable storage medium 500 may be an electronic storage device such as flash memory, electrically erasable programmable read-only memory (EEPROM), hard disk, or read-only memory (ROM). Optionally, the computer-readable storage medium includes a non-volatile computer-readable medium. The computer-readable storage medium 500 has storage space for program code that performs any of the method steps in the above-described human-machine collaborative vehicle scene hazard identification and decision support method. This program code can be read from or written to one or more computer program products. The program code may, for example, be compressed in an appropriate form.

[0207] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards.

[0208] In summary, this application provides a human-machine collaborative method and electronic device for identifying and assisting decision-making regarding on-site vehicle hazards. The method includes: acquiring a multimodal dataset of the vehicle, comprising video data, acoustic data, inertial measurement data, and ultra-wideband radar data; determining whether a collision has occurred based on the multimodal dataset; and determining whether a collision has occurred by performing time-series change point detection on each type of data in the multimodal dataset to obtain multiple change point time series, including time series of change points for video data, acoustic data, inertial measurement data, and ultra-wideband radar data; acquiring detection delay time information for each type of sensor; and determining the order and time interval of change points for the multiple types of data when a collision occurs based on the detection delay time information. This application determines whether a vehicle collision has occurred based on the sequence of abrupt change points, the time interval between abrupt change points, and the time series of multiple abrupt change points. When a collision is confirmed, it acquires the user's driving behavior data and determines the user's operational intent based on this data. It then uses a multimodal dataset to determine the collision time period, collision location, collision severity, collision action type, and number of collisions for each collision, including both active and passive collisions. Finally, it uses ultra-wideband radar data, video data, and acoustic data to determine the collision target type, including people, animals, and inanimate objects. The application further determines collision event information based on the collision time period, collision location, collision severity, collision action type, number of collisions, collision target type, and the user's operational intent. Based on this collision event information, it determines the collision accident type and collision liability information. Finally, it generates an accident handling plan based on the collision severity, collision accident type, and collision liability information. This application, by determining collision event information based on the collision time period, collision location, collision severity, collision action type, number of collisions, collision target type, and the user's operational intent, can record the complete and true situation of the entire collision event from multiple dimensions, thereby generating a suitable accident handling plan.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A human-machine collaborative method for identifying potential hazards and assisting in decision-making at automotive sites, characterized in that, include: Acquire a multimodal dataset of the vehicle, which includes video data, acoustic data, inertial measurement data, and ultra-wideband radar data; Determine whether a car collision has occurred based on the aforementioned multimodal dataset; Determining whether a car collision has occurred based on the multimodal dataset includes: Time series change point detection is performed on each type of data in the multimodal dataset to obtain multiple change point time series, including video data change point time series, acoustic data change point time series, inertial measurement data change point time series, and ultra-wideband radar data change point time series. Obtain the detection delay time information for each type of sensor; Based on the detection delay time information, determine the sequence of abrupt change points and the time interval between abrupt change points for multiple types of data when a car collision occurs; Whether a car collision has occurred is determined based on the sequence of mutation points, the time interval between mutation points, and the time series of multiple mutation points. When it is determined that the car has collided, the user's driving behavior data is acquired, and the user's operating intention is determined based on the driving behavior data; Based on the multimodal dataset, the collision time period, collision location, collision severity, collision action type, and number of collisions for each vehicle collision are determined. The collision action type includes active collisions and passive collisions. The collision target type is determined based on the ultra-wideband radar data, the video data, and the acoustic data. The collision target type includes people, animals, and inanimate objects. The collision event information is determined based on the collision occurrence time period, collision location, collision severity, collision action type, collision frequency, collision target type, and user's operation intention. Based on the collision event information, the collision accident type and collision liability information are determined; An accident handling plan is generated based on the severity of the collision, the type of the collision accident, and the collision liability information.

2. The human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards according to claim 1, characterized in that, The method of determining whether a car has collided based on the sequence of mutation points, the time interval between mutation points, and the time series of multiple mutation points includes: The length of the detection time window is determined based on the time interval between mutation points in the aforementioned multiple types of data. The starting point of the detection time window is determined based on the time point corresponding to each mutation point in the multiple mutation point time series, and the detection time window is determined based on the starting point of the detection time window and the length of the detection time window. A sliding window approach is used to determine whether a collision has occurred based on multiple determined detection time windows and the sequence of abrupt change points, and to determine the time period during which the collision occurred.

3. The human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards according to claim 1, characterized in that, When it is determined that a collision has occurred, acquiring the user's driving behavior data and determining the user's operational intent based on the driving behavior data includes: When it is determined that a collision has occurred, the user's driving behavior data is acquired and the time of the collision is determined. Multiple intent time windows are determined based on the collision occurrence time and multiple preset durations; Extract the first behavior feature vector from the data segment corresponding to each intent time window in the driving behavior data. The first behavior feature vector includes basic behavior feature parameters and rate of change feature parameters. The first row of feature vectors of each data segment of the intent time window is concatenated to obtain the second row of feature vectors; The second behavioral feature vector is input into the trained operation intent determination model to obtain the user's operation intent.

4. The human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards according to claim 1, characterized in that, The process of determining the collision time period, collision location, collision severity, collision action type, and number of collisions for each vehicle collision based on the multimodal dataset includes: The driving scenario type and weather type are determined based on the video data in the multimodal dataset; Based on the driving scenario type and the weather type, determine the fusion feature vector of all class data in the multimodal dataset; Based on the fused feature vector, the collision time period, collision location, collision severity, collision action type, and number of collisions for each vehicle collision are determined.

5. The human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards according to claim 4, characterized in that, The step of determining the fusion feature vector of all classes in the multimodal dataset based on the driving scenario type and the weather type includes: Extract the first feature vector of each data class in the multimodal dataset; Determine the fusion weight for each type of data corresponding to the driving scenario type and the weather type in the preset weight matrix; The fusion feature vector is determined based on the fusion weight of each type of data and the first feature vector.

6. The human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards according to claim 1, characterized in that, The process of determining the collision accident type and collision liability information based on the collision event information includes: Based on the collision target type in the collision event information, a first-level category of the collision accident type is determined. The first-level category includes accidents between motor vehicles, accidents between motor vehicles and pedestrians, accidents between motor vehicles and animals, and accidents between motor vehicles and fixed objects. The second-level category of the collision accident type is determined based on the collision occurrence time period, collision location, collision severity, and collision number in the collision event information. The second-level category includes rear-end collision, frontal collision, side collision, scrape, reversing accident, and chain collision. The collision liability information is determined based on the collision action type in the collision event information and the user's operational intent.

7. The human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards according to claim 6, characterized in that, Determining the collision liability information based on the collision action type in the collision event information and the user's operational intent includes: Traffic rules are determined based on the video data within the data segment of the time period in which the collision occurred. Determine whether the user has committed any wrongdoing based on the traffic rules and the user's operational intent; When a user commits an error, the pre-trained Bayesian causal graph is obtained, and the type of error is determined. The causal contribution of all the said wrongful behaviors to the occurrence of the collision is determined based on the Bayesian causal graph and the type of each said wrongful behavior. The collision liability information is determined based on the causal contribution.

8. The human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards according to claim 1, characterized in that, The process of generating an accident handling plan based on the severity of the collision, the type of the collision accident, and the collision liability information includes: A property damage risk score is determined based on the severity of the collision. A personal injury risk score is determined based on the type of collision accident and the severity of the collision. A liability risk score is determined based on the collision liability information; The accident handling plan is generated based on the preset action space, the property loss risk score, the personal injury risk score, and the liability risk score.

9. The human-machine collaborative method for identifying and assisting in decision-making regarding on-site vehicle hazards according to claim 1, characterized in that, The method further includes: When it is determined that the car has not been involved in a collision, the collision probability of the car is determined based on the multimodal dataset; A collision warning is generated based on the collision probability of the vehicle.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the human-machine collaborative vehicle on-site hazard identification and auxiliary decision-making method as described in any one of claims 1 to 9.

Images