Knowledge-enhanced large model-based autonomous vehicle abnormal behavior detection method

Abstract

The application discloses a kind of based on knowledge enhancement big model's automatic driving vehicle abnormal behavior detection method, it is related to automatic driving car abnormal operating state detection technical field, method includes: obtaining vehicle operation data and vehicle video data, vehicle operation data is analyzed, obtains potential abnormal sample;Vehicle video data is extracted to feature, and visual feature data is obtained;Potential abnormal sample and visual feature data are processed in space-time alignment, and multiple-source space-time database is obtained;Establish abnormal event classification system, calibrate each event in multiple-source space-time database based on abnormal event classification system, and obtain multi-modal dataset;According to multi-modal dataset, train knowledge enhancement big model, and obtain abnormal behavior detection model;According to abnormal behavior detection model, detect vehicle behavior data.The application can accurately identify vehicle abnormal behavior, overcome data utilization is insufficient, scene adaptability is poor and the problems such as low detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of abnormal operating state detection technology for autonomous vehicles, and in particular to a method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model. Background Technology

[0002] As autonomous driving technology enters the stage of large-scale demonstration and application, the safe operation and supervision of autonomous vehicles in open road environments has become a core issue that urgently needs to be addressed in the field of intelligent transportation. Autonomous vehicles frequently exhibit abnormal behaviors such as unexplained deceleration and braking during actual road operation. These abnormal behaviors not only disrupt normal traffic flow and reduce road efficiency but also pose a potential threat to driving safety, seriously affecting passenger travel experience and public acceptance of autonomous driving technology. Therefore, researching and promptly identifying technical means for abnormal behavior of autonomous vehicles is of great significance for ensuring road traffic safety and improving the reliability of autonomous driving transportation systems.

[0003] To address the issue of abnormal behavior detection in autonomous vehicles, existing technologies primarily employ detection methods based on a single data source. Some solutions rely on operational data collected by onboard sensors, monitoring vehicle speed and acceleration parameters by setting fixed thresholds. Other solutions utilize computer vision technology to analyze road surveillance videos, extracting vehicle behavior features through target detection and tracking algorithms. Still other solutions attempt to simply stitch together the aforementioned single data sources and use traditional machine learning models to identify abnormal patterns.

[0004] However, existing technical solutions have significant shortcomings. Technologies based on single operational data struggle to acquire contextual information about the vehicle's surrounding environment and cannot distinguish between normal avoidance behavior and abnormal braking behavior. Technologies based on single video data are limited by viewpoint occlusion and lighting conditions, and struggle to accurately obtain vehicle dynamic parameters. Simple data stitching methods lack effective spatiotemporal alignment mechanisms, resulting in misalignment of heterogeneous data in time and space, making it impossible to establish a precise correspondence between vehicle operating status and traffic scenarios. Furthermore, existing technologies generally lack structured abnormal event classification systems, have fixed detection thresholds, and are ill-suited to complex and ever-changing traffic scenarios. Traditional models also lack the ability to model long-term temporal dependencies, resulting in anomaly detection accuracy and stability that fail to meet practical application requirements.

[0005] In summary, existing technologies suffer from shortcomings such as insufficient data utilization, poor scene adaptability, and low detection accuracy, making it difficult to effectively solve the problem of accurate identification of abnormal behavior of autonomous vehicles in complex traffic scenarios. Summary of the Invention

[0006] The technical problem this invention aims to solve is to address the shortcomings of existing technologies, specifically the poor scene adaptability and low detection accuracy in the abnormal behavior detection process of autonomous vehicles. Specifically, it provides a method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model, as detailed below: 1) In a first aspect, the present invention provides a method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model, the specific technical solution of which is as follows: S1, acquire vehicle operation data and vehicle video data, analyze the vehicle operation data to obtain potential abnormal samples; extract features from the vehicle video data to obtain visual feature data; S2, perform spatiotemporal alignment processing on the potential abnormal samples and the visual feature data to obtain a multi-source spatiotemporal database; S3, establish an abnormal event classification system, and label each event in the multi-source spatiotemporal database based on the abnormal event classification system to obtain a multimodal dataset; S4. Train a knowledge-enhanced large model based on the multimodal dataset to obtain an abnormal behavior detection model; S5, input the vehicle behavior data to be detected into the abnormal behavior detection model to obtain the abnormal behavior detection result.

[0007] The beneficial effects of the knowledge-enhanced large model-based abnormal behavior detection method for autonomous vehicles provided by this invention are as follows: By acquiring and processing vehicle operation data and video data separately, the system achieves the fusion and complementarity of vehicle dynamic parameters and surrounding environmental visual information, overcoming the incompleteness of information from a single data source and providing multi-dimensional data support for abnormal behavior recognition. Through spatiotemporal alignment processing of potential abnormal samples and visual feature data, a precise correspondence between vehicle operating status and traffic scenes in both time and space dimensions is established, effectively solving the data misalignment problem between heterogeneous data sources and ensuring the accuracy of data association. By establishing an abnormal event classification system and labeling each event in the multi-source spatiotemporal database, a structured abnormal event judgment standard and classification basis are formed, making the definition of abnormal behavior clearer and more standardized, and providing clear knowledge guidance for the detection model. By training a knowledge-enhanced model using the labeled multimodal dataset, the model can effectively learn and capture the dynamic evolution of abnormal events in the time dimension and the correlation patterns between multi-source features, significantly improving the accuracy and reliability of abnormal behavior detection in complex traffic scenarios.

[0008] Based on the above solution, the present invention can be further improved as follows.

[0009] Furthermore, the analysis of the vehicle operation data includes: Based on GIS spatial matching technology, the vehicle's geographical location information in the vehicle operation data is matched with the intersection location information in the intersection database to determine the traffic scenario in which the vehicle is located. The vehicle operation data is divided according to the traffic scenario, and the vehicle operation data under different traffic scenarios is clustered to obtain potential abnormal vehicle behaviors and corresponding abnormal behavior judgment thresholds. The potential abnormal behaviors of all vehicles and their corresponding abnormal behavior judgment thresholds are screened to obtain the potential abnormal samples.

[0010] Furthermore, the feature extraction from the vehicle video data includes: Target identification and localization are performed on video frames in the vehicle video data, and targets in consecutive frames are correlated to obtain multi-target tracking results; the multi-target tracking results include the motion trajectories of multiple vehicle targets; Based on the multi-target tracking results, behavioral features of multiple vehicle targets are extracted and combined with the multi-target tracking results to form the visual feature data.

[0011] Furthermore, the abnormal event classification system includes: unexplained deceleration events, unexplained braking events, and traffic accident events; The criteria for determining the unexplained deceleration event include: the vehicle continuously decelerates within a preset travel distance in the direction of travel when there are no obstacles and the deceleration magnitude exceeds a preset acceleration threshold. The criteria for determining the unexcused braking event include: the vehicle decelerates to a stop when there are no obstacles within a preset travel distance in the direction of travel, and this deceleration is accompanied by a continuous deceleration process; The criteria for determining a traffic accident include: the distance between the vehicle and an obstacle in the direction of travel is less than a preset collision threshold and is accompanied by severe deceleration.

[0012] 2) In a second aspect, the present invention also provides an abnormal behavior detection system for autonomous vehicles based on a knowledge-enhanced large model. The specific technical solution is as follows, including: a data acquisition module, a spatiotemporal alignment module, a classification and calibration module, a knowledge enhancement module, and an anomaly detection module. The data acquisition module is used to acquire vehicle operation data and vehicle video data, analyze the vehicle operation data to obtain potential abnormal samples, and extract features from the vehicle video data to obtain visual feature data. The spatiotemporal alignment module is used to perform spatiotemporal alignment processing on the potential abnormal samples and the visual feature data to obtain a multi-source spatiotemporal database. The classification and labeling module is used to establish an abnormal event classification system, and to label each event in the multi-source spatiotemporal database based on the abnormal event classification system to obtain a multimodal dataset; The knowledge enhancement module is used to train a large knowledge enhancement model based on the multimodal dataset to obtain an abnormal behavior detection model. The anomaly detection module is used to input the behavior data of the vehicle to be detected into the anomaly behavior detection model to obtain the anomaly behavior detection result.

[0013] Based on the above solution, the present invention can be further improved as follows.

[0014] Furthermore, the analysis of the vehicle operation data includes: Based on GIS spatial matching technology, the vehicle's geographical location information in the vehicle operation data is matched with the intersection location information in the intersection database to determine the traffic scenario in which the vehicle is located. The vehicle operation data is divided according to the traffic scenario, and the vehicle operation data under different traffic scenarios is clustered to obtain potential abnormal vehicle behaviors and corresponding abnormal behavior judgment thresholds. The potential abnormal behaviors of all vehicles and their corresponding abnormal behavior judgment thresholds are screened to obtain the potential abnormal samples.

[0015] Furthermore, the feature extraction from the vehicle video data includes: Target identification and localization are performed on video frames in the vehicle video data, and targets in consecutive frames are correlated to obtain multi-target tracking results; the multi-target tracking results include the motion trajectories of multiple vehicle targets; Based on the multi-target tracking results, behavioral features of multiple vehicle targets are extracted and combined with the multi-target tracking results to form the visual feature data.

[0016] Furthermore, the abnormal event classification system includes: unexplained deceleration events, unexplained braking events, and traffic accident events; The criteria for determining the unexplained deceleration event include: the vehicle continuously decelerates within a preset travel distance in the direction of travel when there are no obstacles and the deceleration magnitude exceeds a preset acceleration threshold. The criteria for determining the unexcused braking event include: the vehicle decelerates to a stop when there are no obstacles within a preset travel distance in the direction of travel, and this deceleration is accompanied by a continuous deceleration process; The criteria for determining a traffic accident include: the distance between the vehicle and an obstacle in the direction of travel is less than a preset collision threshold and is accompanied by severe deceleration.

[0017] 3) In a third aspect, the present invention also provides a computer device, the computer device including a processor coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the computer device to implement any of the above methods.

[0018] 4) In a fourth aspect, the present invention also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0019] It should be noted that the beneficial effects of the technical solutions of the second to fourth aspects of the present invention and their corresponding possible implementations can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0020] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the steps of an abnormal behavior detection method for autonomous vehicles based on a knowledge-enhanced large model, according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating data acquisition and processing for an abnormal behavior detection method for autonomous vehicles based on a knowledge-enhanced large model, according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the overall technical route of an abnormal behavior detection method for autonomous vehicles based on a knowledge-enhanced large model according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0022] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model, comprising the following steps: S1. Acquire vehicle operation data and vehicle video data, analyze the vehicle operation data to obtain potential abnormal samples; extract features from the vehicle video data to obtain visual feature data. S2, perform spatiotemporal alignment processing on potential abnormal samples and visual feature data to obtain a multi-source spatiotemporal database; S3. Establish an abnormal event classification system, and label each event in the multi-source spatiotemporal database based on the abnormal event classification system to obtain a multimodal dataset; S4. Train a knowledge-enhanced large model based on a multimodal dataset to obtain an abnormal behavior detection model; S5. Input the vehicle behavior data to be detected into the abnormal behavior detection model to obtain the abnormal behavior detection results.

[0023] The beneficial effects of the knowledge-enhanced large model-based abnormal behavior detection method for autonomous vehicles provided by this invention are as follows: By acquiring and processing vehicle operation data and video data separately, the system achieves the fusion and complementarity of vehicle dynamic parameters and surrounding environmental visual information, overcoming the incompleteness of information from a single data source and providing multi-dimensional data support for abnormal behavior recognition. Through spatiotemporal alignment processing of potential abnormal samples and visual feature data, a precise correspondence between vehicle operating status and traffic scenes in both time and space dimensions is established, effectively solving the data misalignment problem between heterogeneous data sources and ensuring the accuracy of data association. By establishing an abnormal event classification system and labeling each event in the multi-source spatiotemporal database, a structured abnormal event judgment standard and classification basis are formed, making the definition of abnormal behavior clearer and more standardized, and providing clear knowledge guidance for the detection model. By training a knowledge-enhanced model using the labeled multimodal dataset, the model can effectively learn and capture the dynamic evolution of abnormal events in the time dimension and the correlation patterns between multi-source features, significantly improving the accuracy and reliability of abnormal behavior detection in complex traffic scenarios.

[0024] It should be noted that, for ease of understanding, the technical terms used in this solution will be explained one by one, and will not be repeated hereafter: Vehicle operation data refers to the digital information collected in real time from the onboard sensors and positioning devices of autonomous vehicles, including but not limited to vehicle geographical location information, instantaneous speed, acceleration values, dwell time, and timestamp information. Vehicle operation data reflects the vehicle's own motion state and time attributes.

[0025] Vehicle video data refers to the dynamic image stream obtained by continuously capturing images of the monitored area through image acquisition devices deployed along the road. It contains visual information about the road traffic environment and is used to capture vehicle appearance features and surrounding scene context.

[0026] Potentially abnormal samples: These refer to data records identified from massive amounts of raw data through cluster analysis and threshold screening of vehicle operation data, indicating a risk of abnormal behavior. Potentially abnormal samples characterize the operational state of a vehicle that deviates from the normal driving mode under specific spatiotemporal conditions.

[0027] Visual feature data refers to the set of digital information extracted from vehicle video data that characterizes the movement trajectory and behavioral features of vehicle targets, including but not limited to the vehicle target's movement trajectory, position coordinates, speed estimates, acceleration estimates, dwell time records, crossing behavior descriptions, distance values ​​between the vehicle and objects in front, and object category information.

[0028] Spatiotemporal alignment processing: Based on timestamp information and vehicle geographic location information, potential abnormal samples and visual feature data from different acquisition devices are matched and integrated in the time and spatial dimensions to ensure that potential abnormal samples and visual feature data point to the same vehicle at the same time.

[0029] Multi-source spatiotemporal database: refers to a heterogeneous data fusion storage structure constructed after spatiotemporal alignment processing. It contains the correlation information of vehicle operation data and video data in a unified spatiotemporal coordinate system, that is, potential abnormal samples and visual feature data pointing to the same vehicle at the same time, providing complete data support for subsequent abnormal event analysis.

[0030] Abnormal event classification system: refers to a structured knowledge framework that systematically defines and classifies abnormal behaviors of autonomous vehicles, including categories such as unexplained deceleration events, unexplained braking events, and traffic accident events, as well as the corresponding judgment conditions and judgment thresholds for each category.

[0031] Multimodal dataset: refers to a data set formed by manually or automatically labeling each event in a multi-source spatiotemporal database based on an anomaly event classification system. It includes input features and corresponding anomaly event category labels and is used for model training.

[0032] Knowledge-enhanced large models: These are deep learning models based on the Transformer architecture, trained on multimodal datasets labeled with an anomaly event classification system. The models learn the dynamic features of anomaly events through a self-attention mechanism and optimize their parameters, thus possessing the ability to capture long-distance temporal dependencies.

[0033] Abnormal behavior detection model: refers to a knowledge-enhanced large model that has been trained and can receive the behavior data of the vehicle to be detected as input and output the corresponding abnormal behavior category judgment.

[0034] Vehicle behavior data to be detected: refers to real-time vehicle operation data and video data that need to be input into the abnormal behavior detection model for analysis. The data format is consistent with the multimodal dataset used in the training phase.

[0035] Abnormal behavior detection result: refers to the classification conclusion output by the abnormal behavior detection model after reasoning through the behavior data of the vehicle to be detected, indicating whether the vehicle has abnormal behaviors such as deceleration without reason, braking without reason, or traffic accidents.

[0036] GIS spatial matching technology refers to a spatial analysis method in geographic information systems. It uses a nearest neighbor search algorithm to calculate the spatial distance between a vehicle's geographic location information and the intersection location information stored in the intersection database, and maps the vehicle's GPS coordinates corresponding to the vehicle's geographic location information to a specific road intersection or road segment.

[0037] Intersection database: refers to a pre-built data set containing geographical location information and topological relationships of urban road intersections, used to provide a benchmark for spatial matching.

[0038] Traffic scenario: refers to the specific road environment type where the vehicle is currently located, which is determined by GIS spatial matching technology based on the vehicle's geographical location information, including but not limited to different types of road areas such as crossroads, T-junctions, or straight road sections.

[0039] Cluster analysis: refers to the unsupervised classification process of vehicle operation data using the K-means algorithm. By calculating the similarity of features such as speed, acceleration, and dwell time, the data is divided into different behavioral pattern clusters, thereby identifying the distribution boundaries between normal driving patterns and abnormal behavior patterns.

[0040] Abnormal behavior judgment threshold: refers to the critical value used to distinguish between normal and abnormal behavior, calculated through cluster analysis, including but not limited to the acceleration change amplitude threshold and the speed change range threshold. The above thresholds are set differently according to the data distribution characteristics of different traffic scenarios.

[0041] Target recognition and localization: refers to the process of automatically identifying vehicle targets in video frames and determining their pixel coordinates using deep learning target detection algorithms.

[0042] Target association between consecutive frames: This refers to calculating the feature similarity of vehicle targets between adjacent frames in a continuous video frame sequence using the Hungarian algorithm, establishing the position correspondence of the same vehicle at different times, thereby achieving continuous tracking of motion trajectories.

[0043] Multi-target tracking result: refers to the continuous position sequence of multiple vehicle targets on the video timeline obtained by associating targets in consecutive frames, forming a complete record of the motion trajectory of multiple vehicle targets.

[0044] Unexplained deceleration event: refers to the behavior category defined in the abnormal event classification system. The judgment condition is that when there are no obstacles within the preset driving distance in the driving direction, the vehicle continuously decelerates and the deceleration magnitude exceeds the preset acceleration threshold.

[0045] Unexplained braking event: refers to the behavior category defined in the abnormal event classification system. Its judgment condition is that when there are no obstacles within a preset driving distance in the direction of travel, the speed drops to zero and is accompanied by a continuous deceleration process.

[0046] Traffic accident incident: refers to the behavioral category defined in the abnormal event classification system. Its judgment condition is that the distance between the vehicle and the obstacle in the direction of travel is reduced to less than the preset collision threshold, accompanied by the characteristic behavior of violent deceleration.

[0047] Preset driving distance: refers to the spatial range parameter set in the abnormal event classification system for detecting obstacles ahead. It is usually calculated and determined based on the vehicle's current speed and safe braking distance, and can be dynamically adjusted based on the linear relationship between vehicle speed and safe following distance.

[0048] Preset acceleration threshold: refers to the critical value of acceleration change used to determine whether deceleration behavior constitutes abnormality. It is obtained through cluster analysis and represents the boundary standard between normal deceleration and abnormal deceleration.

[0049] Preset collision threshold: refers to the critical distance value used to determine whether a vehicle has been involved in a collision. It is usually set to a minimum distance value that is zero or close to zero, and represents the standard for determining whether a vehicle has made physical contact with an obstacle.

[0050] In another embodiment of this solution, S1 is specifically implemented as follows: Figure 2 This is a diagram illustrating data acquisition and processing, such as... Figure 2 As shown, vehicle operation data and vehicle video data are acquired. The vehicle operation data includes vehicle geographical location information, instantaneous speed, acceleration value, dwell time and timestamp information. The vehicle video data is a dynamic video stream obtained by continuously shooting the monitoring area through image acquisition devices deployed along the road.

[0051] Figure 3 This is a schematic diagram of the overall technical roadmap, such as... Figure 3As shown in the "Data Feature Extraction and Fusion" section, for vehicle operation data, based on Geographic Information System (GIS) spatial matching technology, the Nearest Neighbor Search algorithm is used to match the vehicle's geographical location information in the vehicle operation data with the intersection location information in the intersection database to determine the traffic scenario in which the vehicle is located. The traffic scenario includes different types of road areas such as crossroads, T-junctions, or straight road sections. Based on the traffic scenario, the vehicle operation data is divided, and the K-means clustering algorithm is used to perform cluster analysis on key indicators such as speed, acceleration, and dwell time in the vehicle operation data under different traffic scenarios to identify potential distribution patterns of abnormal behavior, obtain potential abnormal vehicle behaviors and corresponding abnormal behavior judgment thresholds; all potential abnormal vehicle behaviors and corresponding abnormal behavior judgment thresholds are then filtered to obtain potential abnormal samples.

[0052] Cluster analysis includes: identifying potential distribution patterns of normal driving patterns and abnormal behaviors by analyzing the cluster center location and sample distribution density; calculating the judgment threshold of potential abnormal vehicle behaviors based on the boundary distance between the normal driving pattern cluster and the abnormal behavior pattern cluster or based on the statistical characteristics of the normal driving pattern cluster; and filtering data records that deviate from the center of the normal driving pattern cluster by more than a preset distance or meet the abnormal judgment threshold as potential abnormal samples.

[0053] For vehicle video data, the YOLOv11 target detection model is used for real-time target recognition and localization of video frames. The YOLOv11 target detection model is a lightweight real-time detection model based on an improved cross-stage local Darknet (CSPDarknet) backbone network and incorporating a bidirectional feature pyramid network (BiFPN) structure. It features high accuracy and low latency. Based on target recognition and localization, the Deep Simple Online Real-Time Tracking (DeepSORT) algorithm is introduced, and the Hungarian algorithm is used to correlate targets between consecutive frames, achieving multi-target tracking. Through this process, multiple vehicle targets can be continuously tracked, and their complete motion trajectories in the vehicle video data can be extracted, resulting in multi-target tracking results. Based on these results, behavioral features such as speed, acceleration, dwell time, and crossing characteristics of the corresponding vehicle targets are extracted and combined with the multi-target tracking results to form visual feature data.

[0054] In another embodiment of this solution, S2 is specifically implemented as follows: Spatiotemporal alignment processing is performed on potential anomalous samples and visual feature data.

[0055] In the time dimension, timestamp information is used as the key benchmark for time dimension alignment. Potential abnormal samples from vehicle operation data and visual feature data from road monitoring videos are accurately matched on the time axis to ensure that vehicle operation data and vehicle video data collected at the same time establish a time correlation.

[0056] In the spatial dimension, the spatial coordinates of potential anomaly samples are matched with the spatial locations in the visual feature data based on vehicle geographic location information, thus establishing a spatial correspondence between the vehicle geographic location information corresponding to the vehicle operation data and the spatial scene captured by the vehicle video data. Specifically, the spatial coordinates of potential anomaly samples are obtained from the vehicle geographic location information in the vehicle operation data, while the spatial locations in the visual feature data are obtained from the spatial locations of the image acquisition devices.

[0057] Through the dual matching of the time and spatial dimensions described above, a multi-source spatiotemporal database (corresponding to) is obtained. Figure 2 The "multi-source heterogeneous spatiotemporal database" provides cross-modal data support for subsequent anomaly identification and model training. Each event in the multi-source spatiotemporal database corresponds to the fusion result of the operational and visual features of the same vehicle under the same spatiotemporal coordinates. Each event includes a global vehicle identifier, timestamp information, geographic location coordinates, operational features, and visual features. Among them, the operational features are derived from key indicators in the clustering analysis process.

[0058] In another embodiment of this solution, S3 is specifically implemented as follows: like Figure 3 As shown in the "Definition and Classification of Abnormal Events" section, an abnormal event classification system should be established. The establishment of the abnormal event classification system can adopt a combination of inductive analysis and expert judgment to systematically define and classify abnormal events.

[0059] In this embodiment, combining research findings and management experience, common abnormal behavior types were identified and summarized through manual analysis of video data from a large number of typical events, including unexplained deceleration, unexplained braking, and traffic accidents. Based on this, the criteria for determining each type of abnormal event were clarified, establishing a structured and operable abnormal event classification system.

[0060] Specifically, the abnormal event classification system includes: unexplained deceleration events, unexplained braking events, and traffic accident events.

[0061] For unexplained deceleration events, the determination criteria are set as follows: the vehicle continuously decelerates within a preset driving distance in the direction of travel when there are no obstacles in front, and the deceleration exceeds a preset acceleration threshold. The determination process involves key variables such as acceleration value and whether there is an object in front.

[0062] For unexplained braking events, the determination condition is set as follows: the vehicle decelerates to a stop within a preset driving distance in the direction of travel when there are no obstacles in front, and the deceleration process is continuous. The determination process involves key variables such as vehicle speed, acceleration, and whether there is an object in front.

[0063] For traffic accident incidents, the criteria for judgment are set as follows: the distance between the vehicle and an obstacle in the direction of travel is less than a preset collision threshold, accompanied by a sharp deceleration. The judgment process involves key variables such as distance value, approach speed value, and object category. It should be noted that the key variables are used to transform abstract behavioral descriptions into calculable and discriminable numerical standards or status indicators, thereby achieving automated identification of abnormal events.

[0064] The abnormal event classification system clarifies the specific judgment conditions and key variables for each type of abnormal event, forming a structured and operable abnormal event judgment model, which provides a classification basis and technical support for the subsequent design of anomaly detection algorithms.

[0065] Each event in the multi-source spatiotemporal database is labeled based on an anomaly event classification system. The specific process is as follows: For each event in the multi-source spatiotemporal database, key variables related to the anomaly event determination are extracted. These include acceleration values ​​and instantaneous speeds extracted from vehicle operation data, and distance values, approach speed values, and object category information between the vehicle and objects ahead extracted from visual feature data. The extracted key variables are then substituted into the judgment conditions of the anomaly event classification system for logical operations to determine the anomaly event type label corresponding to each event. If the unexplained deceleration judgment condition is met, the event is labeled as an unexplained deceleration event; if the unexplained braking judgment condition is met, the event is labeled as an unexplained braking event; if the traffic accident judgment condition is met, the event is labeled as a traffic accident event; if none of these conditions are met, the event is labeled as normal driving.

[0066] Each event, which includes timestamp information, geographic location information, operational characteristics, visual features, and the aforementioned abnormal event type labels, is organized as a case. Each case contains observation records from multiple time steps. Each observation record integrates multi-source feature information extracted from vehicle operation data and vehicle video data to form a multimodal dataset for subsequent training of large knowledge-enhanced models.

[0067] In another embodiment of this solution, S4 is specifically implemented as follows: like Figure 3 As shown in the “Anomaly Detection Algorithm Design”, the multimodal dataset obtained in S3 is divided into a training set and a test set according to a preset ratio.

[0068] A knowledge-enhanced large-scale model is trained based on a multimodal dataset. This model mainly consists of four parts: input representation, self-attention mechanism for modeling temporal dependencies, event-level representation construction, and loss function and training objective settings. Specifically, the knowledge-enhanced large-scale model employs an improved Transformer model, mapping the input features at each time step to vector representations. It utilizes a self-attention mechanism to model long-distance dependencies, effectively capturing the dynamic evolution of anomalous events over time and the correlation patterns between multi-source features. A positional encoding mechanism preserves the sequential information of the time series, and event-level representation construction enables the prediction of anomalous event labels. By setting the loss function and training objective and adjusting the model parameters through optimization algorithms, an anomalous behavior detection model is obtained.

[0069] The abnormal behavior detection model is validated and evaluated using a test set. Confusion matrix, accuracy, recall, precision, and F1 score are calculated to examine the model's performance in identifying various abnormal events. By comparing the performance differences between traditional machine learning models and deep learning models, the advantages of the method used in this embodiment in terms of accuracy and stability in abnormal event detection are verified.

[0070] In another embodiment of this solution, S5 is specifically implemented as follows: The system acquires vehicle behavior data, which includes real-time vehicle operation data and vehicle video data. The vehicle operation data includes vehicle location information, speed information, acceleration information, and timestamp information, while the vehicle video data is road monitoring video data for the corresponding time period.

[0071] The same processing steps as steps S1 and S2 are performed on the vehicle behavior data to be detected to extract the input features to be detected. The input features to be detected refer to the set of feature vectors extracted from the vehicle behavior data to be detected and processed, and the feature format is consistent with that in the multimodal dataset used in the training phase.

[0072] The input features to be detected are input into the abnormal behavior detection model. The abnormal behavior detection model performs calculations on the input features based on the learned parameters and outputs a prediction result about the type of abnormal event. The abnormal behavior detection result is determined based on the prediction result. The abnormal behavior detection result includes a judgment conclusion that the vehicle behavior belongs to a category such as normal driving, unexplained deceleration event, unexplained braking event, or traffic accident event.

[0073] Furthermore, the vehicle operation data is analyzed, including: Based on GIS spatial matching technology, the vehicle's geographical location information in the vehicle operation data is matched with the intersection location information in the intersection database to determine the traffic scenario in which the vehicle is located. Vehicle operation data is divided according to traffic scenarios, and cluster analysis is performed on vehicle operation data under different traffic scenarios to obtain potential abnormal vehicle behaviors and corresponding abnormal behavior judgment thresholds. All potential abnormal behaviors of vehicles and their corresponding abnormal behavior judgment thresholds are screened to obtain potential abnormal samples.

[0074] Furthermore, feature extraction is performed on the vehicle video data, including: Target identification and localization are performed on video frames in vehicle video data, and targets in consecutive frames are correlated to obtain multi-target tracking results; the multi-target tracking results include the motion trajectories of multiple vehicle targets; Behavioral features of multiple vehicle targets are extracted based on the multi-target tracking results, and these features are combined with the multi-target tracking results to form visual feature data.

[0075] Furthermore, the abnormal event classification system includes: unexplained deceleration events, unexplained braking events, and traffic accident events; The criteria for determining an unexplained deceleration event include: the vehicle continuously decelerates within a preset travel distance in the direction of travel when there are no obstacles, and the deceleration magnitude exceeds a preset acceleration threshold. The criteria for determining an unexcused braking event include: the vehicle decelerates to a stop when there are no obstacles within a preset travel distance in the direction of travel, and this deceleration is accompanied by a continuous deceleration process; The criteria for determining a traffic accident include: the distance between the vehicle and an obstacle in the direction of travel is less than a preset collision threshold and is accompanied by severe deceleration.

[0076] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, and these situations are also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0077] Furthermore, the acquisition process of the data involved in this application follows the principles of legality, legitimacy, and necessity. Based on obtaining the explicit authorization and consent of the user, only the minimum necessary information required to achieve the purpose is collected, and data security protection obligations are fulfilled in accordance with the law.

[0078] The present invention also provides an abnormal behavior detection system for autonomous vehicles based on a knowledge-enhanced large model. The specific technical solution is as follows: a data acquisition module, a spatiotemporal alignment module, a classification and calibration module, a knowledge enhancement module, and an anomaly detection module. The data acquisition module is used to acquire vehicle operation data and vehicle video data, analyze the vehicle operation data to obtain potential abnormal samples, and extract features from the vehicle video data to obtain visual feature data. The spatiotemporal alignment module is used to perform spatiotemporal alignment processing on potential abnormal samples and visual feature data to obtain a multi-source spatiotemporal database. The classification and labeling module is used to establish an abnormal event classification system, and to label each event in the multi-source spatiotemporal database based on the abnormal event classification system to obtain a multimodal dataset; The knowledge enhancement module is used to train a large knowledge-enhanced model based on a multimodal dataset to obtain an abnormal behavior detection model; The anomaly detection module is used to input the behavior data of the vehicle to be detected into the anomaly behavior detection model and obtain the anomaly behavior detection results.

[0079] It should be noted that the beneficial effects of the knowledge-enhanced large-scale autonomous vehicle behavior detection system provided in the above embodiments are the same as those of the knowledge-enhanced large-scale autonomous vehicle behavior detection method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0080] like Figure 4 As shown, an embodiment of the present invention provides a computer device 300, which includes a processor 320 coupled to a memory 310. The memory 310 stores at least one computer program 330, which is loaded and executed by the processor 320 to enable the computer device 300 to implement any of the above-described methods. Specifically: The computer device 300 can vary considerably due to differences in configuration or performance. It may include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310. The one or more memories 310 store at least one computer program 330, which is loaded and executed by the one or more processors 320 to enable the computer device 300 to implement the knowledge-enhanced large model-based abnormal behavior detection method for autonomous vehicles provided in the above embodiments. Of course, the computer device 300 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device 300 may also include other components for implementing device functions, which will not be elaborated upon here.

[0081] An embodiment of the present invention provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above-described methods.

[0082] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0083] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the above-described knowledge-enhanced large model-based methods for detecting abnormal behavior of autonomous vehicles.

[0084] It should be noted that the terms "first," "second," etc., used in the specification of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown in the figures or description.

[0085] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0086] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0087] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model, characterized in that, include: S1, acquire vehicle operation data and vehicle video data, analyze the vehicle operation data, and obtain potential abnormal samples; Visual feature data is obtained by extracting features from the vehicle video data. S2, perform spatiotemporal alignment processing on the potential abnormal samples and the visual feature data to obtain a multi-source spatiotemporal database; S3, establish an abnormal event classification system, and label each event in the multi-source spatiotemporal database based on the abnormal event classification system to obtain a multimodal dataset; S4. Train a knowledge-enhanced large model based on the multimodal dataset to obtain an abnormal behavior detection model; S5, input the vehicle behavior data to be detected into the abnormal behavior detection model to obtain the abnormal behavior detection result.

2. The method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model according to claim 1, characterized in that, The analysis of the vehicle operation data includes: Based on GIS spatial matching technology, the vehicle's geographical location information in the vehicle operation data is matched with the intersection location information in the intersection database to determine the traffic scenario in which the vehicle is located. The vehicle operation data is divided according to the traffic scenario, and the vehicle operation data under different traffic scenarios is clustered to obtain potential abnormal vehicle behaviors and corresponding abnormal behavior judgment thresholds. The potential abnormal behaviors of all vehicles and their corresponding abnormal behavior judgment thresholds are screened to obtain the potential abnormal samples.

3. The method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model according to claim 1, characterized in that, The feature extraction of the vehicle video data includes: Target identification and localization are performed on video frames in the vehicle video data, and targets in consecutive frames are correlated to obtain multi-target tracking results; the multi-target tracking results include the motion trajectories of multiple vehicle targets; Based on the multi-target tracking results, behavioral features of multiple vehicle targets are extracted and combined with the multi-target tracking results to form the visual feature data.

4. The method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model according to claim 1, characterized in that, The abnormal event classification system includes: unexplained deceleration events, unexplained braking events, and traffic accident events; The criteria for determining the unexplained deceleration event include: the vehicle continuously decelerates within a preset travel distance in the direction of travel when there are no obstacles and the deceleration magnitude exceeds a preset acceleration threshold. The criteria for determining the unexcused braking event include: the vehicle decelerates to a stop when there are no obstacles within a preset travel distance in the direction of travel, and this deceleration is accompanied by a continuous deceleration process; The criteria for determining a traffic accident include: the distance between the vehicle and an obstacle in the direction of travel is less than a preset collision threshold and is accompanied by severe deceleration.

5. An abnormal behavior detection system for autonomous vehicles based on a knowledge-enhanced large model, characterized in that, include: The module includes a data acquisition module, a spatiotemporal alignment module, a classification and labeling module, a knowledge enhancement module, and an anomaly detection module. The data acquisition module is used to acquire vehicle operation data and vehicle video data, analyze the vehicle operation data, and obtain potential abnormal samples. Visual feature data is obtained by extracting features from the vehicle video data. The spatiotemporal alignment module is used to perform spatiotemporal alignment processing on the potential abnormal samples and the visual feature data to obtain a multi-source spatiotemporal database. The classification and labeling module is used to establish an abnormal event classification system, and to label each event in the multi-source spatiotemporal database based on the abnormal event classification system to obtain a multimodal dataset; The knowledge enhancement module is used to train a large knowledge enhancement model based on the multimodal dataset to obtain an abnormal behavior detection model. The anomaly detection module is used to input the behavior data of the vehicle to be detected into the anomaly behavior detection model to obtain the anomaly behavior detection result.

6. The abnormal behavior detection system for autonomous vehicles based on a knowledge-enhanced large model according to claim 5, characterized in that, The analysis of the vehicle operation data includes: Based on GIS spatial matching technology, the vehicle's geographical location information in the vehicle operation data is matched with the intersection location information in the intersection database to determine the traffic scenario in which the vehicle is located. The vehicle operation data is divided according to the traffic scenario, and the vehicle operation data under different traffic scenarios is clustered to obtain potential abnormal vehicle behaviors and corresponding abnormal behavior judgment thresholds. The potential abnormal behaviors of all vehicles and their corresponding abnormal behavior judgment thresholds are screened to obtain the potential abnormal samples.

7. The abnormal behavior detection system for autonomous vehicles based on a knowledge-enhanced large model according to claim 5, characterized in that, The feature extraction of the vehicle video data includes: Target identification and localization are performed on video frames in the vehicle video data, and targets in consecutive frames are correlated to obtain multi-target tracking results; the multi-target tracking results include the motion trajectories of multiple vehicle targets; Based on the multi-target tracking results, behavioral features of multiple vehicle targets are extracted and combined with the multi-target tracking results to form the visual feature data.

8. The abnormal behavior detection system for autonomous vehicles based on a knowledge-enhanced large model according to claim 5, characterized in that, The abnormal event classification system includes: unexplained deceleration events, unexplained braking events, and traffic accident events; The criteria for determining the unexplained deceleration event include: the vehicle continuously decelerates within a preset travel distance in the direction of travel when there are no obstacles and the deceleration magnitude exceeds a preset acceleration threshold. The criteria for determining the unexcused braking event include: the vehicle decelerates to a stop when there are no obstacles within a preset travel distance in the direction of travel, and this deceleration is accompanied by a continuous deceleration process; The criteria for determining a traffic accident include: the distance between the vehicle and an obstacle in the direction of travel is less than a preset collision threshold and is accompanied by severe deceleration.

9. A computer device, characterized in that, The computer device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the computer device to implement a method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to implement a method for detecting abnormal behavior of autonomous vehicles based on a knowledge-enhanced large model as described in any one of claims 1 to 4.

Images