Vehicle behavior prediction method, system and device and storage medium
By introducing CBAM and BIFPN structures in YOLOv5s, combined with the Drools inference engine, the problem of insufficient detection capabilities in vehicle behavior prediction is solved, the accuracy and interpretability of vehicle behavior prediction is improved, and the vehicle behavior prediction in complex scenarios is realized .
Patent Information
- Application Number
- CN202411184627.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-15
AI Technical Summary
The existing vehicle behavior prediction methods have limited ability to detect key information in complex scenarios and cannot comprehensively consider the relationship between various factors, resulting in inaccurate vehicle behavior prediction.
By introducing CBAM attention mechanism and BIFPN structure in the object detection model YOLOv5s, the weight of brake light information in the image is enhanced, and a predictive ontology model is constructed with the Drools inference engine to identify the vehicle's brake light status and behavioral intention, and predict the future behavior dynamics of the vehicle. .
It improves the accuracy and interpretability of vehicle behavior prediction, enhances the small target recognition ability in complex scenarios, and realizes accurate extraction of front-line light signals and reliable prediction of vehicle behavior.
Smart Images

Figure CN118968436A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of autonomous driving technology, and specifically relates to a vehicle behavior prediction method, system, device and storage medium. Background Art
[0002] Based on the progress of science and technology and people's pursuit of safer and more convenient transportation, self-driving cars have gradually become the future development direction of automobile technology. In self-driving cars, vehicle behavior prediction, as one of its core technologies, is of great significance in improving the safety of self-driving cars, optimizing path planning and decision-making basis, and improving the level of intelligence.
[0003] Since vehicle light signal detection is critical to improving road safety and autonomous driving system decision-making, many researchers predict vehicle behavior by detecting and identifying the taillights of the vehicle in front. Some researchers use the YOLOv3 target detection algorithm to identify the taillights of the vehicle in front. However, this method has limited recognition effect on small targets in complex scenes, so the recognition effect of the taillights of the vehicle in front is not accurate enough.
[0004] In addition, predicting the behavior of a vehicle is not a simple task. First, the behaviors of vehicles are interdependent, and the behavior of one vehicle may affect its specific vehicle, and vice versa. Therefore, predicting the behavior of a vehicle requires considering the behavior of surrounding cars. Second, the changing roads and restrictions of traffic rules can reshape the behavior of a vehicle, so the prediction of vehicle behavior cannot only consider a specific driving environment. Third, there are multiple possible future behaviors of a vehicle, which means that even given the motion history of a vehicle, it may have more than one future behavior. For example, when a vehicle slows down at an intersection without changing its direction of travel, the vehicle may also turn. Therefore, the behavior prediction module in an autonomous vehicle should identify all possible future movements so that the car can work reliably.
[0005] In summary, since vehicle behavior prediction is a complex process, existing prediction methods have limited ability to detect key information in complex scenarios and cannot comprehensively consider the relationship between various factors during prediction, resulting in inaccurate vehicle behavior prediction. Summary of the invention
[0006] In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a vehicle behavior prediction method, comprising the following steps:
[0007] Obtain video data of the driving scene to be tested;
[0008] The video data of the driving scene to be tested is input into the improved YOLOv5s, and the CBAM attention mechanism is used to increase the brake light information weight of the input image; the BIFPN structure is used to fuse the feature maps of different levels in the image, and the brake light information weight is introduced at each fusion node to obtain the brake light status of the target vehicle; the CBAM attention mechanism is introduced before the SPPF module of the backbone network of the target detection model YOLOv5s, and the PANET structure of the neck of the target detection model YOLOv5s is replaced with the BIFPN structure to obtain the improved YOLOv5s;
[0009] Define multiple ontology types related to vehicle behavior prediction in driving scenarios, determine the relationship between multiple ontology types, and build a prediction ontology model based on multiple ontology types and the relationship between multiple ontology types;
[0010] According to the brake light status and vehicle behavior reasoning rules, corresponding instances are created in the prediction ontology model, and the Drools reasoning engine is used for reasoning to predict the future behavior dynamics of the target vehicle.
[0011] Preferably, the entity type includes a target vehicle, a driving environment, a brake light status, and a maneuvering intention; the driving environment includes a driving lane and a traffic light status; and the maneuvering intention includes a turning intention and a speed change.
[0012] Preferably, the relationship between the multiple entity types is specifically as follows: the target vehicle is associated with the driving environment, indicating the specific driving conditions of the vehicle; the target vehicle is associated with the brake light status, indicating the vehicle's immediate intention to decelerate or stop; the target vehicle is associated with the maneuvering intention, indicating the vehicle's future driving direction and speed changes.
[0013] Preferably, the target vehicle inference rule is as follows:
[0014] Rule 1: If the target vehicle's brake lights are on, then the vehicle is slowing down;
[0015] Rule 2: If the target vehicle's brake lights are not on, the vehicle is accelerating or traveling at the original speed;
[0016] Rule 3: If the target vehicle is driving in the leftmost lane, the vehicle will turn left;
[0017] Rule 4: If the target vehicle is in the rightmost lane, the vehicle will turn right;
[0018] Rule 5: If the target vehicle is in the middle lane, the vehicle will go straight;
[0019] Rule 6: If the target vehicle is decelerating and the red light is on, the vehicle will decelerate to a stop;
[0020] Rule 7: If the target vehicle is slowing down and the green light is on, then the vehicle will slow down;
[0021] Rule 8: If the target vehicle is slowing down and the yellow light is on, the vehicle will slow down to a stop.
[0022] Preferably, before inputting the video data of the driving scene to be tested into the improved YOLOv5s, the method further includes training the improved YOLOv5s, specifically: collecting video data of the target vehicle in different time periods and different road conditions, and constructing a data set, dividing the data set into a training set, a validation set and a test set in a ratio of 8:1:1, using the training set to train the parameters of the improved YOLOv5s; using the validation set to tune the model hyperparameters; and using the test set to evaluate the accuracy of the improved YOLOv5s.
[0023] Preferably, before training the improved YOLOv5s, the method further includes filtering the data in the data set, annotating the filtered data, and drawing a bounding box for each target object.
[0024] The present invention also provides a vehicle behavior prediction system, comprising:
[0025] A data acquisition module, used to obtain video data of the driving scene to be tested;
[0026] The brake light status recognition module is used to input the video data of the driving scene to be tested into the improved YOLOv5s, and use the CBAM attention mechanism to increase the brake light information weight of the input image; use the BIFPN structure to fuse the feature maps of different levels in the image, and introduce the brake light information weight at each fusion node to obtain the brake light status of the target vehicle; among them, the CBAM attention mechanism is introduced before the SPPF module of the backbone network of the target detection model YOLOv5s, and the PANET structure of the neck of the target detection model YOLOv5s is replaced with the BIFPN structure to obtain the improved YOLOv5s
[0027] A prediction ontology model building module is used to define multiple ontology types related to vehicle behavior prediction in driving scenarios, determine the relationship between multiple ontology types, and build a prediction ontology model based on multiple ontology types and the relationship between multiple ontology types;
[0028] The vehicle behavior prediction module is used to create corresponding instances in the prediction ontology model according to the brake light status and vehicle behavior reasoning rules, and use the Drools reasoning engine for reasoning to predict the future behavior dynamics of the target vehicle.
[0029] The present invention also provides a computer device, comprising a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the vehicle behavior prediction method.
[0030] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the vehicle behavior prediction method.
[0031] The vehicle behavior prediction method, system, device and storage medium provided by the present invention have the following beneficial effects:
[0032] The present invention introduces the CBAM attention mechanism before the SPPF module of the backbone network of the target detection model YOLOv5s, which can increase the weight of the brake light information in the input image, thereby enhancing the feature extraction and detection capabilities of the model in the recognition of complex scenes and small targets. By replacing the PANet in YOLOv5s with the BIFPN, the multi-level feature information can be better utilized, thereby more accurately detecting targets of different sizes and scales, and realizing accurate extraction of the front vehicle light signal.
[0033] The present invention defines multiple ontology types related to vehicle behavior prediction in driving scenarios, determines the relationship between the multiple ontology types, and constructs a prediction ontology model based on the multiple ontology types and the relationship between the multiple ontology types, thereby performing a structured representation of the entities, attributes and relationships in the light language information; the present invention can enhance the interpretability and reliability of the prediction results and improve the accuracy of vehicle behavior prediction by combining with an ontology reasoning engine. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 A flow chart of a vehicle behavior prediction method according to an embodiment of the present invention;
[0036] Figure 2 It is the structure of the improved YOLOv5s;
[0037] Figure 3 Flowchart of the channel attention module;
[0038] Figure 4 Flowchart of the spatial attention module;
[0039] Figure 5 This is a comparison chart of the sampling methods of PANet and BIFPN; Figure 5 (a) and Figure 5 (b) are the sampling methods of PANet network and BIFPN structure respectively;
[0040] Figure 6 Predict the structure diagram of the entity for the preceding vehicle behavior;
[0041] Figure 7 is the brake light test result; among them, Figure 7 (a) Figure 7 (b) Figure 7 (c) Figure 7 (d) Figure 7 (e) and Figure 7 (f) is the detection result of 6 consecutive frames of video;
[0042] Figure 8 is the detection result of the situation including the turn signal; Figure 8 (a) Figure 8 (d) refers to the situation where only the brake lights are on; Figure 8 (b), Figure 8 (e) refers to the situation where the brake lights and turn signals are on at the same time; Figure 8 (c) and Figure 8 (f) is when only the turn signal is on;
[0043] Fig. 9 Schematic diagram created for the example;
[0044] Fig.10 The ontology model reasoning results; Fig.10 (a) Fig.10 (b) and Fig.10 (c) is the reasoning result at different times. DETAILED DESCRIPTION
[0045] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.
[0046] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the technical solutions of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0047] In addition, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified or limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In the description of the present invention, unless otherwise specified, "plurality" means two or more, which will not be described in detail here.
[0048] Example
[0049] The present invention provides a vehicle behavior prediction method, specifically Figure 1 As shown, the following steps are included:
[0050] Step 1: Obtain video data of the driving scene to be tested.
[0051] Step 2: Introduce the CBAM attention mechanism before the SPPF module of the backbone network of the target detection model YOLOv5s, replace the PANET structure of the target detection model YOLOv5s neck with the BIFPN structure, and obtain the improved YOLOv5s.
[0052] As an intuitive way of communication between vehicles, light information contains rich information about driving intentions and status, providing new possibilities for predicting the behavior of the vehicle ahead. However, traditional methods of detecting and recognizing light information are often limited by manually defined rules and features, making it difficult to adapt to complex and changing road environments. In recent years, the rapid development of deep learning technology has provided new ideas for the automatic detection and recognition of light information.
[0053] The YOLO series has become the most popular target recognition method in practical applications due to its excellent recognition accuracy and speed.
[0054] Considering the real-time requirements of the application, the present invention selects YOLOv5s as the basic model. YOLOv5s is the smallest model in the YOLOv5 series. It uses the CSPDarknet structure as the Backbone, combined with FPN (feature pyramid network) and PANet (path aggregation network) as the Neck part. The detection head of YOLOv5s includes three detection layers of different scales, each of which predicts the category and location information of the object. This design enables YOLOv5s to effectively detect objects of different sizes at multiple scales. Although it saves a lot of calculations, if no improvements are made, vehicle intention prediction based on deep learning and ontology reasoning may not achieve the ideal detection accuracy. In response to this problem, the present invention proposes two improvements to the YOLOv5s model to further improve the accuracy of vehicle taillight detection. (1) In order to enhance the model's feature extraction and detection capabilities in the recognition of complex scenes and small targets, the CBAM attention mechanism is introduced in the backbone network. (2) In order to balance the feature information of different scales and improve the detection performance of multi-scale targets, the present invention replaces the PANet network in the original network and introduces the BIFPN network. By adopting this structure based on the feature pyramid network, it is possible to better utilize the multi-level feature information, thereby more accurately detecting targets of different sizes and scales. The improved network structure is shown in FIG. Figure 2 shown.
[0055] Step 3: Input the video data of the driving scene to be tested into the improved YOLOv5s, and use the CBAM attention mechanism to increase the brake light information weight of the input image; use the BIFPN structure to fuse the feature maps of different levels in the image, and introduce the brake light information weight at each fusion node to obtain the brake light status of the target vehicle.
[0056] When performing brake light detection, it is often necessary to process images under different lighting conditions, different perspectives, and different occlusion conditions. Embedding the CBAM module in the YOLOv5s network can effectively improve network performance by increasing the weight of brake light information, allowing the model to pay more attention to the target-related areas in the image, while suppressing other useless information and reducing the brake lights being blocked by other objects or interfered with by other light sources. The CBAM module uses two sub-modules, the channel attention module and the spatial attention module, to weight the input feature map in the channel dimension and spatial dimension respectively, and derives a one-dimensional channel attention map and a two-dimensional spatial attention map in turn. The entire attention mechanism process can be summarized as follows:
[0057]
[0058] Where, F is the input feature map, M c is a one-dimensional channel attention map, M sis a two-dimensional spatial attention map, represents element-by-element multiplication, F' is the result of element-by-element multiplication of the one-dimensional channel attention map and the input feature map, and F" is the final refined output of the CBAM module.
[0059] Figure 3 The workflow of the channel attention module is shown. The input feature map F(H×W×C) is first aggregated through average pooling and maximum pooling operations to generate the average pooled feature F c avg And the maximum pooling feature Fc max Next, the results of average pooling and maximum pooling are processed using a shared fully connected layer (Shared MLP), first through a fully connected layer to reduce the number of channels, and then through another fully connected layer to restore the number of channels. Then, the results obtained by the shared fully connected layer are added and the Sigmoid activation function is used to generate the final channel attention map M. c Finally, the weights are multiplied channel by channel to the input feature layer to generate the input features required by the spatial attention module. The channel attention module can be summarized as follows:
[0060]
[0061] Where σ represents the Sigmoid activation function, W0 and W1 are the operation weights of MLP, They are the average pooling results and the maximum pooling results of the channel attention module respectively.
[0062] Figure 4 The workflow of the spatial attention module is shown. The spatial attention module uses the feature map F' output by the above CBAM module as the input feature map of this module. First, the input feature map is subjected to maximum pooling and average pooling in the channel dimension to obtain two 1×H×W feature maps. The two pooled feature maps are stacked in the channel dimension to obtain a 2×H×W feature map. Then, a 7×7 convolution operation is performed to obtain a 1×H×W feature map again. Finally, the convolution result is normalized by the sigmoid function to obtain the spatial weight of the feature map to obtain the spatial attention map M. s , and then multiply the input feature map and the weight to obtain a feature map of the same size as the input feature map and containing channel and spatial attention. The channel attention module can be summarized as follows:
[0063]
[0064] In the formula, f 7×7 represents a 7×7 convolution kernel, They are the average pooling and maximum pooling results of the spatial attention module, respectively.
[0065] Figure 5 (a) and Figure 5 (b) are the sampling methods of the PANet network and the BIFPN structure respectively. In the fusion process of the PANet network, all information comes from the processing results of the previous stage. However, this design does not fully utilize the information of the original features, especially for the extraction of small-scale target features, there are certain disadvantages. To address this problem, the present invention adopts the BIFPN network to replace the feature fusion method of the neck part, aiming to better utilize the original feature information and enhance the ability to extract small-scale target features. In order to ensure the feature extraction of small-scale targets, BIFPN adopts a combination of downsampling paths and upsampling paths. Specifically, through the downsampling path, the network can obtain rich semantic information from high-resolution feature maps, which helps to better locate small-scale targets; and through the upsampling path, the network can fuse low-resolution feature maps into high-resolution feature maps, thereby improving the feature expression ability of small-scale targets.
[0066] The introduction of BIFPN plays an important role in enhancing the performance of FPN. By introducing learnable weights, BIFPN can effectively determine the importance of each input feature and better balance the information between different scales. The model optimizes the feature fusion process by introducing a learnable mechanism. In BIFPN, the weight learning mechanism learns the connection weights between each feature layer through training data, so that the network can adaptively adjust the importance of features to improve the performance of object detection.
[0067] Before inputting the video data of the driving scene to be tested into the improved YOLOv5s, the improved YOLOv5s is also trained, specifically: collecting video data of the target vehicle in different time periods and different road conditions, and constructing a data set. The present invention divides the data set into a training set, a validation set and a test set in a ratio of 8:1:1. The training set is used to train the parameters of the improved YOLOv5s; the validation set is used to tune the model hyperparameters to prevent the model from overfitting; and the test set is used to evaluate the accuracy of the model on unseen data.
[0068] In addition, the data in the dataset is filtered and annotated, and a bounding box is drawn for each target object to ensure the applicability and robustness of the model.
[0069] After training the improved YOLOv5s, the video data of the driving scene to be tested is input into the improved YOLOv5s, and the CBAM attention mechanism is used to increase the weight of the brake light information in the input image. The BIFPN structure is used to obtain the weights of different layers in the image, and the feature maps of different layers are fused to obtain the brake light status of the target vehicle.
[0070] Step 4: Define multiple ontology types related to vehicle behavior prediction in driving scenarios, determine the relationship between multiple ontology types, and build a prediction ontology model based on multiple ontology types and the relationship between multiple ontology types; formulate inference rules based on driving rules, and create corresponding instances in the prediction ontology model in combination with the brake light status, use the Drools inference engine for inference, and predict the future behavior dynamics of the target vehicle.
[0071] The predictive ontology model is a conceptual model used to describe domain knowledge. It provides a basis for knowledge representation and reasoning by clearly defining and classifying entities, attributes, and relationships in the domain. In the prediction of the preceding vehicle behavior based on light language information, the ontology model will be used to describe various entities, attributes, and relationships related to the vehicle's maneuvering intention, thereby achieving effective management and utilization of traffic information and vehicle information. In view of the limited vehicle information extracted by the visual detection model, the present invention focuses on a specific and challenging application scenario - a four-lane intersection, to construct a refined predictive ontology model.
[0072] The specific process of predicting the future behavior dynamics of the target vehicle is as follows:
[0073] (1) Define multiple ontology types related to vehicle behavior prediction in driving scenarios.
[0074] Classes represent different types of entities in driving scenarios. First, the present invention identifies and defines core categories that are closely related to the prediction of the preceding vehicle behavior, such as Figure 6As shown in the figure, the ontology of the preceding vehicle behavior prediction includes "target vehicle", "driving environment", "brake light status" and "maneuver intention". These categories represent the subject, external environment, immediate status and future intention of the preceding vehicle behavior respectively. Starting from these four top-level concepts, the lower-level structure is subdivided. Among them, "driving environment" is a composite category used to describe the driving conditions and environment of the target vehicle. It consists of two subcategories: driving lane and traffic light status. The driving lane has subcategories: the leftmost lane, the rightmost lane and the middle lane, which are used to describe the lane where the vehicle is currently located. The traffic light status has subcategories: red light on, green light on and yellow light on, which are used to describe the current traffic light status. The target vehicle is associated with its driving lane, the vehicle condition of the adjacent lane and the traffic light status through object attributes. "Maneuver intention" is used to represent the future driving intention and plan of the target vehicle. It consists of two subcategories: turning intention and speed change. Turning intention has subcategories: turn left, turn right and go straight, which are used to describe the vehicle's turning plan. Speed change has subcategories: deceleration, maintain speed or accelerate and decelerate to stop, which are used to describe the vehicle's speed change plan. The target vehicle is associated with its steering intention and speed change through object attributes.
[0075] (2) Define the related attributes between multiple ontology types.
[0076] Establish the association between categories. For example, "target vehicle" is associated with "driving environment", indicating the specific driving conditions of the vehicle; "target vehicle" is associated with "brake light status", reflecting the vehicle's immediate intention to slow down or stop; "target vehicle" is associated with "maneuvering intention", revealing the vehicle's future driving direction and speed changes. These associations together constitute a complete description of the preceding vehicle's behavior. The complete object attributes are shown in Table 1.
[0077] (3) Construction of target vehicle reasoning rules.
[0078] Inference rules are formulated based on common sense and general driving behavior. These rules will help the present invention to infer the future behavior of the preceding vehicle based on known information, as shown in Table 2.
[0079] Table 1 Object property settings
[0080] Object property name Domain range IsInLane TargetVehicle Driving_Lane IsDoing TargetVehicle Speed_Change Controlled by TargetVehicle Traffic_Light HasBrakeLightStatus TargetVehicle Brake_Light_Status LikelyTo TargetVehicle Turn_Intention
[0081] The state of the brake lights clearly indicates whether the vehicle is slowing down, while the state of the traffic lights directly determines whether the vehicle needs to stop and wait at the intersection or obtain permission to pass. In a typical three-lane intersection layout, the left lane is designed as a dedicated left-turn lane. Vehicles driving in this lane usually need to turn left while complying with the instructions of the traffic lights. The middle lane is used as a straight lane for straight-moving vehicles. As for the rightmost lane, it is flexible and allows vehicles to turn right, but can also go straight according to traffic conditions. It is particularly noteworthy that in most cases, vehicles in this lane are not directly restricted by traffic lights when turning right.
[0082] Table 2 Ontological rules of the future behavior of the front vehicle
[0083]
[0084] The present invention also provides a vehicle behavior prediction system, comprising:
[0085] A data acquisition module, used to obtain video data of the driving scene to be tested;
[0086] The brake light status recognition module is used to input the video data of the driving scene to be tested into the improved YOLOv5s, and use the CBAM attention mechanism to increase the brake light information weight of the input image; use the BIFPN structure to fuse the feature maps of different levels in the image, and introduce the brake light information weight at each fusion node to obtain the brake light status of the target vehicle; wherein, the CBAM attention mechanism is introduced before the SPPF module of the backbone network of the target detection model YOLOv5s, and the PANET structure of the neck of the target detection model YOLOv5s is replaced with the BIFPN structure to obtain the improved YOLOv5s;
[0087] A prediction ontology model building module is used to define multiple ontology types related to vehicle behavior prediction in driving scenarios, determine the relationship between multiple ontology types, and build a prediction ontology model based on multiple ontology types and the relationship between multiple ontology types;
[0088] The vehicle behavior prediction module is used to create corresponding instances in the prediction ontology model according to the brake light status and vehicle behavior reasoning rules, and use the Drools reasoning engine for reasoning to predict the future behavior dynamics of the target vehicle.
[0089] The present invention also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute a vehicle behavior prediction method.
[0090] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor to execute the vehicle behavior prediction method.
[0091] In order to fully verify the effectiveness of the proposed algorithm, the present invention conducted a series of module ablation experiments on the above data sets. These experiments are aimed at deeply exploring the impact of the CBAM attention mechanism and the weighted bidirectional feature pyramid network on the algorithm performance. The present invention designed a total of 4 groups of ablation experiments, and the experimental results are shown in Table 3. Through the data presented in Table 3, the key experimental results of the improved algorithm of the present invention can be clearly understood. The first group of experiments selected the Yolov5s algorithm as the original model, and the subsequent experiments were compared and analyzed with it. The mAP@0.5 of the original model was 88.5%, and mAP@0.5 is an important indicator for measuring the performance of the target detection model. When the value of map0.5 is high, it means that the model can detect the target well under the condition that the IoU threshold is 0.5, and the overlap between the predicted box and the real box is high. This data provides a benchmark and also provides a clearer direction for subsequent improvement and optimization. The second group of experiments introduced the CBAM attention mechanism based on the original network. This change helps to enhance the model's attention to key information, and can more comprehensively capture the key features in the image, and improve the mAP@0.5 index by 0.8%.
[0092] Table 3 Ablation experiment
[0093] Group CBAM BIFPN mAP 1 0.883 2 √ 0.891 3 √ 0.89 4 √ √ 0.908
[0094] The third group of experiments introduced the BiFPN network module on the basis of the original network. The addition of this module effectively solved the problem of multi-scale target detection in taillight detection, and the mAP@0.5 index increased by 0.7% compared with the original network. The fourth group of experiments introduced the CBAM attention mechanism and the BiFPN network module on the original network at the same time, and the mAP@0.5 index increased by 2.5% compared with the original network.
[0095] Figure 7 (a) Figure 7 (b) Figure 7 (c) Figure 7 (d) Figure 7 (e) and Figure 7 (f) shows the brake light detection results of 6 consecutive frames of video. In the first frame, the brake lights of the three vehicles are not on. In the second frame, the brake lights of the front vehicle are on one after another, and they are all on and have been stably detected since then. In addition, the brake light detection results of the adjacent lane on the left are also very stable. In the case of partial occlusion, the brake lights in all 7 frames are detected.
[0096] Near an intersection, braking behavior is often accompanied by a turn signal. Considering that the turn signal may interfere with the detection of the brake light, the present invention specifically intercepts several videos of the intersection for detection. Figure 8 The detection results of the turn signal condition are shown. Figure 8 (a), Figure 8 (d) is the case where only the brake lights are on. Figure 8 (b), Figure 8 (e) indicates the situation where the brake lights and turn signals are on at the same time. Figure 8 (c) and Figure 8 (f) is when only the turn signal is on. The above models all made correct detections, and the test results show that the turn signal has no effect on the detection model.
[0097] Creation of instances and rule writing of the ontology model for predicting the behavior of the preceding vehicle: When constructing the semantic relationship framework, the present invention breaks down the detected brake light status into specific instances, namely, "brake light on" and "brake light off", and these two instances directly belong to the "brake light status" category. In order to comprehensively and logically connect the four top-level concepts mentioned above - brake light status, target vehicle, driving environment and maneuvering intention, the present invention also creates relevant instances under these categories. Among them, the car (Awhitecar) instance belongs to a subclass of the "Target_Vehicle" class; the leftmost lane (LeftmostLane) instance belongs to the Leftmost_Lane subclass under the driving environment class; the red light on (Red_Light_On) instance belongs to the Traffic_Light subclass under the driving environment class; the left turn (LeftTurn) instance belongs to the Turn_Intention subclass under the maneuvering intention class; the slowdown (Slow_Down) instance, slowing down to a stop belongs to the Speed_Change subclass under the maneuvering intention class; the schematic diagram of instance creation is shown as follows Fig. 9 As shown in Table 4, according to the different characteristics of the different concept classes to which the instances belong, the object attributes and data attributes corresponding to the instances are added. Then the specific instance rules are converted into SWRL rules as shown in Table 4.
[0098] Table 4 SWRL rules
[0099]
[0100] SWRL rules are a rule language. When used in ontology, the relationships and vocabulary described in the ontology can be directly used to write rules. After creating the corresponding instance in the ontology model of the preceding vehicle behavior prediction, the Drools rule engine is used to reason about the preceding vehicle behavior in combination with the preceding vehicle behavior reasoning rules based on SWRL. First, the brake light status, driving lane, and traffic light conditions of the target vehicle are converted into individual instance knowledge, stored in the ontology model, and represented as Drools facts. Then, the multiple SWRL rules constructed in the previous article are converted into Drools rules, and finally, the Drools reasoning engine is run based on facts and rules for reasoning.
[0101] Depend on Fig.10 It can be seen from the reasoning results that after reasoning based on SWRL rules and Drools reasoning engine, the preceding vehicle behavior prediction model constructed by the present invention shows reliable performance and can accurately predict the future behavior dynamics of the target vehicle, such as Fig.10 As shown in (b), relative to Fig.10 (a) In the object attribute list of "Awhitecar" (i.e., the target vehicle), the key attributes "IsDoing SlowDown" and "IsDoing SlowDowntostop" are newly added. The activation of these two attributes directly maps the clear deceleration action that the vehicle is currently performing. When the vehicle's brake lights are on, this visual signal is undoubtedly an intuitive confirmation that the vehicle is in the process of decelerating. Next, considering that the red light is on at the intersection ahead, this sign further strengthens the inference that the target vehicle is gradually slowing down until it may eventually stop and wait. In addition, the annotation of the "LikelytoLeftTurn" attribute indicates that the target vehicle has a high probability of performing a left turn after completing the deceleration to a stop or near stop state. For example Fig.10 In the subsequent scenario shown in (c), the actual behavior of the target vehicle perfectly verifies the accuracy of the prediction made by the ontology model, demonstrating the model's powerful ability to predict vehicle behavior in complex traffic environments.
[0102] The embodiments described above are only preferred specific implementation modes of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belong to the protection scope of the present invention.
Claims
1. A vehicle behavior prediction method, characterized in that: The steps include: Obtain video data of the driving scene to be tested; The video data of the driving scene to be tested is input into the improved YOLOv5s, and the CBAM attention mechanism is used to increase the weight of the brake light information of the input image; The BIFPN structure is used to fuse feature maps of different levels in the image, and the brake light information weight is introduced at each fusion node to obtain the brake light status of the target vehicle; the CBAM attention mechanism is introduced before the SPPF module of the backbone network of the target detection model YOLOv5s, and the PANET structure of the target detection model YOLOv5s is replaced with the BIFPN structure to obtain the improved YOLOv5s; Define multiple ontology types related to vehicle behavior prediction in driving scenarios, determine the relationship between multiple ontology types, and build a prediction ontology model based on multiple ontology types and the relationship between multiple ontology types; According to the brake light status and vehicle behavior reasoning rules, corresponding instances are created in the prediction ontology model, and the Drools reasoning engine is used for reasoning to predict the future behavior dynamics of the target vehicle.
2. The vehicle behavior prediction method according to claim 1, characterized in that: The entity type includes the target vehicle, the driving environment, the brake light status and the maneuvering intention; the driving environment includes the driving lane and the traffic light status; the maneuvering intention includes the turning intention and the speed change.
3. The vehicle behavior prediction method according to claim 2, characterized in that: The relationship between the multiple entity types is specifically as follows: the target vehicle is associated with the driving environment, indicating the specific driving conditions of the vehicle; the target vehicle is associated with the brake light status, indicating the vehicle's immediate deceleration intention or parking intention; the target vehicle is associated with the maneuvering intention, indicating the vehicle's future driving direction and speed changes.
4. The vehicle behavior prediction method according to claim 2, characterized in that: The target vehicle inference rules are as follows: Rule 1: If the target vehicle's brake lights are on, then the vehicle is slowing down; Rule 2: If the target vehicle's brake lights are not on, the vehicle is accelerating or traveling at the original speed; Rule 3: If the target vehicle is driving in the leftmost lane, the vehicle will turn left; Rule 4: If the target vehicle is in the rightmost lane, the vehicle will turn right; Rule 5: If the target vehicle is in the middle lane, the vehicle will go straight; Rule 6: If the target vehicle is decelerating and the red light is on, the vehicle will decelerate to a stop; Rule 7: If the target vehicle is slowing down and the green light is on, then the vehicle will slow down; Rule 8: If the target vehicle is slowing down and the yellow light is on, the vehicle will slow down to a stop.
5. The vehicle behavior prediction method according to claim 1, characterized in that: Before inputting the video data of the driving scene to be tested into the improved YOLOv5s, the method also includes training the improved YOLOv5s, specifically: collecting video data of the target vehicle in different time periods and different road conditions, and constructing a data set, dividing the data set into a training set, a validation set and a test set in a ratio of 8:1:1, using the training set to train the parameters of the improved YOLOv5s; using the validation set to tune the model hyperparameters; and using the test set to evaluate the accuracy of the improved YOLOv5s.
6. The vehicle behavior prediction method according to claim 5, characterized in that: Before training the improved YOLOv5s, the method further includes filtering the data in the data set, annotating the filtered data, and drawing a bounding box for each target object.
7. A vehicle behavior prediction system, characterized in that: include: A data acquisition module, used to obtain video data of the driving scene to be tested; The brake light state recognition module is used to input the video data of the driving scene to be tested into the improved YOLOv5s, and use the CBAM attention mechanism to increase the weight of the brake light information of the input image; The BIFPN structure is used to fuse feature maps of different levels in the image, and the brake light information weight is introduced at each fusion node to obtain the brake light status of the target vehicle; the CBAM attention mechanism is introduced before the SPPF module of the backbone network of the target detection model YOLOv5s, and the PANET structure of the target detection model YOLOv5s is replaced with the BIFPN structure to obtain the improved YOLOv5s; A prediction ontology model building module is used to define multiple ontology types related to vehicle behavior prediction in driving scenarios, determine the relationship between multiple ontology types, and build a prediction ontology model based on multiple ontology types and the relationship between multiple ontology types; The vehicle behavior prediction module is used to create corresponding instances in the prediction ontology model according to the brake light status and vehicle behavior reasoning rules, and use the Drools reasoning engine for reasoning to predict the future behavior dynamics of the target vehicle.
8. A computer device, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the vehicle behavior prediction method according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the vehicle behavior prediction method according to any one of claims 1-6.
Citation Information
Patent Citations
Front vehicle driving intention analysis method based on tail lamp recognition
CN108357418A
Vehicle behavior prediction method and device, electronic equipment and storage medium
CN111380555A
Road traffic sign abnormal state detection method based on YOLOv5
CN115331202A
Signal lamp-based intention prediction method and device, and storage medium
CN116311159A
Unmanned driving system and method based on image processing
CN117141519A
Cited By
Traffic signal lamp rapid identification method and system of lightweight classifier coupling detection head
CN121259784A
Lightweight classifier coupled traffic signal light fast recognition method and system
CN121259784B
