A fast detection method for lightweight multimodal vehicle behavior

Abstract

The present invention discloses a fast detection method for lightweight multimodal vehicle behavior, which is applied to the fields of vehicle safety driving monitoring and vehicle assisted driving technology. The present invention obtains vehicle driving data and its associated information with the scene, quickly extracts and fuses multimodal information and identifies vehicle driving characteristics, establishes understanding rules through driving behavior semantic decomposition, converts continuous vehicle driving trajectory data into simple vehicle operation semantic information, and then determines a series of simple vehicle operation combinations to form vehicle driving behaviors and define multiple driving styles, and finally identifies the on-the-way driving behaviors of each vehicle-vehicle combination in the overall scene within a specific time and space range as different driving styles. The present invention significantly improves the accuracy and comprehensiveness of driving behavior recognition, realizes abnormal state and level evaluation through driving ability evaluation based on the relationship between people, vehicles and roads, and helps to realize a wider range of analysis and evaluation associated with traffic scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle safe driving monitoring, and more specifically to a method for rapid detection of lightweight multi-modal vehicle behavior. Background Art

[0002] In recent years, with the rapid development of electronic information and communication technology, driving assistance technology has been continuously improving. However, the intelligentization process of roads and vehicles in different regions is uneven, which has brought significant differences in driving management and vehicle operation, and also posed new challenges to the analysis of human-machine co-driving characteristics. Human-machine co-driving refers to the collaborative operation between human drivers and intelligent systems in the vehicle's partially automated driving mode to complete driving tasks in a specific area. Accurately analyzing and understanding the driver's on-the-road driving style can improve the intelligent system's understanding of the driver's status and operation, thereby enhancing the real-time matching between the driver and the intelligent system, which can not only improve driving efficiency, but also help to effectively prevent potential safety risks.

[0003] The driving process of a vehicle requires the driver to continuously pay attention to the surrounding vehicles and road conditions, but the driver may be distracted by factors such as the broadcasting system, road signs, passenger communication, and vehicle control characteristics. The deployment of intelligent vehicle computers and vehicle intelligent control systems has further increased the dynamic assistance and interference of driving operations, changing the driver's attention allocation and the threat factors of road traffic accidents. Driver attention allocation errors may lead to improper driving behaviors, such as unnecessary sudden acceleration, sudden braking, or sharp turns. These errors may cause injuries, accidents, and even life-threatening situations, as well as economic losses. Faced with the diversity of transportation needs, including commuting, leisure, transportation, and emergency tasks, the risk of accidents always exists. Among all road traffic accidents, the vast majority are caused by abnormalities in the driver, vehicle, or road conditions. In addition, the driver's physical health and mental state, as well as the vehicle's operating performance, also have an impact on driving behavior. Traditional monitoring systems rely on vehicle-mounted ad hoc networks (VANETs) and similar intelligent technologies to evaluate driver capabilities and vehicle operability through complex vehicle motion and operation rules. However, these methods rely on strict operation association design and preset rules, and may not accurately identify driving behaviors when encountering insufficient simulation data or occasional events. Therefore, based on on-board equipment and roadside equipment, a comprehensive model with stronger analytical capabilities is needed to utilize multimodal data and reduce reliance on the design of clear event association rules, thereby improving the ability to analyze the association of different types of driving behaviors and road conditions.

[0004] At present, there are two main methods for obtaining driving style data: one is based on subjective surveys, which obtains driving style data through pre-designed questionnaires and extracts the required driving characteristic parameters for analysis; the other is based on objective detection, which obtains driving data through driving simulators or actual vehicle roadside detectors, and performs subsequent processing for driving behavior analysis. Subjective surveys have different levels of understanding and willingness to express the questionnaire, which may lead to biased results and make it impossible to continuously obtain detailed driving data; on the other hand, although objective detection methods can quantify driving behavior more accurately, they are still limited by preset test conditions and equipment. In view of the complex scenarios of modern traffic and the diverse needs of drivers, traditional methods are difficult to meet the task requirements of driving behavior analysis. Therefore, how to provide a fast detection method for lightweight multimodal vehicle behavior is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a lightweight multimodal vehicle behavior rapid detection method, which adopts a real-time communication method to collect information on the same target from multiple angles, and establishes a multimodal vehicle behavior detection network by analyzing the real-time vehicle-road detection data information collected from a multi-source method, thereby realizing continuous analysis of the driving behavior of each vehicle.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A fast detection method for lightweight multimodal vehicle behavior comprises the following steps:

[0008] S1. The driver drives the vehicle according to his / her own preferences in the experimental scenario to complete the driving task, collects the vehicle-road cooperative data in real time and continuously during the driving task, and obtains the vehicle state quantity information and roadside state quantity information;

[0009] S2, preprocessing the data collected in S1, including normalization of one-dimensional signal data and windowing of two-dimensional image data;

[0010] S3. Using the original ResNet-152 as the benchmark model, a multi-layer neural network is established through transfer learning. Each single-layer neural network in the multi-layer neural network corresponds to the detection of different driving tasks. A driving behavior classification network is established with a fully connected layer and a bidirectional long short-term memory classification network. The multi-layer neural network and the driving behavior classification network are combined to obtain a benchmark multimodal vehicle behavior detection network.

[0011] S4, inputting the preprocessed data into a benchmark multimodal vehicle behavior detection network, comparing the output of each single-layer neural network in the benchmark multimodal vehicle behavior detection network with the output of the original neural network corresponding to the driving task, calculating their three-dimensional distance, and obtaining a transfer learning performance evaluation result;

[0012] S5. Optimize the single-layer neural network in the benchmark multimodal vehicle behavior detection network according to the improved rules, calculate multiple optimization results of the single-layer neural network, compare the multiple optimization results with the transfer learning performance evaluation results by cross entropy, select the best optimization result as the tuning result of the corresponding single-layer neural network, tune all single-layer neural networks, and obtain the final multimodal vehicle behavior detection network;

[0013] S6. Detect driving behavior based on the final multimodal vehicle behavior detection network.

[0014] Optionally, the vehicle-mounted state quantity information includes driver status, driver operation, pedal stroke, pedal stroke change rate, steering wheel turning angle and force, vehicle position, vehicle speed, vehicle acceleration and vehicle driving direction change; the roadside state quantity information includes vehicle speed, driving speed change, driving direction change, vehicle position, vehicle posture, and the relationship between the vehicle and the driving environment.

[0015] Optionally, perform parallel normalization on the multivariate input signal as a whole and build a signal matrix for the original input as a whole Std ( σ ), the processing of a single one-dimensional signal is specifically represented by the normalization of the data:

[0016] ;

[0017] In the formula, a s is the standardized value, S is the step time, and the corresponding actual input value of the current signal is T s , the overall mean of multi-dimensional input is Mean ( μ ), Std.deviation ( σ ) is the component of the signal matrix in the current single one-dimensional signal;

[0018] The windowing of two-dimensional image data is as follows:

[0019] ;

[0020] By the current image X Divide into windows with horizontal and vertical dimensions ranging from 1 to x t y and n y , with the help of t y and n y For images X Dimensions dxy The pane processing is performed to finally obtain the pane segmented image d xy .

[0021] Optionally, a fully connected network layer is used to build an attention module to improve the traditional bidirectional long short-term memory classification network.

[0022] Optionally, S4 is specifically:

[0023] S41. Obtain the output evaluation weight matrix of the single-layer neural network in the benchmark multimodal vehicle behavior detection network W q ;

[0024] S42, obtain the output of the original neural network corresponding to the single-layer neural network W k ;

[0025] S43, calculation W q and W k The three-dimensional spacing between:

[0026] ;

[0027] In the formula, D mixed ( W q , W k )for W q and W k The three-dimensional distance between m Indicates the width dimension range of the matrix, j ∈[1, m ], n Represents the length dimension range of the matrix, k ∈[1, n ], i represents the depth of the matrix, l Before the corresponding i -1 depth is the result of scalar signal processing, i The depth represents the image input and the corresponding output matrix, for l The layer matrix ( j, k ) corresponds to the element of the input quantity, for l The layer matrix ( j, k ) position corresponds to the element of the output quantity, For the i The layer matrix ( j，k ) corresponds to the element of the input quantity, For the i The layer matrix ( j, k ) position corresponds to the element of the output, and the scalar input and image input are calculated for the multi-dimensional spacing. i -1 scalar signal The spacing of each input is calculated by corresponding and The spacing is calculated and the root mean square is obtained. For the image input, the target information difference in the pane is calculated separately according to the pane result.

[0028] Optionally, S5 is specifically:

[0029] S51, optimizing the single-layer neural network in the benchmark multimodal vehicle behavior detection network based on four improvement rules: parameter dimension scaling, parameter horizontal adjustment, parameter vertical adjustment, and internal parameter adjustment;

[0030] S52, calculate the output of all optimized single-layer neural networks and W k The three-dimensional spacing between

[0031] S53, performing cross entropy comparison on all three-dimensional distances and transfer learning performance evaluation results;

[0032] S54, selecting the best optimization result among the three most matching optimization results, replacing the corresponding single-layer neural network in the benchmark multimodal vehicle behavior detection network, and realizing single-layer neural network tuning;

[0033] S55. Tune all single-layer neural networks to obtain the final multimodal vehicle behavior detection network.

[0034] Optionally, the four improvement rules all include two change directions, namely, augmentation and contraction. Parameter dimension scaling, parameter horizontal adjustment, and parameter vertical adjustment have three levels of change in both augmentation and contraction directions, and internal parameter adjustment only performs contraction verification.

[0035] Optionally, the preprocessed data is divided into a labeled data set and an unlabeled data set, the unlabeled data set is used for single-layer neural network tuning, and the labeled data set is used for verifying the tuned single-layer neural network.

[0036] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a fast detection method for lightweight multi-modal vehicle behavior, which has the following beneficial effects:

[0037] 1. Aiming at multimodal information driving scenarios, in the absence of high-precision maps and a large amount of systematic data, the present invention is task-oriented, uses typical dynamic factors as classification conditions, and reversely fuses the associated interactions of two-dimensional continuous execution tasks of adjacent layers and the classification features of single operation tasks to generate a series of rich simulation rule classification requirements and data requirement features for multiple layers, thereby realizing the detailed classification of the entire driving behavior of human-operated vehicles passing through the area. It is a new multimodal driving behavior classification method;

[0038] 2. Different from the simple superposition of traditional multi-level sensors and multi-class neural networks, the present invention uses sensor data of multiple related modes, improves the accuracy of multiple sensors through cross-validation, uses related estimation to identify potential anomalies and data missing, obtains the real detection information under the current conditions through parallel understanding of multiple inputs, and corrects individual deviation information. With the help of prediction standard information and real detection results, the distribution of detection results that is closer to the real situation is obtained, thereby improving the accuracy and stability of target detection and state detection in real scenarios;

[0039] 3. The present invention is suitable for training of small sample models. When the use of large sample data is limited due to high data acquisition or annotation costs, the present invention generates simulated data through forward propagation, and generates more generalized supplementary data by combining the noise distribution characteristics of the real acquired data. At the same time, cross-restrictions are introduced to multi-level associated sensors to obtain input signal supplements that conform to the actual situation. Combined with the distribution characteristics of the real detection results of multiple detections, the full utilization and expansion of all relevant data is achieved, and the learnability of the data is improved;

[0040] 4. Based on small sample vehicle-side data, roadside annotated data and a large amount of unlabeled detection data, the present invention uses the task requirements contained in the small sample to conduct in-depth detection feature fusion analysis of driver operations, vehicle feedback and changes in road driving conditions, and proposes a new multimodal data classification network establishment and dynamic parameter adjustment method. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 It is a flow chart of the rapid detection method of lightweight multi-modal vehicle behavior of the present invention;

[0043] Figure 2A schematic diagram of a fully connected layer and a bidirectional long short-term memory classification network of the present invention;

[0044] Figure 3 A schematic diagram of tuning a single-layer neural network according to the present invention;

[0045] Figure 4 It is a schematic diagram of the neural network layer optimization of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The embodiment of the present invention discloses a fast detection method for lightweight multi-modal vehicle behavior, such as Figure 1 As shown, the following steps are included:

[0048] S1. The driver drives the vehicle according to his / her own preferences in the experimental scenario to complete the driving task, collects the vehicle-road cooperative data in real time and continuously during the driving task, and obtains the vehicle state quantity information and roadside state quantity information;

[0049] S2, preprocessing the data collected in S1, including normalization of one-dimensional signal data and windowing of two-dimensional image data;

[0050] S3. Using the original ResNet-152 as the benchmark model, a multi-layer neural network is established through transfer learning. Each single-layer neural network in the multi-layer neural network corresponds to the detection of different driving tasks. A driving behavior classification network is established with a fully connected layer and a bidirectional long short-term memory classification network. The multi-layer neural network and the driving behavior classification network are combined to obtain a benchmark multimodal vehicle behavior detection network.

[0051] S4, inputting the preprocessed data into a benchmark multimodal vehicle behavior detection network, comparing the output of each single-layer neural network in the benchmark multimodal vehicle behavior detection network with the output of the original neural network corresponding to the driving task, calculating their three-dimensional distance, and obtaining a transfer learning performance evaluation result;

[0052] S5. Optimize the single-layer neural network in the benchmark multimodal vehicle behavior detection network according to the improved rules, calculate multiple optimization results of the single-layer neural network, compare the multiple optimization results with the transfer learning performance evaluation results by cross entropy, select the best optimization result as the tuning result of the corresponding single-layer neural network, tune all single-layer neural networks, and obtain the final multimodal vehicle behavior detection network;

[0053] S6. Detect driving behavior based on the final multimodal vehicle behavior detection network.

[0054] Furthermore, the vehicle-mounted state quantity information includes driver status, driver operation, pedal stroke, pedal stroke change rate, steering wheel rotation angle and force, vehicle position, vehicle speed, vehicle acceleration and vehicle driving direction change; the roadside state quantity information includes vehicle speed, driving speed change, driving direction change, vehicle position, vehicle posture, and the relationship between the vehicle and the driving environment.

[0055] Furthermore, in an embodiment of the present invention, real-time detection data of the experimental scene (i.e., the driver driving in real conditions) is continuously collected, and the data frequency can be 1Hz to 300Hz, and the total time sequence of the collected samples is not less than 1000, so as to collect multiple complete vehicle-side driving operations and vehicle driving information, and the driver is required to drive the vehicle according to his own preferences, including speed control, steering control, driving route control and management control rule response speed. The selected drivers are familiar with the driving of all typical vehicles, so they can drive all the vehicles involved to complete the driving tasks.

[0056] Furthermore, the multi-input signals are normalized in parallel as a whole, and a signal matrix is ​​established for the original input as a whole. Std ( σ ), the processing of a single one-dimensional signal is specifically represented by the normalization of the data:

[0057] ;

[0058] In the formula, a s is the standardized value, S is the step time, and the corresponding actual input value of the current signal is T s , the overall mean of multi-dimensional input is Mean ( μ ), Std.deviation ( σ ) is the component of the signal matrix in the current single one-dimensional signal;

[0059] The windowing of two-dimensional image data is as follows:

[0060] ;

[0061] By the current image X Divide into windows with horizontal and vertical dimensions ranging from 1 to x t y and n y , with the help of t y andn y For images X Dimensions d xy The pane processing is performed to finally obtain the pane segmented image d xy .

[0062] Normalizing data is currently considered a preferred practice for building advanced models. Normalization techniques are used to standardize input data and obtain feature vectors containing a standard deviation of 1 and a mean of 0. Important goals of the standardization stage include converting the dataset to a common scale; using sliding window technology to spatially regularize multi-source real-time datasets at a single moment, and dividing them into driving behavior events containing multiple discrete location units based on location relationships.

[0063] Further, such as Figure 2 As shown in the figure, a fully connected network layer is used to build an attention module to improve the traditional bidirectional long short-term memory classification network. Compared with the traditional bidirectional search method, the improved network information retrieval with the addition of the attention mechanism is more targeted, and the variable information search method improves the operating efficiency of the system.

[0064] Furthermore, in an embodiment of the present invention, target tracking is combined with the edge record of the secondary grid and the dynamic motion model, and the target is obtained by means of the small-range change of the target edge in a single grid and the overall edge change, which can stably adapt to the target tracking needs of traffic scenes.

[0065] Furthermore, in the embodiment of the present invention, the extraction results of driving characteristics of visual information and scalar signals are used for driving characteristics analysis at three associated levels: driver operation level, vehicle execution level, and vehicle-road relationship level, comprehensively considering the three levels of driving behavior design, and establishing driving behavior scenario semantic understanding implementation rules for multimodal data;

[0066] The definition of driver operation status is shown in Table 1:

[0067] Table 1

[0068]

[0069] For specific driving purposes, corresponding basic vehicle operations are adopted, and the driver realizes the driving control operation output of the vehicle by operating the steering wheel and the accelerator and brake pedals.

[0070] Level I: Driving in a straight line at a constant speed

[0071] When driving in a straight line at a constant speed, the driver needs to keep the accelerator pedal stable to ensure that the vehicle moves forward at a constant speed. Specifically, the rate of change of the accelerator pedal position should be kept within 2%, while the forward direction should remain stable, and the steering wheel rotation angle should not change by more than 5 degrees. In this state, the longitudinal acceleration of the vehicle is close to zero, and the driving trajectory is a straight line.

[0072] Level II: Accelerated

[0073] In the acceleration state, the driver gradually increases the position of the accelerator pedal, and the change rate should be controlled within 10% / second, and the upshift operation should be coordinated to ensure smooth engine acceleration. The longitudinal acceleration of the vehicle should be kept within 2 m / s² to ensure ride comfort and driving stability.

[0074] Level III: Slowdown

[0075] When decelerating, the driver needs to gradually reduce the accelerator or lightly step on the brake. The pressure change rate of the brake pedal should be controlled within 10% / second, and the downshift operation should be assisted to ensure smooth deceleration of the vehicle. The deceleration of the vehicle should be kept within 2 m / s² to avoid obvious forward tilt or discomfort.

[0076] Level IV: Parking

[0077] In the parking state, the driver operates the brake pedal to stop the vehicle smoothly. The pressure of the brake pedal should reach more than 50%, the deceleration of the vehicle should be controlled within 2.5 m / s², and the parking process should be completed within 3 seconds. In this state, the vehicle should stop smoothly at the predetermined position to avoid the impact caused by sudden stop.

[0078] Level V: Turn

[0079] When turning, the driver needs to reduce the speed to below 20 km / h according to the road conditions and turn the steering wheel smoothly. The rate of change of the turning angle should be controlled within 30 degrees / second to ensure the basic stability of the vehicle posture and the passengers in the car. The lateral acceleration should be kept within 0.5 g to avoid roll and instability.

[0080] Level VI: Dangerous Driving Maneuvers

[0081] In dangerous driving conditions, the driver takes aggressive actions, such as operating the accelerator or brake pedal significantly, the accelerator pedal position changing rate exceeding 20% / second, the brake pedal pressure changing rate exceeding 20% / second, and the steering wheel angle changing rate exceeding 45 degrees / second. These actions may cause the vehicle to lose control or cause an accident, increasing driving risks.

[0082] The vehicle execution layer state definition is shown in Table 2:

[0083] Table 2

[0084]

[0085] The vehicle execution layer is the specific response characteristics of the vehicle to driving operations such as stable driving, acceleration, deceleration, parking, steering and danger. It is the task of the vehicle execution layer to evaluate whether the vehicle can complete the driver's operations in a timely and accurate manner.

[0086] Level I: Stable driving

[0087] In steady driving conditions, the vehicle's motion should remain linear and smooth. The lateral acceleration of the vehicle measured by the roadside equipment and on-board motion sensors should be less than 0.2 g, and the longitudinal acceleration should be close to zero, ensuring that the vehicle drives smoothly on straight sections of road.

[0088] Level II: Accelerated

[0089] When accelerating, the vehicle should respond smoothly to the driver's acceleration operation. The acceleration should be kept within 2 m / s² to ensure that the vehicle does not experience abrupt speed changes during acceleration. At the same time, the vehicle should stay in the fixed lane, and the lane deviation should be less than 10 cm to ensure driving safety.

[0090] Level III: Slowdown

[0091] In the deceleration state, the vehicle should respond smoothly to the driver's deceleration operation. The deceleration rate should be kept within 2 m / s² to ensure that the vehicle remains stable during the deceleration process. The vehicle should adapt to the traffic conditions ahead and maintain a safe distance from the vehicle in front. The distance to the vehicle in front should be kept at least 2 body lengths.

[0092] Level IV: Parking

[0093] When parking, the vehicle should respond to the driver's parking command in a timely manner and stop smoothly. The deceleration should be controlled within 2.5 m / s² to avoid discomfort or danger caused by emergency braking. After parking, the vehicle should remain stopped for more than 5 seconds to ensure a complete stop.

[0094] Level V: Turn

[0095] When turning, the vehicle should respond smoothly to the driver's steering command. The steering wheel angle change rate should be controlled within 30 degrees per second to ensure a smooth steering process. The vehicle should stay in the lane and the lane deviation should be less than 10 cm to ensure driving safety.

[0096] Level VI: Dangerous driving behavior

[0097] In dangerous driving behavior, the vehicle performs dangerous actions in response to the driver's operation. The vehicle exceeds the normal road speed limit (the speed on expressways should be kept within 120 km / h), changes lanes suddenly (changing lanes too quickly or too frequently), stops suddenly, or increases the speed significantly.

[0098] The definition of the vehicle-road relationship change layer state is shown in Table 3:

[0099] Table 3

[0100]

[0101] The vehicle-road relationship change layer requires a comprehensive analysis of the target vehicle's surrounding vehicles, weather, and road information to evaluate the impact on driving operations. The perception data can be obtained by relying on the vehicle-road cooperative system. The environment directly related to the vehicle includes vehicles near the target vehicle, roadside facilities, road conditions, lighting conditions, and weather conditions. For each driving condition, the following are its specific detailed features:

[0102] Level I: Stable driving

[0103] In a stable driving state, the vehicle is able to drive continuously through the current area without interruption. This means that the road conditions are good, the traffic flow is moderate, there is no need to change lanes or stop frequently, the vehicle is driving at a constant speed or slowly changing speed, and the operation of the accelerator pedal and steering wheel remain stable. The vehicle's driving speed should be within the road speed limit and the fluctuation is less than 10%, and the vehicle's longitudinal and lateral acceleration should be kept within 0.1 g.

[0104] Level II: Driving with signal lights

[0105] When driving under traffic light control, the driver needs to follow the control of the traffic light to operate the vehicle. This includes stopping at a red light, starting at a green light, and preparing to stop at a yellow light. The driver must slow down and stop at the traffic light, and the deceleration should be kept within 2 m / s², and start smoothly when the traffic light turns green, and the acceleration should be kept within 2 m / s² to ensure safety and comfort.

[0106] Level III: Reasonable weaving of conflicting vehicles

[0107] When conflicting vehicles are reasonably interweaving, drivers need to pay extra attention to other relevant conflicting vehicles on the road based on the signal lights and management control rules. This includes dealing with complex traffic conditions at intersections, such as oncoming vehicles and vehicles merging into intersections. Drivers need to make reasonable judgments and avoid other vehicles to ensure safe passage through the conflict area. During the interweaving process, the lateral acceleration of the vehicle should be controlled within 0.3 g to ensure smooth avoidance and lane change.

[0108] Level IV: Cumulative traffic congestion

[0109] In the state of cumulative traffic congestion, the road is congested due to the accumulation of vehicles from multiple directions between time intervals. The driver needs to start and stop frequently, and the speed is usually below 10 km / h. The frequent starts and stops of the vehicle should be kept smooth, the acceleration of the starting vehicle should be controlled within 1.5 m / s², and the deceleration when stopping should be controlled within 1.5 m / s² to reduce the discomfort of the driver and passengers and give the following vehicles enough reaction time. The driver needs to concentrate highly and avoid frequent lane changes and sudden braking to reduce further traffic congestion.

[0110] Level V: Vehicles cannot pass

[0111] When vehicles cannot pass, some vehicles cannot continue to drive due to traffic accidents or unexpected conditions, resulting in reduced road driving capacity and stagnation of related vehicles. In this case, the driving of vehicles in front, behind, left and right is stagnant, and it is impossible to rely on traffic to organize congestion evacuation in a short period of time. Some vehicles can only take illegal actions to pass through the congested area, and forced lane changes or reverse driving may cause more serious traffic chaos and safety accidents.

[0112] The driving behavior recognition results are shown in Table 4:

[0113] Table 4

[0114]

[0115] Furthermore, S4 is specifically:

[0116] S41. Obtain the output evaluation weight matrix of the single-layer neural network in the benchmark multimodal vehicle behavior detection network W q ;

[0117] S42, obtain the output of the original neural network corresponding to the single-layer neural network W k ;

[0118] S43, calculation W q and W k The three-dimensional spacing between:

[0119] ;

[0120] In the formula, D mixed ( W q , W k )for W q and Wk The three-dimensional distance between m Indicates the width dimension range of the matrix, j ∈[1, m ], n Represents the length dimension range of the matrix, k ∈[1, n ], i represents the depth of the matrix, l Before the corresponding i -1 depth is the result of scalar signal processing, i The depth represents the image input and the corresponding output matrix, for l The layer matrix ( j, k ) corresponds to the element of the input quantity, for l The layer matrix ( j, k ) position corresponds to the element of the output quantity, For the i The layer matrix ( j， k ) corresponds to the element of the input quantity, For the i The layer matrix ( j, k ) position corresponds to the element of the output, and the scalar input and image input are calculated for the multi-dimensional spacing. i -1 scalar signal The spacing of each input is calculated by corresponding and The spacing is calculated and the root mean square is obtained. For image input, the target information difference within the pane is calculated separately according to the pane-based result.

[0121] Furthermore, in the embodiment of the present invention, the single time series of the initial single-layer neural network input t The detection information is X t =( X 10 , X 20 ,..., X m0 )Total m Layer input, for the middle l The input of the layer is Layer Neural Network t Multi-modal parallel input corresponding to timing , after the After the feature extraction of the layer, the evaluation weight matrix is ​​obtained by using the output of the single-layer neural network W q , used to generate the performance evaluation of the derived neural network; for the original neural network layer output obtained by single-task neural network transfer learning Wk ,use W q and W k The three-dimensional distance evaluates the feature extraction performance of transfer learning from input to output, and a distance is obtained between each task and the driving behavior classification network. i The weight corresponding to the actual input data dimension is used as the weight for cross entropy mixing calculation.

[0122] Evaluation weight matrix W q is a m × n × i For each single task input, the parameter feature weight is m × n ,forward i -1 input is image input, and the last layer is multi-class scalar signal input. Since it is the same neural network, W q and W k The structure is similar, but the parameter weight dimensions are different. m k × n k × i The Euclidean distance is used to measure the similarity of the graph output, and the cosine similarity is used to measure the similarity of the network output results. i In the layer, two different networks are j The result after row scalar output vectorization and The cosine of the angle between .

[0123] Furthermore, S5 is specifically:

[0124] S51, optimizing the single-layer neural network in the benchmark multimodal vehicle behavior detection network based on four improvement rules: parameter dimension scaling, parameter horizontal adjustment, parameter vertical adjustment, and internal parameter adjustment;

[0125] S52, calculate the output of all optimized single-layer neural networks and W k The three-dimensional spacing between

[0126] S53, performing cross entropy comparison on all three-dimensional distances and transfer learning performance evaluation results;

[0127] S54, selecting the best optimization result among the three most matching optimization results, replacing the corresponding single-layer neural network in the benchmark multimodal vehicle behavior detection network, and realizing single-layer neural network tuning;

[0128] S55. Tune all single-layer neural networks to obtain the final multimodal vehicle behavior detection network.

[0129] In the embodiment of the present invention, the tuning process is as follows: Figure 3 and Figure 4 shown.

[0130] Furthermore, the four improvement rules all include two change directions: augmentation and contraction. Among them, parameter dimension scaling, parameter horizontal adjustment, and parameter vertical adjustment have three levels of change in both augmentation and contraction directions, and internal parameter adjustment only performs contraction verification.

[0131] Furthermore, in the embodiment of the present invention, S52 is specifically:

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139] ;

[0140] In the formula, is the output of the optimized single-layer neural network and W k The three-dimensional distance between W A is the best result among the optimization results. In order to avoid falling into the local optimum, the similarity of a single operation is not used as an absolute indicator of the degree of matching between the two networks. W A The best results of the three improved network layers are selected as the optimization results, and the corresponding network layers in the benchmark neural network are replaced to achieve single-layer network optimization.

[0141] Furthermore, the preprocessed data is divided into a labeled data set and an unlabeled data set. The unlabeled data set is used for single-layer neural network tuning, and the labeled data set is used to verify the tuned single-layer neural network.

[0142] Furthermore, in an embodiment of the present invention, the verification parameters include the timeliness of the system response, and the sensitivity of the system is evaluated by measuring the time interval between the response and the actual occurrence of the event. On the basis of timely response, the precision and recall of the system for the event are also evaluated. The precision is used to measure the proportion of the events detected by the system that are actually correct, while the recall is used to measure the proportion of all the events that actually occurred that the system can correctly detect. These indicators jointly detect the performance and reliability of the system in practical applications.

[0143] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0144] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fast detection method for lightweight multimodal vehicle behavior, characterized in that: The following steps are involved: S1. The driver drives the vehicle according to his / her own preferences in the experimental scenario to complete the driving task, collects the vehicle-road cooperative data in real time and continuously during the driving task, and obtains the vehicle state quantity information and roadside state quantity information; S2, preprocessing the data collected in S1, including normalization of one-dimensional signal data and windowing of two-dimensional image data; S3. Using the original ResNet-152 as the benchmark model, a multi-layer neural network is established through transfer learning. Each single-layer neural network in the multi-layer neural network corresponds to the detection of different driving tasks. A driving behavior classification network is established with a fully connected layer and a bidirectional long short-term memory classification network. The multi-layer neural network and the driving behavior classification network are combined to obtain a benchmark multimodal vehicle behavior detection network. S4, inputting the preprocessed data into a benchmark multimodal vehicle behavior detection network, comparing the output of each single-layer neural network in the benchmark multimodal vehicle behavior detection network with the output of the original neural network corresponding to the driving task, calculating their three-dimensional distance, and obtaining a transfer learning performance evaluation result; S5. Optimize the single-layer neural network in the benchmark multimodal vehicle behavior detection network according to the improved rules, calculate multiple optimization results of the single-layer neural network, compare the multiple optimization results with the transfer learning performance evaluation results by cross entropy, select the best optimization result as the tuning result of the corresponding single-layer neural network, tune all single-layer neural networks, and obtain the final multimodal vehicle behavior detection network; S6. Detect driving behavior based on the final multimodal vehicle behavior detection network.

2. A fast detection method for lightweight multimodal vehicle behavior according to claim 1, characterized in that: The on-board status information includes driver status, driver operation, pedal stroke, pedal stroke change rate, steering wheel rotation angle and force, vehicle position, vehicle speed, vehicle acceleration and vehicle direction change. The roadside status information includes vehicle speed, speed change, direction change, vehicle position, vehicle posture, and the relationship between the vehicle and the driving environment.

3. A fast detection method for lightweight multimodal vehicle behavior according to claim 1, characterized in that: Perform parallel normalization on the multivariate input signal as a whole and establish a signal matrix for the original input as a whole Std ( σ ), the processing of a single one-dimensional signal is specifically represented by the normalization of the data: ； In the formula, a s is the standardized value, S is the step time, and the corresponding actual input value of the current signal is T s , the overall mean of multi-dimensional input is Mean ( μ ), Std.deviation ( σ ) is the component of the signal matrix in the current single one-dimensional signal; The windowing of two-dimensional image data is as follows: ； By the current image X Divide into windows with horizontal and vertical dimensions ranging from 1 to x t y and n y , with the help of t y and n y For images X Dimensions d Xy The pane processing is performed to finally obtain the pane segmented image d Xy .

4. A fast detection method for lightweight multimodal vehicle behavior according to claim 1, characterized in that: The attention module is built using a fully connected network layer to improve the traditional bidirectional long short-term memory classification network.

5. The method for rapid detection of lightweight multimodal vehicle behavior according to claim 1, characterized in that: S4 is specifically: S41. Obtain the output evaluation weight matrix of the single-layer neural network in the benchmark multimodal vehicle behavior detection network W q ; S42, obtain the output of the original neural network corresponding to the single-layer neural network W k ; S43, calculation W q and W k The three-dimensional spacing between: ； In the formula, D mixed ( W q , W k )for W q and W k The three-dimensional distance between m Indicates the width dimension range of the matrix, j ∈[1, m ], n Represents the length dimension range of the matrix, k ∈[1, n ], i represents the depth of the matrix, l Before the corresponding i -1 depth is the result of scalar signal processing, i The depth represents the image input and the corresponding output matrix, for l The layer matrix ( j, k ) corresponds to the element of the input quantity, for l The layer matrix ( j, k ) position corresponds to the element of the output quantity, For the i The layer matrix ( j, k ) corresponds to the element of the input quantity, For the i The layer matrix ( j, k ) position corresponds to the element of the output, and the scalar input and image input are calculated for the multi-dimensional spacing. i -1 scalar signal The spacing of each input is calculated by corresponding and The spacing is calculated and the root mean square is obtained. For the image input, the target information difference in the pane is calculated separately according to the pane result.

6. A fast detection method for lightweight multimodal vehicle behavior according to claim 5, characterized in that: S5 is as follows: S51, optimizing the single-layer neural network in the benchmark multimodal vehicle behavior detection network based on four improvement rules: parameter dimension scaling, parameter horizontal adjustment, parameter vertical adjustment, and internal parameter adjustment; S52, calculate the output of all optimized single-layer neural networks and W k The three-dimensional spacing between S53, performing cross entropy comparison on all three-dimensional distances and transfer learning performance evaluation results; S54, selecting the best optimization result among the three most matching optimization results, replacing the corresponding single-layer neural network in the benchmark multimodal vehicle behavior detection network, and realizing single-layer neural network tuning; S55. Tune all single-layer neural networks to obtain the final multimodal vehicle behavior detection network.

7. A fast detection method for lightweight multimodal vehicle behavior according to claim 6, characterized in that: The four improvement rules all include two change directions: augmentation and contraction. Among them, parameter dimension scaling, parameter horizontal adjustment, and parameter vertical adjustment have three levels of change in both augmentation and contraction directions. Internal parameter adjustment only performs contraction verification.

8. The method for rapid detection of lightweight multimodal vehicle behavior according to claim 1, characterized in that: The preprocessed data is divided into labeled data sets and unlabeled data sets. The unlabeled data sets are used for single-layer neural network tuning, and the labeled data sets are used to verify the tuned single-layer neural network.

Images