Vision-based Vehicle Steering State Discrimination Method and System

Abstract

The present invention provides a vision-based method and system for discriminating vehicle steering states. The method includes: acquiring multi-dimensional features of the road environment and constructing a dynamic road environment model; using a multi-scale convolutional neural network to extract features of the road geometric features and dynamic environment states; constructing a path prediction model based on the final road environment feature representation to obtain a predicted path; analyzing the adaptability between the road environment and the path to determine whether to trigger an adjustment of the expected steering angle; and transmitting the finally corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path. The present invention provides an adaptive steering state discrimination method that can fully consider complex road environments and dynamic change factors, not only improving the accuracy of discrimination but also enhancing the robustness of the system.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle steering state discrimination, and particularly relates to a method and system for discriminating vehicle steering state based on vision. Background Art

[0002] With the rapid development of intelligent transportation and autonomous driving technologies, accurate discrimination of vehicle steering state has become a core problem in autonomous driving systems. Discrimination of the steering state is crucial for achieving safe and reliable autonomous driving, especially in complex traffic scenarios. Traditional steering state discrimination technologies usually rely on simple models based on sensors or judgments based on vehicle internal information (such as steering angle, vehicle speed, etc.). However, these methods usually cannot achieve high-precision steering discrimination in complex and variable road environments.

[0003] Currently, in the prior art, discrimination of vehicle steering state mostly relies on two mainstream solutions: discrimination methods based on vehicle internal information and discrimination methods based on road environment characteristics. Discrimination methods based on vehicle internal information usually judge whether a vehicle is steering by detecting physical parameters such as steering angle, vehicle speed, and acceleration. The advantage of this type of method is simple implementation and strong real-time performance, but its disadvantages are also very obvious. When the road environment changes complexly (for example, curve changes, lane changes, or the appearance of traffic signs), simply relying on vehicle internal information cannot provide sufficient accuracy and robustness, and false judgments or missed judgments are likely to occur.

[0004] In addition, discrimination methods based on road environment characteristics have gradually become the focus of research. This type of method usually uses sensors such as cameras, lidar (LiDAR), and radars to sense geometric features of the road, such as lane lines, road curvature, traffic signs, etc., and combines machine learning algorithms to judge the steering state of the vehicle. By introducing road environment data, these methods can make up for the deficiencies of relying only on vehicle information and provide higher accuracy, especially in complex road environments. However, this type of method also faces several key challenges: First, the performance of existing vision perception technologies in dynamic environments is often unstable. Especially in different weather, lighting, or complex traffic scenarios, the data of cameras and lidar may be greatly interfered, resulting in inaccurate judgment of the steering state. Second, existing path prediction and road curvature modeling methods, although they can predict the driving trajectory of the vehicle to a certain extent, lack effective coping strategies for some sudden traffic events (such as sudden lane changes, changes in traffic signs, etc.), resulting in poor robustness of the system. Finally, existing methods usually lack the ability to adapt to real-time environmental changes and cannot dynamically adjust discrimination strategies according to complex changes in the environment, resulting in limited accuracy of steering discrimination.

[0005] To solve the above problems, the existing technology urgently needs a steering state discrimination method that can comprehensively utilize various sensor information (such as vision, radar, lidar, etc.), simultaneously fuse the dynamic characteristics of the road environment and the real-time path prediction ability, and have an adaptive adjustment function. Such a method should be able to provide high-precision and high-robustness discrimination ability in complex and changing road environments, and have the ability to respond to sudden traffic events and environmental changes, thereby improving the safety and stability of the autonomous driving system. Summary of the Invention

[0006] The object of the present invention is to propose a vision-based vehicle steering state discrimination method and system, which not only improves the accuracy of discrimination, but also enhances the robustness of the system. Especially in changing urban roads or complex traffic scenarios, it can accurately judge the steering state of the vehicle in real time, thereby providing more reliable technical support for autonomous driving and advanced driver assistance systems (ADAS).

[0007] To achieve the above object, the present invention provides a vision-based vehicle steering state discrimination method, and the method includes:

[0008] Step 1: Obtain the multi-dimensional features of the road environment and construct a dynamic road environment model, and the road environment model outputs road geometric features and dynamic environment states;

[0009] Step 2: Use a multi-scale convolutional neural network to extract features from the road geometric features and dynamic environment states. Each layer of the multi-scale convolutional neural network extracts features of different scales, and the features output by each layer are weighted and fused to obtain multi-scale visual features. A spatio-temporal fusion mechanism is introduced into the multi-scale visual features to weight and fuse the visual features from different times and different scales to obtain the finally fused visual features, and the finally fused visual features are reconstructed into the final road environment feature representation through a fully connected layer;

[0010] Step 3: Construct a path prediction model based on the final road environment feature representation to obtain a predicted path, and derive the future steering angle of the vehicle according to the relationship between the vehicle's current position and the future path. At the same time, a smoothing constraint term is introduced in the derivation process to limit the change range of the steering angle and avoid excessive adjustment, and finally obtain the expected steering angle;

[0011] Step 4: Analyze the adaptability between the road environment and the path to determine whether to trigger an adjustment of the expected steering angle:

[0012] Design a discrimination function to output an adaptive discrimination value for judgment:

[0013] If the adaptive discrimination value is the lowest, it means that the path is the least adaptable and requires the most adjustment;

[0014] Meanwhile, based on the adaptive discrimination value, a steering angle adjustment formula is designed to adjust the expected steering angle to obtain the adjusted steering angle.

[0015] Step Five: Receive the adjusted steering angle, design a steering state discrimination function to determine whether the steering angle matches the current road conditions and vehicle driving state. If the steering angle does not match the actual situation, it may lead to trajectory deviation or vehicle instability, and appropriate feedback adjustment is required. Output the finally corrected steering angle and transfer the finally corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

[0016] Furthermore, the dynamic road environment model is constructed as follows:

[0017] Use a multi-scale convolutional neural network to extract the geometric features of the road. The multi-scale convolutional neural network at different layers can extract road features from local to global. Then introduce a spatio-temporal fusion neural network, enabling the network to automatically assign different weights to features according to the changes in the road environment at different time periods, eliminating the interference of time changes on feature extraction, and obtaining the road geometric features at time t.

[0018] Based on the road geometric features at time t, use the Kalman filter to dynamically model the road environment state, and at the same time combine the motion information of the vehicle to predict and update the road state. Then introduce an adaptive adjustment factor to enable the Kalman filter to automatically adjust the road environment model according to the dynamic changes of the road.

[0019] Furthermore, the multi-scale visual features are processed using a multi-scale convolutional neural network. Among them, each convolutional neural network extracts features of different scales. The goal is to extract high-precision visual features from the complex environment through gradually refined feature maps. Each convolutional layer outputs $F_s$, where s represents different scales, from coarse to fine.

[0020] Furthermore, a spatio-temporal fusion mechanism is introduced into the multi-scale visual features, and dynamic adjustment is performed through temporal modeling based on the LSTM network to perform weighted fusion on visual features from different times and different scales.

[0021] Learn the historical features before and after time t through the LSTM network, and combine real-time perception data to dynamically adjust the fusion weights. By introducing a regularization term, ensure that the change of the weights is smoother and will not cause excessive fluctuations due to sudden changes in the environment.

[0022] Finally, reconstruct it through a fully connected layer into the final representation of the road environment features.

[0023] Further, the path prediction model is a weighted path regression model based on the road environment feature representation, which predicts the future path by combining the current behavior and historical path information of the vehicle.

[0024] Based on the predicted path, calculate the future steering angle of the vehicle; the future steering angle of the vehicle is derived from the relationship between the current position of the vehicle and the future path.

[0025] Further, the road geometric features include lane lines, obstacles, and lane curvature.

[0026] Further, the discriminant function determines whether to adjust the steering angle based on the difference between the path and the road environment.

[0027] Further, the steering state discriminant function is based on considering the current steering angle, the estimated path, and the road features to evaluate the rationality of the steering angle and determine the steering state discriminant value.

[0028] Further, design a feedback adjustment mechanism based on the steering state discriminant function:

[0029] If the steering state discriminant value is lower than the set threshold, adjust the steering angle through feedback to ensure that the vehicle can maintain the correct driving path.

[0030] In the feedback mechanism, when the steering state discriminant value is the lowest, dynamically fine-tune the steering angle through feedback to correct the steering angle towards the correct direction and avoid deviating from the target path.

[0031] In the second aspect of the present invention, a vision-based vehicle steering state discrimination system is provided, and the system includes:

[0032] A road environment model construction unit for obtaining multi-dimensional features of the road environment and constructing a dynamic road environment model, and the road environment model outputs road geometric features and dynamic environment states.

[0033] A road environment analysis unit for extracting features from the road geometric features and dynamic environment states by using a multi-scale convolutional neural network. Each layer of the multi-scale convolutional neural network extracts features of different scales, and the features output by each layer are weighted and fused to obtain multi-scale visual features. A spatio-temporal fusion mechanism is introduced into the multi-scale visual features to weight and fuse visual features from different times and different scales to obtain the finally fused visual features, and the finally fused visual features are reconstructed into the final road environment feature representation through a fully connected layer.

[0034] A path prediction model construction unit, which is used to construct a path prediction model based on the final road environment feature representation to obtain a predicted path, deduce the future steering angle of the vehicle according to the relationship between the current position of the vehicle and the future path, and introduce a smoothing constraint term during the deduction process to limit the change range of the steering angle and avoid excessive adjustment, and finally obtain the expected steering angle;

[0035] A vehicle state discrimination unit, which is used to analyze the adaptability between the road environment and the path, and judge whether to trigger the adjustment of the expected steering angle:

[0036] Design a discrimination function, output an adaptive discrimination value, and make a judgment:

[0037] If the adaptive discrimination value is the lowest, it means that the path is the most mismatched and requires the most adjustments;

[0038] At the same time, design a steering angle adjustment formula based on the adaptive discrimination value to adjust the expected steering angle to obtain the adjusted steering angle;

[0039] A vehicle state correction unit, which is used to receive the adjusted steering angle, design a steering state discrimination function to judge whether the steering angle matches the current road conditions and the vehicle driving state. If the steering angle does not match the actual situation, it may cause trajectory deviation or vehicle instability, and appropriate feedback adjustment is required. Output the finally corrected steering angle and transmit the finally corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

[0040] The beneficial technical effects of the present invention are at least as follows:

[0041] (1) Through the fusion of multiple sensors (such as cameras, lidar, radars), the present invention dynamically captures environmental features such as road curvature, lane width, and traffic signs, and establishes an environmental model in real time. Different from the prior art that relies on a static road model, the present invention can automatically adjust the discrimination criteria according to real-time road changes (such as changes in road curvature, changes in lane width, etc.). This modeling method based on dynamic environmental features solves the problem that the traditional method cannot accurately discriminate the steering state in complex environments (such as sudden traffic changes or curve changes), and improves the accuracy and stability of steering discrimination.

[0042] (2) The present invention adopts a multi-camera system to simultaneously collect visual information at close range and long range. By using a multi-scale convolutional neural network (CNN) to extract and fuse features from image data of different scales, it can accurately identify key visual features such as road markings, traffic signals, and obstacles. This method of multi-scale information fusion overcomes the problem that single-scale visual information in traditional methods may ignore long-range features of the road, ensuring accurate discrimination of the steering state even at long range or in complex scenarios, and improving the comprehensiveness and accuracy of discrimination.

[0043] (3) By combining path prediction and steering state inference, the present invention can predict the future driving trajectory of the vehicle while real-time judging the steering state. Through advanced time series modeling techniques such as LSTM or Transformer, the present invention can accurately predict the upcoming steering behavior of the vehicle (such as entering a curve or intersection), and dynamically adjust the steering state discrimination strategy according to the prediction results. This adaptive mechanism solves the problem in the prior art that it is unable to cope with sudden traffic events (such as sudden changes in traffic signals at intersections or the sudden appearance of obstacles), making the system more flexible and robust.

[0044] (4) The present invention provides an adaptive steering state discrimination method that can fully consider complex road environments and dynamic change factors, not only improving the accuracy of discrimination, but also enhancing the robustness of the system. Especially in variable urban roads or complex traffic scenarios, it can real-time and accurately judge the steering state of the vehicle, thus providing more reliable technical support for autonomous driving and advanced driver assistance systems (ADAS). Description of the Drawings

[0045] The present invention is further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the following drawings.

[0046] Figure 1 It is a flowchart of the method for discriminating the steering state of a vehicle based on vision according to the present invention.

[0047] Figure 2 It is a framework diagram of the system for discriminating the steering state of a vehicle based on vision according to the present invention. Detailed Embodiments

[0048] The embodiments of the present invention are described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0049] Such asFigure 1 As shown in Figure 1 , the vision-based vehicle steering state discrimination method provided by the embodiments of the present invention includes the following steps S1 - S5:

[0050] S1. Obtain multi-dimensional features of the road environment and construct a dynamic road environment model, and the road environment model outputs road geometric features and dynamic environment states.

[0051] Specifically, collect data from multiple sensors, including front-view and side-view cameras, lidar, millimeter-wave radar, etc., to obtain road environment information such as lane lines, obstacles, traffic signs, etc.

[0052] Furthermore, fuse different sensor data, use a dynamic perception fusion algorithm based on spatio-temporal correlation, and dynamically adjust the weight of each sensor according to the state of the sensor. The weight adjustment is based on factors such as signal-to-noise ratio, error rate, environmental conditions (such as illumination), etc. Formula:

[0053] ;

[0054] Wherein, : The observation data of the th sensor at time . : The dynamic weight of the th sensor. : The fused environmental data.

[0055] Furthermore, use a multi-scale convolutional neural network (CNN) to extract road geometric features, covering lane lines, road curvature, lane width, obstacles, etc. Different layers of the CNN can extract road features from local to global. Introduce a spatio-temporal fusion neural network (STF-CNN) so that the network can automatically assign different weights to features according to changes in the road environment at different time periods, eliminating the interference of time changes on feature extraction. Formula:

[0056] ;

[0057] Wherein, : The fused sensor data. : The weight of the th layer of the convolutional neural network, reflecting the importance of features. : The th layer of the convolutional neural network, extracting features at a specific scale. : The road geometric features at time .

[0058] Furthermore, based on the feature extraction results, a Kalman filter is used to dynamically model the road environment state. The motion state of the vehicle (such as vehicle speed, steering angle) affects the prediction of the road environment. Therefore, the road state is predicted and updated by combining the motion information of the vehicle.

[0059] An adaptive adjustment factor is introduced to enable the Kalman filter to automatically adjust the prediction model according to the dynamic changes of the road. The adjustment factor updates the model based on real-time data (such as road curvature changes, lane width changes), further improving the prediction accuracy. The formula:

[0060] ;

[0061] where : the environmental state at time , including information such as road curvature and lane lines. : the state transition matrix, representing the change of the road state over time. : the state of the vehicle, including vehicle speed and steering angle. : the control matrix, representing the impact of the vehicle state on the road environment model. : the process noise, representing the unknown changes in the environmental model.

[0062] Adaptive adjustment formula:

[0063] ;

[0064] where and : the dynamic adjustment matrix at time , dynamically adjusted according to real-time data.

[0065] Finally, a dynamic road environment model is obtained, which contains all real-time information related to road geometric features (such as road curvature, lane lines, obstacles, traffic signs, etc.).

[0066] S2. A multi-scale convolutional neural network is used to extract features from the road geometric features and dynamic environment state. Each layer of the multi-scale convolutional neural network extracts features of different scales, and the features output by each layer are weighted and fused to obtain multi-scale visual features. A spatio-temporal fusion mechanism is introduced into the multi-scale visual features, and the visual features from different times and different scales are weighted and fused to obtain the finally fused visual features. The finally fused visual features are reconstructed into the final road environment feature representation through a fully connected layer.

[0067] Specifically, based on the output of the first step , the present invention uses a multi-scale convolutional neural network (CNN) to process it. The multi-scale network can simultaneously focus on different scale features from local to global, which is crucial for accurately identifying information such as lane lines, obstacles, and road curvature.

[0068] Furthermore, each layer of the convolutional neural network extracts features of different scales, aiming to extract high-precision visual features from complex environments through gradually refined feature maps. Each convolutional layer outputs , where represents different scales (from coarse to fine).

[0069] The output is the multi-scale visual feature at time , which is obtained by weighted fusion of the visual features of each scale. The formula: , where

[0070] ;

[0071] Among them, : the road geometric features and dynamic environment state at time , from the output of step 1. : the visual features after extraction by the multi-scale CNN. : the weight of the th layer of the convolutional neural network, reflecting the importance of the features of this layer. : the th layer of the convolutional neural network, extracting features corresponding to the scale.

[0072] Each layer of the convolutional network extracts features for different scales, using the size of the convolutional kernel of each layer to identify structures of different sizes, from lane lines to distant obstacles. This step ensures that the model can capture visual features at different distances and different levels.

[0073] Furthermore, based on the output of step one , the present invention introduces a spatio-temporal fusion mechanism to perform weighted fusion on visual features from different times and different scales. In a dynamic environment, the features at different time points may have different weights, so different weights need to be assigned to the features at each moment , which can be dynamically adjusted through temporal modeling based on LSTM (Long Short-Term Memory network).

[0074] Spatio-temporal fusion formula:

[0075] ;

[0076] Among them, : the features extracted from the th scale network at time . : The spatio-temporal adaptive weight of the scale feature. The weight is calculated by an LSTM model according to the changes in spatio-temporal features. : The finally fused visual features. :

[0077] Furthermore, the LSTM network is used to learn the historical features before and after time and combines the real-time perception data to dynamically adjust the weight . By introducing a regularization term , it is ensured that the change of the weight is smoother and there will be no excessive fluctuations due to sudden changes in the environment:

[0078] ;

[0079] Among them, : The long short-term memory network is used to learn the dynamic changes of temporal information, : The regularization term is used to smooth the weight change and avoid excessive fluctuations. : The Sigmoid function maps the output of the LSTM to the interval.

[0080] In this step, the LSTM generates the dynamic weight of each scale feature by modeling the visual features of historical and current times . The introduction of the regularization term helps the change of the weight to be more stable and avoids unnecessary biases in the evaluation of the importance of features due to sudden environmental changes.

[0081] Furthermore, based on the features after multi-scale fusion , the present invention further reconstructs them through a fully connected layer into the final road environment feature representation , and this feature will be used for subsequent steering discrimination or path planning. Formula:

[0082] ;

[0083] Among them, : The final output after feature reconstruction, including the environmental features for steering state determination. : The fully connected layer is used to map the fused features to the final road state features.

[0084] The finally output will be used as the basis for steering determination or path planning. The fully connected layer maps the fused features to generate a refined road environment description.

[0085] S3. Construct a path prediction model based on the final representation of the road environment characteristics to obtain a predicted path. Derive the future steering angle of the vehicle based on the relationship between the current position of the vehicle and the future path. At the same time, introduce a smoothing constraint term during the derivation process to limit the change amplitude of the steering angle and avoid excessive adjustment, and finally obtain the expected steering angle.

[0086] Specifically, based on the final representation of the road environment characteristics and the vehicle state , the present invention constructs a path prediction model to generate the trajectory of the vehicle within a future period of time .

[0087] The present invention proposes a weighted path regression model to predict the future path by combining the current behavior of the vehicle and historical path information. The specific formula is as follows:

[0088] ;

[0089] where, : the predicted path at time , representing the future trajectory of the vehicle. : historical path data, including the trajectory points of the vehicle's past driving. : the weighting coefficient, which adjusts the influence of the historical path based on the matching degree between the current state of the vehicle and the road environment. The weighting coefficient is calculated by the following formula:

[0090] ;

[0091] where, : the current vehicle speed. : the road environment characteristics at time , including information such as lane lines and obstacles. : the sensitivity factor, which adjusts the influence of the vehicle state on path prediction.

[0092] This regression model can adaptively generate the future path according to the vehicle behavior and the road environment. By weighting the historical path and the current environment, the path predicted by the model is more in line with the actual driving situation.

[0093] Furthermore, through the predicted path , the present invention can calculate the future steering angle of the vehicle . This steering angle is derived based on the relationship between the current position of the vehicle and the future path. The specific formula is as follows:

[0094] ;

[0095] where, : Moment The expected steering angle, representing the steering angle that the vehicle needs to adjust. : The current position of the vehicle. : The predicted path The future position on it.

[0096] This formula derives the required steering angle by calculating the coordinate differences between the current position of the vehicle and the predicted path points, ensuring that the vehicle travels along the predicted trajectory.

[0097] Furthermore, in order to avoid drastic steering adjustments caused by sudden changes or path prediction errors, the present invention introduces a smoothing constraint term in the calculation of the steering angle. This constraint term limits the change amplitude of the steering angle and avoids excessive adjustment. Its calculation formula is as follows:

[0098] ;

[0099] Where : The smoothing constraint of the steering angle, which limits the change amplitude of the steering angle. : The smoothing factor, which is used to adjust the smoothness of the steering angle. : Moment The actual steering angle.

[0100] This smoothing constraint term ensures that the change of the steering angle is not too drastic, thus improving the driving smoothness.

[0101] It can be understood that through the path prediction model and the steering state inference model, the present invention finally obtains two outputs:

[0102] The predicted path : Provided to the path planning module to describe the expected trajectory of the vehicle in the future for a period of time.

[0103] The expected steering angle : For use by the steering control module to determine the next steering action of the vehicle.

[0104] Through the implementation of this solution, the vehicle can accurately predict the future path and infer the steering state according to the dynamic environment characteristics and the current state, providing stable control decisions for the autonomous driving system.

[0105] S4. Analyze the adaptability between the road environment and the path, and judge whether to trigger the adjustment of the expected steering angle:

[0106] Design a discrimination function, output an adaptive discrimination value, and make a judgment:

[0107] If the adaptive discrimination value is the lowest, it means that the path is the least adaptable and requires the most adjustment;

[0108] Design a steering angle adjustment formula based on an adaptive discrimination value to adjust the expected steering angle and obtain the adjusted steering angle.

[0109] Specifically, the input of this step is the predicted path output in Step 3 and the expected steering angle , and the current road characteristics obtained from the final road environment feature representation .

[0110] It should be noted that is obtained by predicting the future path, while is based on the steering angle inferred in the previous step.

[0111] Furthermore, the goal is to use the road environment information and the predicted path through an adaptive discrimination mechanism to determine whether the steering angle needs to be adjusted. If the predicted path does not match the road environment or the steering angle exceeds the safety threshold, the steering strategy needs to be adjusted in a timely manner.

[0112] Furthermore, design a discrimination function to determine whether to trigger the adjustment of the steering angle according to the adaptability between the road environment and the path. Assume that information such as the current road curvature and road width has been obtained through the final road environment feature representation . If the predicted path has a large gap with the final road environment feature representation , it indicates that there may be a deviation in path prediction and needs to be corrected. The discrimination function is defined as:

[0113] ;

[0114] where : Adaptive discrimination value, reflecting the matching degree between the path and the environment. : The future path obtained from Step 3. : The final road environment feature representation obtained through Step 2. : Adjustment factor, used to adjust the sensitivity of the discrimination function. The discrimination function determines whether to adjust the steering angle based on the difference between the path and the road environment. When the gap is larger, the lower the value, indicating that the path is not well - matched and more adjustments are needed.

[0115] Furthermore, based on the output of the discrimination function, the present invention designs a steering angle adjustment formula. If the discrimination value is low, it indicates that the path does not match the actual road characteristics and the steering angle needs to be corrected through the adjustment factor. The adaptive adjustment formula is:

[0116] ;

[0117] Among them, : The adjusted steering angle, which is the final output. : The original target steering angle obtained from step 3. : The adaptive adjustment factor, which is dynamically adjusted based on the value of the discrimination function, and the discrimination value is inversely proportional. : The fine-tuning amount of the steering angle, which reflects the difference between the predicted path and the road characteristics. Through this adaptive adjustment formula, the system can adjust the steering angle in real time to adapt to the changing road environment and path prediction, thereby improving the steering accuracy and driving safety.

[0118] Furthermore, the finally output adjusted steering angle , as the input of the vehicle control system, is used to adjust the actual driving direction of the vehicle. At the same time, the discrimination function value is output, which is used to evaluate the matching situation between the current path and the road environment, and as feedback information to help with subsequent path optimization and adjustment of the environmental perception strategy.

[0119] Through this adaptive discrimination mechanism, the system can make more intelligent steering decisions in complex or uncertain road environments, ensuring that the vehicle's path and driving direction always maintain high precision and high safety in a changing environment.

[0120] S5. Receive the adjusted steering angle, design a steering state discrimination function to determine whether the steering angle matches the current road conditions and vehicle driving state. If the steering angle does not match the actual situation, it may lead to trajectory deviation or vehicle instability, and appropriate feedback adjustment is required. Output the finally corrected steering angle, and transmit the finally corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

[0121] Specifically, the goal of this step is to determine whether the steering angle matches the current road conditions and vehicle driving state. If the steering angle does not match the actual situation, it may lead to trajectory deviation or vehicle instability, and appropriate feedback adjustment is required.

[0122] Furthermore, the present invention designs a steering state discrimination function , which comprehensively considers the current steering angle, the predicted path and the road characteristics to evaluate the rationality of the steering angle. If the discrimination value is low, it indicates that there is a deviation in the steering angle and feedback correction is required. The design of the discrimination function can be quantified in the following way:

[0123] ;

[0124] Among them, : The adjusted steering angle, from step 4. : The target steering angle, obtained from path prediction and environmental analysis. : The maximum steering angle allowed by the system, used to normalize the steering deviation. : The weighting factor, which adjusts the tolerance of the deviation between the target angle and the actual angle.

[0125] Through this discriminant function, the present invention can obtain a quantization value , indicating the rationality of the current steering state. If is relatively low, it indicates that there is a large steering deviation, and the system should activate the feedback mechanism.

[0126] Furthermore, based on the steering state discriminant value , a feedback adjustment mechanism is designed. If is lower than the set threshold, the system will adjust the steering angle through feedback to ensure that the vehicle can maintain the correct driving path. The steering angle feedback formula can be expressed as follows:

[0127] ;

[0128] Among them, : The corrected steering angle, output as the feedback result. : The feedback gain factor, which controls the adjustment strength. : The steering angle fine-tuning amount, indicating the magnitude of the correction.

[0129] In the feedback mechanism, when the discriminant value is relatively low, The feedback correction controlled by will be fine-tuned to correct the steering angle towards the correct direction and avoid deviating from the target path.

[0130] It can be understood that the final output of this step is the corrected steering angle , which will be transmitted to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path. At the same time, the discriminant value is output, providing a basis for subsequent path optimization. Through this steering state discrimination and feedback mechanism, the system can detect the rationality of the current steering angle in real time and perform dynamic adjustment when necessary to ensure that the vehicle can stably and accurately travel along the predetermined path.

[0131] As Figure 2 shown, in another embodiment of the present invention, a vision-based vehicle steering state discrimination system is provided. The system includes:

[0132] The road environment model construction unit 1011 is used to obtain multi-dimensional features of the road environment and construct a dynamic road environment model, and the road environment model outputs road geometric features and dynamic environment states;

[0133] The road environment analysis unit 1012 is used to extract features from the road geometric features and dynamic environment states by using a multi-scale convolutional neural network. Each layer of the multi-scale convolutional neural network extracts features of different scales, and the features output by each layer are weighted and fused to obtain multi-scale visual features. A spatio-temporal fusion mechanism is introduced into the multi-scale visual features, and the visual features from different times and different scales are weighted and fused to obtain the finally fused visual features. The finally fused visual features are reconstructed into the final road environment feature representation through a fully connected layer;

[0134] The path prediction model construction unit 1013 is used to construct a path prediction model according to the final road environment feature representation to obtain a predicted path. According to the relationship between the current position of the vehicle and the future path of the predicted path, the future steering angle of the vehicle is deduced. At the same time, a smoothing constraint term is introduced in the deduction process to limit the change range of the steering angle and avoid excessive adjustment, and finally the expected steering angle is obtained;

[0135] The vehicle state discrimination unit 1014 is used to analyze the adaptability between the road environment and the path, and judge whether to trigger the adjustment of the expected steering angle:

[0136] Design a discrimination function to output an adaptive discrimination value for judgment:

[0137] If the adaptive discrimination value is the lowest, it means that the path is the most unsuitable and requires the most adjustments;

[0138] Based on the adaptive discrimination value, design a steering angle adjustment formula to adjust the expected steering angle to obtain the adjusted steering angle;

[0139] The vehicle state correction unit 1015 is used to receive the adjusted steering angle, design a steering state discrimination function to judge whether the steering angle matches the current road conditions and the vehicle driving state. If the steering angle does not match the actual situation, it may lead to trajectory deviation or vehicle instability, and appropriate feedback adjustment is required. The finally corrected steering angle is output, and the finally corrected steering angle is transmitted to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

[0140] It should be noted that the above-described work process is only illustrative and does not limit the protection scope of the present invention. In actual applications, those skilled in the art can select some or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and no limitation is made here.

[0141] In addition, for the technical details not described in detail in this embodiment, reference may be made to the parameter operation method provided in any embodiment of the present invention, which will not be elaborated herein.

[0142] It should be noted that, in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or system including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or system including such element.

[0143] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0144] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as a read-only memory / random access memory, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0145] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for distinguishing vehicle steering state based on vision, characterized in that: The method comprises: Step 1: Acquire multi-dimensional features of the road environment and construct a dynamic road environment model, wherein the road environment model outputs road geometric features and dynamic environment states; Step 2: A multi-scale convolutional neural network is used to extract features of the road geometric features and dynamic environment states. Each layer of the multi-scale convolutional neural network extracts features of different scales, and the features output by each layer are weighted and fused to obtain multi-scale visual features. A spatiotemporal fusion mechanism is introduced into the multi-scale visual features, and visual features from different times and scales are weighted and fused to obtain the final fused visual features. The final fused visual features are reconstructed into the final road environment feature representation through a fully connected layer. Step 3: construct a path prediction model according to the final road environment feature representation to obtain a predicted path, and deduce the future steering angle of the vehicle according to the predicted path through the relationship between the current position of the vehicle and the future path. At the same time, a smooth constraint term is introduced in the derivation process to limit the change range of the steering angle to avoid excessive adjustment, and finally obtain the expected steering angle; Step 4: Analyze the compatibility between the road environment and the path to determine whether to trigger the adjustment of the expected steering angle: Design the discriminant function, output the adaptive discriminant value, and make judgments: If the adaptive discrimination value is the lowest, it means that the path is the least suitable and needs the most adjustments; At the same time, a steering angle adjustment formula is designed based on the adaptive discriminant value, and the expected steering angle is adjusted to obtain the adjusted steering angle; Step 5: Receive the adjusted steering angle, design a steering state discrimination function to determine whether the steering angle matches the current road conditions and vehicle driving state. If the steering angle does not match the actual situation, appropriate feedback adjustment is required to output the final corrected steering angle, and transmit the final corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path; The dynamic road environment model is constructed as follows: A multi-scale convolutional neural network is used to extract the geometric features of the road. The multi-scale convolutional neural network at different layers can extract road features from local to global. Then the spatiotemporal fusion neural network is introduced, so that the network can automatically assign different weights to the features according to the changes in the road environment in different time periods, eliminate the interference of time changes on feature extraction, and obtain the time-to-time information. The road geometry characteristics; Based on time Based on the road geometry characteristics, the Kalman filter is used to dynamically model the road environment state. At the same time, the road state is predicted and updated in combination with the vehicle's motion information. Then, an adaptive adjustment factor is introduced so that the Kalman filter can automatically adjust the road environment model according to the dynamic changes of the road.

2. The method for determining vehicle steering state based on vision according to claim 1, characterized in that: The multi-scale visual features are processed by a multi-scale convolutional neural network, where each layer of the convolutional neural network extracts features of different scales. The goal is to extract high-precision visual features from complex environments through gradually refined feature maps. Each convolutional layer outputs ,in Indicates different scales, from coarse to fine.

3. The method for determining vehicle steering state based on vision according to claim 2, characterized in that: A spatiotemporal fusion mechanism is introduced into the multi-scale visual features, and dynamic adjustment is performed through temporal modeling based on an LSTM network to weightedly fuse visual features from different times and scales; Learning moments through LSTM networks The historical features before and after are combined with real-time perception data to dynamically adjust the fusion weights. By introducing regularization terms, the weight changes are ensured to be smoother and will not cause excessive fluctuations due to sudden changes in the environment. Finally, it is reconstructed into the final road environment feature representation through a fully connected layer.

4. The method for determining a vehicle turning state based on vision according to claim 3, characterized in that: The path prediction model is a weighted path regression model based on road environment characteristics, which predicts the future path by combining the current behavior and historical path information of the vehicle; The future steering angle of the vehicle is calculated through the predicted path; the future steering angle of the vehicle is derived through the relationship between the current position of the vehicle and the future path.

5. The method for determining vehicle steering state based on vision according to claim 1, characterized in that: The road geometric features include lane lines, obstacles and lane curvature.

6. The method for determining vehicle steering state based on vision according to claim 4, characterized in that: The discriminant function determines whether the steering angle needs to be adjusted based on the difference between the path and the road environment.

7. The method for determining vehicle steering state based on vision according to claim 6, characterized in that: The steering state discrimination function determines the steering state discrimination value based on considering the current steering angle, the estimated path and the road characteristics to evaluate the rationality of the steering angle.

8. The method for determining vehicle steering state based on vision according to claim 7, characterized in that: Design a feedback adjustment mechanism based on the steering state discrimination function: If the steering state judgment value is lower than the set threshold, the steering angle will be adjusted through feedback to ensure that the vehicle can maintain the correct driving path; In the feedback mechanism, when the steering state judgment value is the lowest, the steering angle is dynamically fine-tuned through feedback so that the steering angle is corrected in the correct direction to avoid deviation from the target path.

9. A vehicle steering state discrimination system based on vision, characterized in that: The system comprises: A road environment model building unit, used to obtain multi-dimensional features of the road environment and build a dynamic road environment model, wherein the road environment model outputs road geometric features and dynamic environment states; A road environment analysis unit, configured to extract features of the road geometric features and dynamic environment states by using a multi-scale convolutional neural network, wherein each layer of the multi-scale convolutional neural network extracts features of different scales, and weightedly fuses the features output by each layer to obtain multi-scale visual features, introduces a spatiotemporal fusion mechanism into the multi-scale visual features, and weightedly fuses visual features from different times and scales to obtain final fused visual features, and reconstructs the final road environment feature representation through a fully connected layer based on the final fused visual features; a path prediction model construction unit, configured to construct a path prediction model according to the final road environment feature representation to obtain a predicted path, and derive the future steering angle of the vehicle according to the predicted path through the relationship between the current position of the vehicle and the future path, while introducing a smooth constraint term in the derivation process to limit the range of change of the steering angle to avoid over-adjustment, and finally obtain the expected steering angle; The vehicle state determination unit is used to analyze the compatibility between the road environment and the path, and determine whether to trigger the adjustment of the expected steering angle: Design the discriminant function, output the adaptive discriminant value, and make judgments: If the adaptive discrimination value is the lowest, it means that the path is the least suitable and needs the most adjustments; At the same time, a steering angle adjustment formula is designed based on the adaptive discriminant value, and the expected steering angle is adjusted to obtain the adjusted steering angle; The vehicle state correction unit is used to receive the adjusted steering angle, design a steering state discrimination function to determine whether the steering angle matches the current road conditions and the vehicle driving state, and if the steering angle does not match the actual situation, appropriate feedback adjustment is required to output the final corrected steering angle, and transmit the final corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path; The dynamic road environment model is constructed as follows: A multi-scale convolutional neural network is used to extract the geometric features of the road. The multi-scale convolutional neural network at different layers can extract road features from local to global. Then the spatiotemporal fusion neural network is introduced, so that the network can automatically assign different weights to the features according to the changes in the road environment in different time periods, eliminate the interference of time changes on feature extraction, and obtain the time-to-time information. The road geometry characteristics; Based on time Based on the road geometry features, the Kalman filter is used to dynamically model the road environment state. At the same time, the road state is predicted and updated in combination with the vehicle's motion information, and then adaptive adjustments are introduced.

Images