Vision-based vehicle steering state judgment method and system

Abstract

The invention provides a vision-based vehicle steering state judgment method and system. The method comprises the following steps: acquiring multi-dimensional features of a road environment and constructing a dynamic road environment model; performing feature extraction on the road geometric features and the dynamic environment state by adopting a multi-scale convolutional neural network; constructing a path prediction model according to the final road environment feature representation to obtain a predicted path; the adaptability between the road environment and the path is analyzed, and whether adjustment of the expected steering angle is triggered or not is judged; the final corrected steering angle is transmitted to a vehicle control system for actual operation, and it is ensured that the vehicle runs according to the optimal path. The self-adaptive steering state judgment method provided by the invention can fully consider complex road environments and dynamic change factors, not only improves the judgment accuracy, but also enhances the robustness of the system.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle steering state determination, and in particular relates to a vehicle steering state determination method and system based on vision. Background Art

[0002] With the rapid development of intelligent transportation and autonomous driving technologies, accurate vehicle steering state identification has become a core problem in autonomous driving systems. Steering state identification is crucial to achieving safe and reliable autonomous driving, especially in complex traffic scenarios. Traditional steering state identification technologies usually rely on simple sensor-based models or make judgments based on vehicle internal information (such as steering angle, vehicle speed, etc.). However, these methods usually cannot achieve high-precision steering identification in complex and changing road environments.

[0003] At present, in the existing technology, the steering state judgment of the vehicle mostly relies on two mainstream solutions: the judgment method based on the internal information of the vehicle and the judgment method based on the characteristics of the road environment. The judgment method based on the internal information of the vehicle usually determines whether the vehicle is turning by detecting physical parameters such as steering angle, vehicle speed, acceleration, etc. The advantages of this type of method are simple implementation and strong real-time performance, but its disadvantages are also very obvious. When the road environment undergoes complex changes (for example, changes in curves, lanes, or the appearance of traffic signs), relying solely on the internal information of the vehicle cannot provide sufficient accuracy and robustness, and it is easy to make misjudgments or missed judgments.

[0004] In addition, the discrimination methods based on road environment characteristics have gradually become the focus of research. Such methods usually use sensors such as cameras, LiDAR, and radar to perceive the geometric features of the road, such as lane lines, road curvature, traffic signs, etc., and combine machine learning algorithms to judge the steering state of the vehicle. By introducing road environment data, these methods can make up for the shortcomings of relying solely on vehicle information and provide higher accuracy, especially in complex road environments. However, such methods also face several key challenges: First, the performance of existing visual perception technologies in dynamic environments is often unstable, especially in different weather, lighting or complex traffic scenes, the data of cameras and LiDARs may be greatly disturbed, resulting in inaccurate judgment of the steering state. Secondly, although the existing path prediction and road curvature modeling methods can predict the vehicle's driving trajectory to a certain extent, they lack effective response strategies for some sudden traffic events (such as sudden lane changes, changes in traffic signs, etc.), resulting in poor system robustness. Finally, existing methods usually lack the ability to adapt to real-time environmental changes and cannot dynamically adjust the discrimination strategy according to the complex changes in the environment, resulting in limited accuracy of steering discrimination.

[0005] In order to solve the above problems, the existing technology urgently needs a steering state identification method that can comprehensively utilize multiple sensor information (such as vision, radar, lidar, etc.), integrate the dynamic characteristics of the road environment and real-time path prediction capabilities, and have adaptive adjustment functions. This method should be able to provide high-precision and high-robustness identification capabilities in complex and changeable road environments, and have the ability to respond to sudden traffic events and environmental changes, thereby improving the safety and stability of the autonomous driving system. Summary of the invention

[0006] The purpose of the present invention is to propose a vehicle steering state judgment method and system based on vision, which not only improves the accuracy of judgment, but also enhances the robustness of the system, especially in changeable urban roads or complex traffic scenes, and can judge the steering state of the vehicle in real time and accurately, thereby providing more reliable technical support for autonomous driving and advanced driver assistance systems (ADAS).

[0007] In order to achieve the above object, the present invention provides a method for distinguishing a vehicle steering state based on vision, the method comprising: Step 1: Acquire multi-dimensional features of the road environment and construct a dynamic road environment model, wherein the road environment model outputs road geometric features and dynamic environment states; Step 2: A multi-scale convolutional neural network is used to extract features of the road geometric features and dynamic environment states. Each layer of the multi-scale convolutional neural network extracts features of different scales, and the features output by each layer are weighted and fused to obtain multi-scale visual features. A spatiotemporal fusion mechanism is introduced into the multi-scale visual features, and visual features from different times and scales are weighted and fused to obtain the final fused visual features. The final fused visual features are reconstructed into the final road environment feature representation through a fully connected layer. Step 3: construct a path prediction model according to the final road environment feature representation to obtain a predicted path, and deduce the future steering angle of the vehicle according to the predicted path through the relationship between the current position of the vehicle and the future path. At the same time, a smooth constraint term is introduced in the derivation process to limit the change range of the steering angle to avoid excessive adjustment, and finally obtain the expected steering angle; Step 4: Analyze the compatibility between the road environment and the path to determine whether to trigger the adjustment of the expected steering angle: Design the discriminant function, output the adaptive discriminant value, and make judgments: If the adaptive discrimination value is the lowest, it means that the path is the least suitable and needs the most adjustments; At the same time, a steering angle adjustment formula is designed based on the adaptive discriminant value, and the expected steering angle is adjusted to obtain the adjusted steering angle; Step 5. Receive the adjusted steering angle and design a steering state discrimination function to determine whether the steering angle matches the current road conditions and vehicle driving status. If the steering angle does not match the actual situation, it may cause trajectory deviation or vehicle instability. Appropriate feedback adjustment is required to output the final corrected steering angle and pass the final corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

[0008] Furthermore, the dynamic road environment model is constructed as follows: A multi-scale convolutional neural network is used to extract the geometric features of the road. The multi-scale convolutional neural network at different layers can extract road features from local to global. Then the spatiotemporal fusion neural network is introduced, so that the network can automatically assign different weights to the features according to the changes in the road environment in different time periods, eliminate the interference of time changes on feature extraction, and obtain the time-to-time information. The road geometry characteristics; Based on time Based on the road geometry characteristics, the Kalman filter is used to dynamically model the road environment state. At the same time, the road state is predicted and updated in combination with the vehicle's motion information. Then, an adaptive adjustment factor is introduced so that the Kalman filter can automatically adjust the road environment model according to the dynamic changes of the road.

[0009] Furthermore, the multi-scale visual features are processed using a multi-scale convolutional neural network, wherein each layer of the convolutional neural network extracts features of different scales, with the goal of extracting high-precision visual features from complex environments through gradually refined feature maps. Each convolutional layer outputs ,in Indicates different scales, from coarse to fine.

[0010] Furthermore, a spatiotemporal fusion mechanism is introduced into the multi-scale visual features, and dynamic adjustment is performed through temporal modeling based on the LSTM network to weightedly fuse visual features from different times and scales; Learning moments through LSTM networks The historical features before and after are combined with real-time perception data to dynamically adjust the fusion weights. By introducing regularization terms, the weight changes are ensured to be smoother and will not cause excessive fluctuations due to sudden changes in the environment. Finally, it is reconstructed into the final road environment feature representation through a fully connected layer.

[0011] Furthermore, the path prediction model is a weighted path regression model based on road environment feature representation, which predicts the future path by combining the current behavior and historical path information of the vehicle; The future steering angle of the vehicle is calculated through the predicted path; the future steering angle of the vehicle is derived through the relationship between the current position of the vehicle and the future path.

[0012] Furthermore, the road geometric features include lane lines, obstacles and lane curvature.

[0013] Furthermore, the discriminant function determines whether the steering angle needs to be adjusted according to the difference between the path and the road environment.

[0014] Furthermore, the steering state discrimination function determines the steering state discrimination value based on considering the current steering angle, the estimated path and the road characteristics to evaluate the rationality of the steering angle.

[0015] Furthermore, a feedback adjustment mechanism is designed based on the steering state discrimination function: If the steering state judgment value is lower than the set threshold, the steering angle will be adjusted through feedback to ensure that the vehicle can maintain the correct driving path; In the feedback mechanism, when the steering state judgment value is the lowest, the steering angle is dynamically fine-tuned through feedback so that the steering angle is corrected in the correct direction to avoid deviation from the target path.

[0016] In a second aspect of the present invention, a vehicle steering state determination system based on vision is provided, the system comprising: A road environment model building unit, used to obtain multi-dimensional features of the road environment and build a dynamic road environment model, wherein the road environment model outputs road geometric features and dynamic environment states; A road environment analysis unit, configured to extract features of the road geometric features and dynamic environment states by using a multi-scale convolutional neural network, wherein each layer of the multi-scale convolutional neural network extracts features of different scales, and weightedly fuses the features output by each layer to obtain multi-scale visual features, introduces a spatiotemporal fusion mechanism into the multi-scale visual features, and weightedly fuses visual features from different times and scales to obtain final fused visual features, and reconstructs the final road environment feature representation through a fully connected layer based on the final fused visual features; a path prediction model construction unit, configured to construct a path prediction model according to the final road environment feature representation to obtain a predicted path, and derive the future steering angle of the vehicle according to the predicted path through the relationship between the current position of the vehicle and the future path, while introducing a smooth constraint term in the derivation process to limit the range of change of the steering angle to avoid over-adjustment, and finally obtain the expected steering angle; The vehicle state determination unit is used to analyze the compatibility between the road environment and the path, and determine whether to trigger the adjustment of the expected steering angle: Design the discriminant function, output the adaptive discriminant value, and make judgments: If the adaptive discrimination value is the lowest, it means that the path is the least suitable and needs the most adjustments; At the same time, a steering angle adjustment formula is designed based on the adaptive discriminant value, and the expected steering angle is adjusted to obtain the adjusted steering angle; The vehicle state correction unit is used to receive the adjusted steering angle and design a steering state discrimination function to determine whether the steering angle matches the current road conditions and vehicle driving status. If the steering angle does not match the actual situation, it may cause trajectory deviation or vehicle instability. Appropriate feedback adjustment is required to output the final corrected steering angle and pass the final corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

[0017] The beneficial technical effects of the present invention are at least as follows: (1) The present invention uses multiple sensors (such as cameras, lidar, radar) to dynamically capture environmental features such as road curvature, lane width, and traffic signs, and establish an environmental model in real time. Unlike the prior art that relies on static road models, the present invention can automatically adjust the judgment criteria according to real-time road changes (such as changes in road curvature, changes in lane width, etc.). This modeling method based on dynamic environmental features solves the problem that traditional methods cannot accurately judge the steering state in complex environments (such as sudden traffic changes or curve changes), and improves the accuracy and stability of steering judgment.

[0018] (2) The present invention uses a multi-camera system to collect visual information at both close and long distances. By extracting and fusing image data of different scales using a multi-scale convolutional neural network (CNN), key visual features such as road markings, traffic signals, and obstacles can be accurately identified. This multi-scale information fusion method overcomes the problem that single-scale visual information in traditional methods may ignore long-distance features of the road, ensuring that the turning state can be accurately determined even at long distances or in complex scenes, thereby improving the comprehensiveness and accuracy of the determination.

[0019] (3) The present invention combines path prediction and steering state inference. The present invention can predict the future driving trajectory of the vehicle while judging the steering state in real time. Through advanced time series modeling technologies such as LSTM or Transformer, the present invention can accurately predict the vehicle's upcoming steering behavior (such as entering a curve or intersection) and dynamically adjust the steering state judgment strategy based on the prediction results. This adaptive mechanism solves the problem that the existing technology cannot cope with sudden traffic events (such as sudden changes in traffic signals at intersections or sudden obstacles), making the system more flexible and robust.

[0020] (4) The present invention provides an adaptive steering state determination method that can fully consider complex road environments and dynamically changing factors, which not only improves the accuracy of determination, but also enhances the robustness of the system. In particular, it can accurately determine the steering state of the vehicle in real time under changeable urban roads or complex traffic scenarios, thereby providing more reliable technical support for autonomous driving and advanced driver assistance systems (ADAS). BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0022] Figure 1 This is a flow chart of the vehicle steering state identification method based on vision of the present invention.

[0023] Figure 2 This is a framework diagram of the vision-based vehicle steering state identification system of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0025] like Figure 1 As shown, the vehicle steering state determination method based on vision provided by an embodiment of the present invention comprises the following steps S1-S5: S1. Acquire multi-dimensional features of the road environment and construct a dynamic road environment model, wherein the road environment model outputs road geometric features and dynamic environment status.

[0026] Specifically, it collects data from multiple sensors, including front-view and side-view cameras, lidar, millimeter-wave radar, etc., to obtain road environmental information such as lane lines, obstacles, traffic signs, etc.

[0027] Furthermore, the data from different sensors are fused, and a dynamic perception fusion algorithm based on spatiotemporal correlation is used to dynamically adjust the weight of each sensor according to the state of the sensor. The weight adjustment is based on factors such as signal-to-noise ratio, error rate, and environmental conditions (such as light). Formula: ; in, : No. Sensors at all times Observation data. : No. Dynamic weights of sensors. : Fused environmental data.

[0028] Furthermore, a multi-scale convolutional neural network (CNN) is used to extract the geometric features of the road, including lane lines, road curvature, lane width, obstacles, etc. CNNs at different layers can extract road features from local to global. The introduction of the spatiotemporal fusion neural network (STF-CNN) enables the network to automatically assign different weights to features according to changes in the road environment in different time periods, eliminating the interference of time changes on feature extraction. Formula: ; in, : Fused sensor data. : No. The weights of the convolutional neural network reflect the importance of the features. : No. Layer convolutional neural network to extract features of specific scales. :time The geometric characteristics of the road.

[0029] Furthermore, based on the feature extraction results, a Kalman filter is used to dynamically model the road environment state. The vehicle's motion state (such as speed and steering angle) affects the prediction of the road environment, so the road state is predicted and updated in combination with the vehicle's motion information.

[0030] The introduction of adaptive adjustment factors enables the Kalman filter to automatically adjust the prediction model according to the dynamic changes of the road. The adjustment factor will update the model based on real-time data (such as changes in road curvature and lane width) to further improve the prediction accuracy. Formula: ; in, :time The environmental status of the vehicle, including road curvature, lane lines and other information. : State transfer matrix, which represents the change of road status over time. : The state of the vehicle, including speed and steering angle. : Control matrix, which represents the impact of vehicle status on the road environment model. : Process noise, representing unknown changes in the environment model.

[0031] Adaptive adjustment formula: ; in, and :time The dynamic adjustment matrix is ​​dynamically adjusted according to real-time data.

[0032] Finally, a dynamic road environment model is obtained , the model contains all real-time information related to road geometric features (such as road curvature, lane lines, obstacles, traffic signs, etc.).

[0033] S2. A multi-scale convolutional neural network is used to extract features of the road geometric features and dynamic environmental status. Each layer of the multi-scale convolutional neural network extracts features of different scales. The features output by each layer are weightedly fused to obtain multi-scale visual features. A spatiotemporal fusion mechanism is introduced into the multi-scale visual features. Visual features from different times and scales are weightedly fused to obtain final fused visual features. The final fused visual features are reconstructed into final road environment feature representations through a fully connected layer.

[0034] Specifically, based on the output of the first step , the present invention uses a multi-scale convolutional neural network (CNN) to process it. The multi-scale network can simultaneously focus on different scale features from local to global, which is crucial for accurately identifying lane lines, obstacles, road curvature and other information.

[0035] Furthermore, each convolutional neural network layer extracts features of different scales, with the goal of extracting high-precision visual features from complex environments through gradually refined feature maps. Each convolutional layer outputs ,in Indicates different scales (from coarse to fine).

[0036] Output is time Multi-scale visual features , which is obtained by weighted fusion of visual features at each scale. Formula: ; in, :time The road geometry characteristics and dynamic environment status come from the output of step 1. : Visual features after multi-scale CNN extraction. : No. The weight of a layer of convolutional neural network reflects the importance of the features of this layer. : No. Layer convolutional neural network to extract features of corresponding scales.

[0037] Each layer of the convolutional network extracts features of different scales, using the size of the convolution kernel at each layer to identify structures of different sizes, from lane lines to distant obstacles. This step ensures that the model can capture visual features of different distances and levels.

[0038] Furthermore, based on the output of step 1 , the present invention introduces a spatiotemporal fusion mechanism to weightedly fuse visual features from different times and scales. In a dynamic environment, features at different time points may have different weights, so it is necessary to assign different weights to features at each moment. , which can be dynamically adjusted through LSTM (Long Short-Term Memory Network) based timing modeling.

[0039] Space-time fusion formula: ; in, :time From Features extracted by scaling networks. : No. Spatiotemporal adaptive weights of scale features. The weights are calculated through the LSTM model based on the changes in spatiotemporal features. : The final fused visual features.

[0040] Furthermore, the LSTM network is used to learn the moment The historical features before and after are combined with real-time perception data to dynamically adjust the weights By introducing the regularization term , ensuring that the weight changes are smoother and do not cause excessive fluctuations due to sudden changes in the environment:

[0041] ;

[0042] in, : Long short-term memory network, used to learn the dynamic changes of temporal information, : Regularization term, used to smooth weight changes and avoid excessive fluctuations. : Sigmoid function, mapping the output of LSTM to interval.

[0043] In this step, LSTM generates dynamic weights for each scale feature by modeling the visual features of historical moments and the current moment. . Regularization term The introduction of helps the weight change more smoothly and avoids unnecessary deviations in the importance assessment of features due to sudden environmental changes.

[0044] Furthermore, based on the multi-scale fusion features The present invention further reconstructs it into the final road environment feature representation through the fully connected layer , this feature will be used for subsequent steering discrimination or path planning. Formula: ; in, : The final output after feature reconstruction, including environmental features for steering state determination. : Fully connected layer, used to map the fused features to the final road state features.

[0045] The final output It will serve as the basis for turning decisions or path planning. The fully connected layer maps the fused features to generate a refined description of the road environment.

[0046] S3. Construct a path prediction model based on the final road environment feature representation to obtain a predicted path, and deduce the future steering angle of the vehicle based on the predicted path through the relationship between the current position of the vehicle and the future path. At the same time, a smooth constraint term is introduced in the derivation process to limit the range of change of the steering angle to avoid excessive adjustment, and finally the expected steering angle is obtained.

[0047] Specifically, based on the final road environment feature representation and vehicle status The present invention constructs a path prediction model to generate the trajectory of the vehicle in the future .

[0048] The present invention proposes a weighted path regression model to predict the future path by combining the current behavior of the vehicle and the historical path information. The specific formula is as follows: ; in, :time The predicted path represents the future trajectory of the vehicle. : Historical path data, including the trajectory points of past vehicle travel. : Weighting coefficient, which adjusts the impact of the historical path based on the matching degree between the vehicle's current state and the road environment. Weighting coefficient It is calculated by the following formula: ; in, : The current speed of the vehicle. :time The road environment features include lane lines, obstacles and other information. : Sensitivity factor, which adjusts the impact of vehicle status on path prediction.

[0049] The regression model can adaptively generate future paths based on vehicle behavior and road conditions. By weighting historical paths and current conditions, the model predicts paths that are more consistent with actual driving situations.

[0050] Furthermore, through the predicted path , the present invention can calculate the future steering angle of the vehicle The steering angle is derived from the relationship between the vehicle's current position and future path. The specific formula is as follows: ; in, :time The expected steering angle indicates the steering angle that the vehicle needs to adjust. : The current position of the vehicle. : Prediction path future position on.

[0051] The formula calculates the coordinate difference between the vehicle's current position and the predicted path points to derive the required steering angle, ensuring that the vehicle follows the predicted trajectory.

[0052] Furthermore, in order to avoid drastic steering adjustments due to sudden changes or path prediction errors, the present invention introduces a smoothing constraint term in the calculation of the steering angle. , this constraint limits the change range of the steering angle to avoid over-adjustment. Its calculation formula is as follows: ; in, : Smoothness constraint of the steering angle, limiting the range of change of the steering angle. : Smoothing factor, used to adjust the smoothness of the steering angle. :time The actual steering angle.

[0053] This smoothness constraint ensures that the change of the steering angle is not too drastic, thereby improving the driving smoothness.

[0054] It can be understood that through the path prediction model and the steering state inference model, the present invention finally obtains two outputs: Predicted Path : Provided to the path planning module to describe the expected trajectory of the vehicle in the future.

[0055] Expected steering angle : Used by the steering control module to determine the vehicle's next steering action.

[0056] Through the implementation of this solution, the vehicle can accurately predict the future path and infer the steering state based on the dynamic environment characteristics and current state, providing stable control decisions for the autonomous driving system.

[0057] S4. Analyze the compatibility between the road environment and the path to determine whether to trigger the adjustment of the expected steering angle: Design the discriminant function, output the adaptive discriminant value, and make judgments: If the adaptive discrimination value is the lowest, it means that the path is the least suitable and needs the most adjustments; A steering angle adjustment formula is designed based on the adaptive discriminant value, and the expected steering angle is adjusted to obtain the adjusted steering angle.

[0058] Specifically, the input of this step is the predicted path output from step 3 and the expected steering angle , and from the final road environment feature representation The current road characteristics obtained from .

[0059] Please note that is obtained by predicting the future path, and is the steering angle inferred based on the previous step.

[0060] Furthermore, the goal is to use the road environment information and predicted path through an adaptive judgment mechanism to determine whether the steering angle needs to be adjusted. If the predicted path does not match the road environment, or the steering angle exceeds the safety threshold, the steering strategy needs to be adjusted in time.

[0061] Furthermore, we design a discriminant function , according to the compatibility between the road environment and the path, determine whether to trigger the adjustment of the steering angle. Assume that the current road curvature and road width information have been represented by the final road environment feature If the predicted path And the final road environment feature representation If the difference is too large, it means that the path prediction may be biased and needs to be corrected. The discriminant function is defined as: ; in, : Adaptive discrimination value, reflecting the matching degree between the path and the environment. : The future path obtained from step 3. : The final road environment feature representation obtained through step 2. : Adjustment factor, used to adjust the sensitivity of the discriminant function. The discriminant function determines whether the steering angle needs to be adjusted based on the difference between the path and the road environment. The larger the difference, Lower values ​​indicate that the path is less suitable and requires more adjustment.

[0062] Furthermore, based on the output of the discriminant function , the present invention designs a steering angle adjustment formula. If the discrimination value is low, it means that the path does not match the actual road characteristics, and the steering angle needs to be corrected by adjusting the factor. The adaptive adjustment formula is: ; in, : The adjusted steering angle as the final output. : The original target steering angle obtained from step 3. : Adaptive adjustment factor, dynamically adjusted based on the value of the discriminant function, and discriminant value Inversely proportional relationship. : The amount of fine-tuning of the steering angle, reflecting the difference between the predicted path and road characteristics. Through this adaptive adjustment formula, the system can adjust the steering angle in real time to adapt to the changing road environment and path prediction, thereby improving steering accuracy and driving safety.

[0063] Furthermore, the final output is the adjusted steering angle , as the input of the vehicle control system, is used to adjust the actual driving direction of the vehicle. At the same time, the discriminant function value is output , which is used to evaluate the matching between the current path and the road environment, and serves as feedback information to help the subsequent path optimization and adjustment of the environmental perception strategy.

[0064] Through this adaptive judgment mechanism, the system can make more intelligent steering decisions in complex or uncertain road environments, ensuring that the vehicle's path and driving direction always maintain high accuracy and high safety in a changing environment.

[0065] S5. Receive the adjusted steering angle, design a steering state discrimination function to determine whether the steering angle matches the current road conditions and vehicle driving status. If the steering angle does not match the actual situation, it may cause trajectory deviation or vehicle instability. Appropriate feedback adjustment is required to output the final corrected steering angle, and the final corrected steering angle is transmitted to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

[0066] Specifically, the goal of this step is to determine whether the steering angle matches the current road conditions and vehicle driving status. If the steering angle does not match the actual situation, it may cause trajectory deviation or vehicle instability, and appropriate feedback adjustment is required.

[0067] Furthermore, the present invention designs a steering state discrimination function , which comprehensively considers the current steering angle, estimated path and road characteristics to evaluate the rationality of the steering angle. If the discriminant value is low, it indicates that there is a deviation in the steering angle and feedback correction is required. The design can be quantified by: ; in, : Adjusted steering angle, from step 4. : Target steering angle, obtained from path prediction and environment analysis. : The maximum steering angle allowed by the system, used to normalize the steering deviation. : Weighting factor, which adjusts the tolerance for the deviation between the target angle and the actual angle.

[0068] Through the discriminant function, the present invention can obtain a quantitative value , indicating the rationality of the current steering state. If it is lower, it means there is a large steering deviation and the system should activate the feedback mechanism.

[0069] Furthermore, based on the steering state judgment value , design a feedback adjustment mechanism. If Below the set threshold, the system will adjust the steering angle through feedback to ensure that the vehicle can maintain the correct driving path. The steering angle feedback formula can be expressed as follows: ; in, : The corrected steering angle is output as the feedback result. : Feedback gain factor, controls the adjustment strength. : Steering angle fine-tuning amount, indicating the magnitude of correction.

[0070] In the feedback mechanism, when the discriminant value When it is lower, The feedback correction of the control will be through Make fine adjustments to correct the steering angle in the correct direction to avoid deviating from the target path.

[0071] It can be understood that the final output of this step is the corrected steering angle The angle will be transmitted to the vehicle control system for actual operation to ensure that the vehicle follows the optimal path. At the same time, the judgment value is output , providing a basis for subsequent path optimization. Through this steering state discrimination and feedback mechanism, the system can detect the rationality of the current steering angle in real time and make dynamic adjustments when necessary to ensure that the vehicle can travel stably and accurately along the predetermined path.

[0072] like Figure 2 As shown, in another embodiment of the present invention, a vehicle steering state determination system based on vision is provided, and the system includes: A road environment model building unit 1011 is used to obtain multi-dimensional features of the road environment and build a dynamic road environment model, wherein the road environment model outputs road geometric features and dynamic environment states; The road environment analysis unit 1012 is used to extract features of the road geometric features and dynamic environment states by using a multi-scale convolutional neural network, wherein each layer of the multi-scale convolutional neural network extracts features of different scales, and the features output by each layer are weightedly fused to obtain multi-scale visual features, and a spatiotemporal fusion mechanism is introduced into the multi-scale visual features to weightedly fuse visual features from different times and different scales to obtain final fused visual features, and the final road environment feature representation is reconstructed through a fully connected layer according to the final fused visual features; A path prediction model building unit 1013 is used to build a path prediction model according to the final road environment feature representation to obtain a predicted path, and deduce the future steering angle of the vehicle according to the predicted path through the relationship between the current position of the vehicle and the future path, while introducing a smooth constraint term in the derivation process to limit the change range of the steering angle to avoid over-adjustment, and finally obtain the expected steering angle; The vehicle state determination unit 1014 is used to analyze the compatibility between the road environment and the path, and determine whether to trigger the adjustment of the expected steering angle: Design the discriminant function, output the adaptive discriminant value, and make judgments: If the adaptive discrimination value is the lowest, it means that the path is the least suitable and needs the most adjustments; Design a steering angle adjustment formula based on the adaptive discriminant value, adjust the expected steering angle, and obtain the adjusted steering angle; The vehicle state correction unit 1015 is used to receive the adjusted steering angle and design a steering state discrimination function to determine whether the steering angle matches the current road conditions and the vehicle driving state. If the steering angle does not match the actual situation, it may cause trajectory deviation or vehicle instability. Appropriate feedback adjustment is required to output the final corrected steering angle and transmit the final corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

[0073] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0074] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the parameter operation method provided in any embodiment of the present invention, and will not be repeated here.

[0075] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0076] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0078] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for distinguishing vehicle steering state based on vision, characterized in that: The method comprises: Step 1: Acquire multi-dimensional features of the road environment and construct a dynamic road environment model, wherein the road environment model outputs road geometric features and dynamic environment states; Step 2: A multi-scale convolutional neural network is used to extract features of the road geometric features and dynamic environment states. Each layer of the multi-scale convolutional neural network extracts features of different scales, and the features output by each layer are weighted and fused to obtain multi-scale visual features. A spatiotemporal fusion mechanism is introduced into the multi-scale visual features, and visual features from different times and scales are weighted and fused to obtain the final fused visual features. The final fused visual features are reconstructed into the final road environment feature representation through a fully connected layer. Step 3: construct a path prediction model according to the final road environment feature representation to obtain a predicted path, and deduce the future steering angle of the vehicle according to the predicted path through the relationship between the current position of the vehicle and the future path. At the same time, a smooth constraint term is introduced in the derivation process to limit the change range of the steering angle to avoid excessive adjustment, and finally obtain the expected steering angle; Step 4: Analyze the compatibility between the road environment and the path to determine whether to trigger the adjustment of the expected steering angle: Design the discriminant function, output the adaptive discriminant value, and make judgments: If the adaptive discrimination value is the lowest, it means that the path is the least suitable and needs the most adjustments; At the same time, a steering angle adjustment formula is designed based on the adaptive discriminant value, and the expected steering angle is adjusted to obtain the adjusted steering angle; Step 5. Receive the adjusted steering angle and design a steering state discrimination function to determine whether the steering angle matches the current road conditions and vehicle driving status. If the steering angle does not match the actual situation, it may cause trajectory deviation or vehicle instability. Appropriate feedback adjustment is required to output the final corrected steering angle and pass the final corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

2. The method for determining vehicle steering state based on vision according to claim 1, characterized in that: The dynamic road environment model is constructed as follows: A multi-scale convolutional neural network is used to extract the geometric features of the road. The multi-scale convolutional neural network at different layers can extract road features from local to global. Then the spatiotemporal fusion neural network is introduced, so that the network can automatically assign different weights to the features according to the changes in the road environment in different time periods, eliminate the interference of time changes on feature extraction, and obtain the time-to-time information. The road geometry characteristics; Based on time Based on the road geometry characteristics, the Kalman filter is used to dynamically model the road environment state. At the same time, the road state is predicted and updated in combination with the vehicle's motion information. Then, an adaptive adjustment factor is introduced so that the Kalman filter can automatically adjust the road environment model according to the dynamic changes of the road.

3. The method for determining a vehicle turning state based on vision according to claim 1, characterized in that: The multi-scale visual features are processed by a multi-scale convolutional neural network, where each layer of the convolutional neural network extracts features of different scales. The goal is to extract high-precision visual features from complex environments through gradually refined feature maps. Each convolutional layer outputs ,in Indicates different scales, from coarse to fine.

4. The method for determining a vehicle turning state based on vision according to claim 3, characterized in that: A spatiotemporal fusion mechanism is introduced into the multi-scale visual features, and dynamic adjustment is performed through temporal modeling based on an LSTM network to weightedly fuse visual features from different times and scales; Learning moments through LSTM networks The historical features before and after are combined with real-time perception data to dynamically adjust the fusion weights. By introducing regularization terms, the weight changes are ensured to be smoother and will not cause excessive fluctuations due to sudden changes in the environment. Finally, it is reconstructed into the final road environment feature representation through a fully connected layer.

5. The method for determining vehicle steering state based on vision according to claim 4, characterized in that: The path prediction model is a weighted path regression model based on road environment characteristics, which predicts the future path by combining the current behavior and historical path information of the vehicle; The future steering angle of the vehicle is calculated through the predicted path; the future steering angle of the vehicle is derived through the relationship between the current position of the vehicle and the future path.

6. The method for determining vehicle steering state based on vision according to claim 2, characterized in that: The road geometric features include lane lines, obstacles and lane curvature.

7. The method for determining vehicle steering state based on vision according to claim 5, characterized in that: The discriminant function determines whether the steering angle needs to be adjusted based on the difference between the path and the road environment.

8. The method for determining vehicle steering state based on vision according to claim 7, characterized in that: The steering state discrimination function determines the steering state discrimination value based on considering the current steering angle, the estimated path and the road characteristics to evaluate the rationality of the steering angle.

9. The method for determining vehicle steering state based on vision according to claim 8, characterized in that: Design a feedback adjustment mechanism based on the steering state discrimination function: If the steering state judgment value is lower than the set threshold, the steering angle will be adjusted through feedback to ensure that the vehicle can maintain the correct driving path; In the feedback mechanism, when the steering state judgment value is the lowest, the steering angle is dynamically fine-tuned through feedback so that the steering angle is corrected in the correct direction to avoid deviation from the target path.

10. A vehicle steering state identification system based on vision, characterized in that: The system comprises: A road environment model building unit, used to obtain multi-dimensional features of the road environment and build a dynamic road environment model, wherein the road environment model outputs road geometric features and dynamic environment states; A road environment analysis unit, configured to extract features of the road geometric features and dynamic environment states by using a multi-scale convolutional neural network, wherein each layer of the multi-scale convolutional neural network extracts features of different scales, and weightedly fuses the features output by each layer to obtain multi-scale visual features, introduces a spatiotemporal fusion mechanism into the multi-scale visual features, and weightedly fuses visual features from different times and scales to obtain final fused visual features, and reconstructs the final road environment feature representation through a fully connected layer based on the final fused visual features; a path prediction model construction unit, configured to construct a path prediction model according to the final road environment feature representation to obtain a predicted path, and derive the future steering angle of the vehicle according to the predicted path through the relationship between the current position of the vehicle and the future path, while introducing a smooth constraint term in the derivation process to limit the range of change of the steering angle to avoid over-adjustment, and finally obtain the expected steering angle; The vehicle state determination unit is used to analyze the compatibility between the road environment and the path, and determine whether to trigger the adjustment of the expected steering angle: Design the discriminant function, output the adaptive discriminant value, and make judgments: If the adaptive discrimination value is the lowest, it means that the path is the least suitable and needs the most adjustments; At the same time, a steering angle adjustment formula is designed based on the adaptive discriminant value, and the expected steering angle is adjusted to obtain the adjusted steering angle; The vehicle state correction unit is used to receive the adjusted steering angle and design a steering state discrimination function to determine whether the steering angle matches the current road conditions and vehicle driving status. If the steering angle does not match the actual situation, it may cause trajectory deviation or vehicle instability. Appropriate feedback adjustment is required to output the final corrected steering angle and pass the final corrected steering angle to the vehicle control system for actual operation to ensure that the vehicle travels along the optimal path.

Images