Vehicle control method and device, vehicle and storage medium

Abstract

Embodiments of the present application provide a vehicle control method and device, a vehicle and a storage medium, and relate to the technical field of intelligent vehicle control. The method obtains a preview turning angle output by a two-point preview visual model constructed based on a driver's visual behavior mechanism during driving; inputs visual scene information into a neural network model to obtain a predicted turning angle output by the neural network model constructed based on the driver's visual attention mechanism; adopts a weighted fusion rule to perform weighted fusion processing on the preview turning angle and the predicted turning angle to obtain a target steering wheel turning angle; and sends the target steering wheel turning angle to a steering wheel turning angle controller to enable the steering wheel turning angle controller to control the steering wheel to follow the target steering wheel turning angle, thereby considering the influence of the driver's driving style and driving behavior characteristics on the vehicle control effect, improving the rationality and accuracy of vehicle control, and reducing the redundancy and complexity of the system without relying on an upper module.

Description

Technical Field

[0001] This application relates to the field of vehicle intelligent control technology, and in particular, to a vehicle control method, device, vehicle, and storage medium. Background Technology

[0002] Lateral control technology is one of the key technologies for autonomous vehicles. Currently, the lateral control scheme for autonomous driving calculates the positional deviation, heading angle deviation, and other deviation values ​​between the actual trajectory of the vehicle operated by the driver or the actual trajectory of the autonomous vehicle and the reference trajectory planned by the upper planning layer. The deviation values ​​are then transmitted as inputs to the vehicle control system, and feedback control is used to reduce the difference to ensure the vehicle's road tracking performance.

[0003] Current autonomous driving lateral control schemes do not consider the impact of the driver's driving style and behavior characteristics on the vehicle control effect. This results in a large phase lag between the expected steering wheel angle and the actual steering wheel angle, leading to poor autonomous driving following performance in complex conditions. It is unable to adapt to the driving styles and operating behavior characteristics of different drivers, thus reducing the driver's user experience of the autonomous driving system.

[0004] Furthermore, current feedback control technology relies on the planned trajectory output by the upper-level decision-making and planning module and the vehicle positioning information output by the upper-level positioning module. This high degree of dependence on the upper-level modules leads to high system redundancy and complexity. Summary of the Invention

[0005] This application provides a vehicle control method, apparatus, vehicle, and storage medium to improve the above-mentioned problems.

[0006] In a first aspect, embodiments of this application provide a vehicle control method. The method includes: acquiring a pre-aiming steering angle output by a two-point pre-aiming visual model, the two-point pre-aiming visual model being constructed based on the driver's visual behavior mechanism during driving; inputting visual scene information into a neural network model to acquire a predicted steering angle output by the neural network model, the neural network model being constructed based on the driver's visual attention mechanism; performing weighted fusion processing on the pre-aiming steering angle and the predicted steering angle using weighted fusion rules to obtain a target steering wheel angle; and sending the target steering wheel angle to a steering wheel angle controller, so that the steering wheel angle controller controls the steering wheel to follow the target steering wheel angle.

[0007] Secondly, embodiments of this application provide a vehicle control device. The device includes: a pre-aiming steering angle acquisition module, used to acquire a pre-aiming steering angle output by a two-point pre-aiming visual model, the two-point pre-aiming visual model being constructed based on the driver's visual behavior mechanism during driving; a predicted steering angle acquisition module, used to input visual scene information to a neural network model, and acquire a predicted steering angle output by the neural network model, the neural network model being constructed based on the driver's visual attention mechanism; a data weighted fusion module, used to perform weighted fusion processing on the pre-aiming steering angle and the predicted steering angle using weighted fusion rules to obtain a target steering wheel angle; and a data transmission module, used to send the target steering wheel angle to a steering wheel angle controller, so that the steering wheel angle controller controls the steering wheel to follow the target steering wheel angle.

[0008] Thirdly, embodiments of this application provide a vehicle. The vehicle includes a memory, one or more processors, and one or more application programs. The one or more application programs are stored in the memory and configured to cause the one or more processors to execute the methods provided in embodiments of this application when invoked by the processors.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium stores program code configured to cause the processor to execute the method provided in embodiments of this application when invoked by the processor.

[0010] This application provides a vehicle control method, device, vehicle, and storage medium. The method outputs a pre-aiming steering angle based on a two-point pre-aiming visual model constructed from the driver's visual behavior mechanism during driving. A neural network model constructed from the driver's visual attention mechanism outputs a predicted steering angle based on visual scene information. The pre-aiming and predicted steering angles are weighted and fused using a weighted fusion rule to obtain the final target steering wheel angle. This method considers the impact of the driver's driving style and behavior characteristics on vehicle control, improving the rationality and accuracy of vehicle control. Furthermore, by fusing the pre-aiming steering angle output from the two-point pre-aiming visual model and the predicted steering angle output from the neural network model using a weighted fusion rule to determine the target steering wheel angle, the method eliminates reliance on upper-level modules, reducing system redundancy and complexity. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a vehicle lateral control model involved in a vehicle control method provided in an exemplary embodiment of this application;

[0013] Figure 2 This is a schematic diagram of a vehicle kinematic model provided in an exemplary embodiment of this application;

[0014] Figure 3 This is a schematic diagram of a vehicle dynamics model provided in an exemplary embodiment of this application;

[0015] Figure 4 This is a schematic diagram of a two-point preview model in a driver's vision model provided in an exemplary embodiment of this application;

[0016] Figure 5 This is a flowchart illustrating an embodiment of the optimization method for near-pre-aiming distance and far-pre-aiming distance provided in this application;

[0017] Figure 6 This is a schematic diagram of a two-point preview visual model provided in an exemplary embodiment of this application;

[0018] Figure 7 This is a schematic diagram of a neural network model provided in an exemplary embodiment of this application;

[0019] Figure 8-1 This is a schematic diagram of the channel attention module in the visual attention mechanism model provided in an exemplary embodiment of this application;

[0020] Figure 8-2 This is a schematic diagram of the spatial attention module in the visual attention mechanism model provided in an exemplary embodiment of this application;

[0021] Figure 9 This is a flowchart illustrating a method for optimizing weighted fusion rules according to an embodiment of this application;

[0022] Figure 10 This is a schematic flowchart of a vehicle control method provided in an embodiment of this application;

[0023] Figure 11 This is a schematic flowchart of a vehicle control method provided in another embodiment of this application;

[0024] Figure 12 This is a structural block diagram of a vehicle control device provided in an embodiment of this application;

[0025] Figure 13 This is a structural block diagram of a vehicle provided in one embodiment of this application;

[0026] Figure 14 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0028] First, it's important to clarify that the visual behavior mechanism refers to how drivers combine the curvature of the expected path from both near and far perspectives to correct the steering wheel angle, ensuring the vehicle stays on the desired path. The visual attention mechanism refers to how drivers focus on specific information during driving to assist them. For example, when approaching an intersection, drivers typically pay attention to pedestrians and other vehicles to help them complete the driving task.

[0029] Figure 1 This is a schematic diagram of a vehicle lateral control model involved in a vehicle control method provided in an exemplary embodiment of this application. This vehicle lateral control model can be deployed in a vehicle after training. Figure 1 As shown, the vehicle lateral control model includes a two-point pre-aiming visual model based on a visual behavior mechanism, a neural network model based on a visual attention mechanism, and a weighted fusion module that weights and fuses the pre-aiming turning angle output by the two-point pre-aiming visual model and the predicted turning angle output by the neural network model. The two-point pre-aiming visual model includes a two-point pre-aiming model, a visual prediction module, a visual compensation module, and a signal transmission delay module. The specific construction and training methods for the model will be described in detail in subsequent embodiments.

[0030] During model training, the loss value can be calculated based on the target steering wheel angle and the actual steering wheel angle output by the weighted fusion module, and the loss value can be fed back to the weighted fusion module so that the weighted fusion module can optimize the weighted fusion rules, specifically by optimizing the pre-aiming angle weight and the predicted angle weight in the weighted fusion rules.

[0031] In addition, during model training, the loss value can be calculated based on the predicted steering angle and the actual steering wheel angle output by the neural network model, and this loss value can be fed back into the neural network model to optimize the network weights of the neural network model.

[0032] based on Figure 1 The vehicle lateral control model shown in this application provides a vehicle control method that can output a pre-aiming angle based on a two-point pre-aiming visual model with a visual behavior mechanism, and output a predicted angle based on a neural network model with a visual attention mechanism according to visual scene information. The pre-aiming angle output by the two-point pre-aiming visual model and the predicted angle output by the neural network model are weighted and fused by a weighted fusion module using weighted fusion rules to finally obtain the target steering wheel angle. The steering wheel controller controls the steering wheel angle according to the target steering wheel angle.

[0033] The vehicle control method provided in this application will now be described in detail in the following order: constructing a vehicle kinematics model, constructing a vehicle dynamics model, constructing a two-point preview model, constructing a two-point preview visual model including the two-point preview model, constructing a neural network model based on a visual attention mechanism, and constructing a neural network model according to weighted fusion rules. Figure 1 The vehicle lateral control model shown is a vehicle control method implemented based on this vehicle lateral control model.

[0034] Figure 2 This is a schematic diagram of a vehicle kinematics model provided in an exemplary embodiment of this application. Figure 2 This embodiment of the application constructs a kinematic model representing the positional relationship between the actual vehicle trajectory and the ideal road trajectory, ignoring the influence of vehicle suspension, tires, etc. In the inertial coordinate system XOY, Figure 2 (X) r Y r (X) represents the position coordinates of the vehicle's rear axle center in the inertial coordinate system; f Y f V represents the coordinate position of the vehicle's front axle center in the inertial coordinate system. r V is the vehicle speed at the center of the rear axle. f Φ is the vehicle speed at the center of the front axle; ω is the steering wheel angle; Φ is the vehicle heading angle; ω is the vehicle yaw rate.

[0035] The vehicle running gear model constructed with the rear axle center as the reference point is shown below:

[0036]

[0037] Among them, (X) r Y r ) represent the position coordinates of the vehicle's rear axle center in the inertial coordinate system; Φ is the vehicle's heading angle; ω is the vehicle's yaw rate; V r The speed is the center speed of the vehicle's rear axle.

[0038] Figure 3 This is a schematic diagram of a vehicle dynamics model provided in an exemplary embodiment of this application. This embodiment ignores the effects of the vehicle steering system and suspension system, assumes that the vehicle's longitudinal velocity remains constant, and the vehicle's lateral acceleration does not exceed 0.4g, and constructs a model as follows: Figure 3 The vehicle dynamics model shown is in the XOY coordinate system. Figure 3 V in the figure represents the vehicle speed; V x V is the longitudinal speed of the vehicle. y α is the vehicle's lateral velocity; β is the sideslip angle; ω is the vehicle's yaw rate; α r It is the rear wheel slip angle; F yrIt is the lateral force of the rear wheel; V r It is the rear wheel speed; L r L is the distance from the vehicle's center of gravity to the rear axle. f L is the distance from the vehicle's center of gravity to the front axle; F is the distance between the front and rear axles; yf It is the lateral force of the front wheel; α f It is the front wheel slip angle V f It is the front wheel speed; δ f This refers to the steering angle of the front wheels.

[0039] based on Figure 3 The vehicle dynamics model can be constructed as shown below:

[0040]

[0041] Among them, V x and V y These represent the vehicle's longitudinal and lateral velocities, respectively; ω represents the vehicle's yaw rate; M represents the vehicle's mass; L... f and L r These are the distances from the vehicle's center of gravity to the centers of the front and rear axles, respectively; C f and C r These are the lateral stiffnesses of the front and rear wheels, respectively; I z Let δ be the moment of inertia of the vehicle about the z-axis; f This refers to the steering angle of the front wheels.

[0042] Figure 4 This is a schematic diagram of a two-point preview model in a driver's visual model provided in an exemplary embodiment of this application. Figure 4 Ψ in n The heading angle deviation near the pre-aiming point; Ψ f Y is the heading angle deviation of the far aiming point; Φ is the vehicle heading angle; n D represents the lateral position deviation near the pre-aiming point. n D is the distance from the vehicle's center of gravity to the near-preview point, i.e., the near-preview distance; f R is the distance from the vehicle's center of gravity to the far aiming point, i.e., the far aiming distance; R is the vehicle's turning radius.

[0043] based on Figure 4 The two-point pre-aiming geometric relationship involved is used to construct a two-point pre-aiming model as shown in the following expression:

[0044]

[0045] Among them, Ψ n The heading angle deviation near the pre-aiming point; Ψ f For the heading angle deviation of the far aiming point; Y n D represents the lateral position deviation near the pre-aiming point. nD is the distance from the vehicle's center of gravity to the near-preview point, i.e., the near-preview distance; f ρ is the distance from the vehicle's center of gravity to the far aiming point, i.e., the far aiming distance; Φ is the vehicle's heading angle; R is the vehicle's turning radius; ρ r The curvature of the road.

[0046] Among them, the road curvature ρ r The vehicle's turning radius R and heading angle Φ can be obtained from the autonomous vehicle's perception and positioning system. The near-target distance and the lateral position deviation at the near-target point are determined by the near-target point, while the far-target distance is determined by the far-target point.

[0047] The current two-point aiming method sets fixed near and far aiming distances for all drivers. It does not take into account the fact that different drivers have different aiming distances in actual driving, and that the aiming distance is constantly changing. This leads to a certain deviation between the aiming point and the driver's actual aiming point, resulting in inaccurate aiming results.

[0048] This application embodiment takes into account the above-mentioned situations. During the training process of the two-point pre-aiming model, the near pre-aiming distance and the far pre-aiming distance are optimized and trained. The pre-aiming angle is determined based on the optimized near pre-aiming distance and the far pre-aiming distance. By reducing the deviation between the pre-aiming point and the driver's actual pre-aiming point, the accuracy of the pre-aiming result is improved, and the anthropomorphism of the two-point pre-aiming model is also improved. For the specific optimization process, please refer to [link / reference]. Figure 5 .

[0049] Figure 5 This is a flowchart illustrating an embodiment of the method for optimizing near and far pre-aiming distances provided in this application. The method for optimizing near and far pre-aiming distances provided in this application includes the following steps S110-S150.

[0050] Step S110: During the training of the two-point pre-aiming visual model, acquire the near pre-aiming distance interval, the far pre-aiming distance interval, and the training dataset, which includes the actual steering wheel angle.

[0051] Among them, the near-aiming distance range and the far-aiming distance range can be distance ranges bound to the driver's account to ensure that the near-aiming distance and the far-aiming distance obtained after optimization training based on the near-aiming distance range and the far-aiming distance range conform to the driver's driving behavior characteristics.

[0052] The near-aiming distance range and the far-aiming distance range can be calibrated or obtained from the driver's historical driving records.

[0053] Step S120: Determine the near aiming distance from the near aiming distance range and the far aiming distance from the far aiming distance range. The near aiming distance is the distance from the vehicle's center of gravity to the near aiming point, and the far aiming distance is the distance from the vehicle's center of gravity to the far aiming point.

[0054] The near and far aiming distances here are the initial values ​​for optimized training and can be arbitrarily selected from the near and far aiming distance intervals. For example, the maximum, minimum, average, or any distance within the near aiming distance interval can be selected as the near aiming distance. Similarly, the maximum, minimum, average, or any distance within the far aiming distance interval can be selected as the far aiming distance.

[0055] Step S130: Input the near-pre-aiming distance and the far-pre-aiming distance into the two-point pre-aiming visual model to obtain the pre-aiming angle training value.

[0056] The near and far pre-aiming distances can be input into the two-point pre-aiming model within the two-point pre-aiming vision model. The two-point pre-aiming model determines the heading angle deviations at the near and far pre-aiming points based on these distances and inputs these deviations into the lower-level vision system. The lower-level vision system then performs data fusion processing on the heading angle deviations at the near and far pre-aiming points to obtain the pre-aiming angle training values.

[0057] Specifically, the lower-level vision system can determine a first angle A1 based on the heading angle deviation of the near pre-aiming point, and a second angle A2 based on the heading angle deviation of the far pre-aiming point. Weight values ​​B2 are set for the first angle B1 and the second angle, respectively. Based on these weight values, the first angle A1 and the second angle A2 are weighted and fused to obtain the pre-aiming angle training value A1*B1+A2*B2.

[0058] Step S140: Calculate the loss value based on the pre-aiming angle training value and the corresponding actual steering wheel angle.

[0059] The training values ​​for the pre-aiming angle can be compared with the corresponding actual steering wheel angles in the training dataset to calculate the loss value. For example, the squared error or mean squared error between the two can be calculated as the loss value, which serves as a standard to measure the accuracy and reasonableness of the near and far pre-aiming distances. Generally, the smaller the loss value, the closer the near and far pre-aiming distances are to the driver's actual near and far pre-aiming distances, and the more accurate and reasonable the near and far pre-aiming distances are.

[0060] Step S150: Based on the near pre-aiming distance interval, the far pre-aiming distance interval, and the loss value, optimize and train the near pre-aiming distance and the far pre-aiming distance in the two-point pre-aiming visual model.

[0061] A pre-defined optimization algorithm can be used, with the loss value as the optimization metric. Based on the near-pre-aiming distance interval and the far-pre-aiming distance interval, the near-pre-aiming distance and the far-pre-aiming distance are optimized and trained. The near-pre-aiming distance and the far-pre-aiming distance corresponding to the minimum loss value are determined as the optimal near-pre-aiming distance and the optimal far-pre-aiming distance. The pre-defined optimization algorithm includes, but is not limited to, traversal algorithms and gradient descent methods.

[0062] After optimizing the near and far aiming distances during training, the optimal near and far aiming distances are bound to the driver's account and input into the two-point aiming model to improve the accuracy and human-likeness of the two-point aiming model.

[0063] As mentioned earlier, by inputting road curvature, vehicle heading angle, near-aiming distance, positional deviation at the near-aiming point, and far-aiming distance into the two-point pre-aiming model, the heading angle deviations at the near and far-aiming points output by the two-point pre-aiming model can be obtained. Inputting these heading angle deviations into the lower-level vision system can further construct a two-point pre-aiming vision model; please refer to [link to relevant documentation] for details. Figure 6 .

[0064] Figure 6 This is a schematic diagram of a two-point preview visual model provided in an exemplary embodiment of this application. The two-point preview visual model includes the aforementioned two-point preview model and a lower-level visual system. The lower-level visual system includes a visual prediction module, a visual compensation module, and a signal transmission delay module.

[0065] like Figure 6 As shown, the visual prediction module is used to perform gain processing on the heading angle deviation at the far aiming point. Specifically, the gain coefficient in the visual prediction module can be designed as G. vp =K vp Among them, the visual prediction scaling gain factor K vp Calibration can be performed. The product of the gain system and the heading angle deviation at the far aiming point can be calculated as the first heading angle deviation, which is then used to gain the heading angle deviation at the far aiming point.

[0066] The visual compensation module is used to compensate for heading angle deviations near the pre-aiming point. Specifically, the visual compensation module is shown below:

[0067]

[0068] Among them, G vp K is the visual compensation coefficient. vc This is the visual compensation proportional gain factor, which can be calibrated; V x This refers to the vehicle speed. This is a commonly used formula in the compensation module and can be called directly, so it will not be described in detail here. It can calculate the product of the compensation coefficient and the heading angle deviation at the near aiming point, and use it as the second heading angle deviation to compensate for the heading angle deviation at the near aiming point.

[0069] The two-point pre-aiming visual model can determine the initial pre-aiming angle based on the first and second heading angle deviations. As mentioned earlier, the first angle can be determined based on the first heading angle deviation, and the second angle can be determined based on the second heading angle deviation. By setting weight values ​​for the first angle B1 and the second angle respectively, and by weightedly fusing the first angle and the second angle based on their respective weight values, the initial pre-aiming angle can be obtained.

[0070] The signal transmission delay module is used to handle delays caused by processes such as visual perception, reaction processing, and information transmission. A signal transmission delay module can be configured as follows:

[0071]

[0072] in, T is the signal transmission delay factor; p This is the transmission delay time.

[0073] The product of the signal transmission delay factor and the initial aiming angle can be calculated as the aiming angle. By aligning the pre-aiming angle with time, the real-time performance and accuracy of vehicle control methods can be improved.

[0074] Figure 7 This is a schematic diagram of a neural network model provided in an exemplary embodiment of this application. The neural network model in this embodiment includes a visual attention mechanism model, a long short-term memory network layer, a fully connected layer, and a backpropagation training model. The visual attention mechanism model and the backpropagation training model form an attention model.

[0075] The visual attention mechanism model extracts features from the input visual scene information, forming a refined feature map. The visual scene information refers to the image information captured by the camera. A Long Short-Term Memory (LSTM) network layer further identifies the extracted features and outputs the recognition results. A fully connected layer fuses the recognition results from all LSM network outputs to obtain a new visual scene feature map. Based on the correspondence between the visual scene feature map and the predicted turning angle, the neural network model determines the predicted turning angle corresponding to the new visual scene feature map.

[0076] The neural network model calculates the error (e.g., squared error or root mean square error) between the predicted steering angle and the corresponding actual steering wheel angle, using this error as a loss value. This loss value is then fed back into the backpropagation training model. The backpropagation training model updates the network weights of the neural network model based on the loss value. The network weights include the weights of all networks involved in the visual attention mechanism model, the long short-term memory network layer, and the fully connected layers.

[0077] The visual attention mechanism model is built upon the driver's visual attention mechanism and is used for high-precision recognition of input visual scene information. For details, please refer to [link / reference needed]. Figure 8-1 and Figure 8-2 . Figure 8-1 This is a schematic diagram of the channel attention module in the visual attention mechanism model provided in an exemplary embodiment of this application. Figure 8-2 This is a schematic diagram of the spatial attention module in the visual attention mechanism model provided in an exemplary embodiment of this application. The visual attention mechanism model includes a channel attention module and a spatial attention module. The channel attention module refines and extracts features from the visual scene information along the channel feature axis to obtain a channel feature map. The spatial attention module extracts features from the channel feature map along the spatial feature axis to form a refined feature map. The visual attention model refines the features of visual scene information from both channel and spatial dimensions, which can identify important features in the visual scene information (e.g., traffic signs, lane lines) and improve the expression and learning ability of regions of interest and important feature regions in the visual scene.

[0078] like Figure 8-1 As shown, the intermediate input feature map of the input visual scene information is fed into the channel attention module. Channel features are extracted from important features of the visual scene information (e.g., traffic signs, lane lines, pedestrians) through average pooling and max pooling layers, respectively. The extracted channel features are then input into a shared multilayer perceptron neural network layer for feature map derivation, resulting in the channel feature map. For example... Figure 8-2 As shown, the channel feature map is input into the spatial attention module, and spatial features are extracted sequentially through the average pooling layer and the max pooling layer. The extracted spatial feature map is then input into the convolutional layer for feature integration to form a spatial feature map. The spatial feature map is then output into the long short-term memory network for further processing.

[0079] It should be noted that those skilled in the art should be familiar with the aforementioned long short-term memory network layers, fully connected layers, average pooling layers, max pooling layers, and convolutional layers, and therefore these are not described in detail here.

[0080] After constructing and training a two-point pre-aiming visual model and a neural network model, a weighted fusion rule can be set to weight and fuse the pre-aiming angle output by the two-point pre-aiming visual model and the predicted angle output by the neural network model to determine the target steering wheel angle. Specifically, pre-aiming angle weights and predicted angle weights can be set, and the pre-aiming angle and predicted angle can be weighted and fused based on these weights. The weighted fusion expression is as follows:

[0081]

[0082] Where, δ req δ is the target steering wheel angle. pre K1 represents the predicted turning angle output by the neural network model based on the visual attention mechanism; K1 is the weight of the predicted turning angle. K1 represents the pre-aiming angle output by the two-point pre-aiming visual model based on visual behavior mechanisms; K2 represents the pre-aiming angle weight.

[0083] Weighted fusion rules can be designed based on the vehicle's environmental conditions and the status of its sensors. Specifically, the vehicle's sensor status can be divided into multiple levels, and a correspondence can be established between each level and the pre-aiming turning angle weight and the predicted turning angle weight. This allows for the dynamic determination of the corresponding predicted turning angle weight and the predicted turning angle weight based on the current vehicle sensor status. Similarly, environmental conditions can be divided into multiple levels, and a correspondence can be established between each level and the pre-aiming turning angle weight interval and the predicted turning angle weight interval. This allows for the determination of the corresponding pre-aiming turning angle weight interval and the predicted turning angle weight based on the current environmental conditions. The pre-aiming turning angle weight is determined from the pre-aiming turning angle weight interval, and the predicted turning angle weight is determined from the predicted turning angle weight interval. Environmental conditions include, but are not limited to, weather conditions, road surface conditions, road conditions, and lighting conditions.

[0084] To improve the accuracy of weighted fusion results and comprehensively address visual detection scenarios that vehicles may detect, such as vehicle sensor status, weather conditions, road surface conditions, road conditions, and lighting conditions (which may include one or more combinations of these), weighted fusion rules can be designed to include: a correspondence between the level of vehicle sensor status and the weights of the pre-aiming and predicted turning angles; a correspondence between the level of weather conditions and the weight ranges of the pre-aiming and predicted turning angles; a correspondence between the level of road surface conditions and the weight ranges of the pre-aiming and predicted turning angles; a correspondence between the level of road conditions and the weight ranges of the pre-aiming and predicted turning angles; and a correspondence between the level of lighting conditions and the weight ranges of the pre-aiming and predicted turning angles, thus addressing all visual detection scenarios that vehicles may detect.

[0085] As an example, taking the vehicle's sensor as a camera sensor, the correspondence between the vehicle sensor's state level and the pre-aiming angle weight and the predicted angle weight can be shown in Table 1. Since the predicted angle is predicted based on the visual scene information detected by the camera, the worse the camera sensor's state, the lower the accuracy of the predicted angle based on the visual scene information. As shown in Table 1, the predicted angle weight value can be set to gradually decrease as the camera sensor's state level increases.

[0086] Table 1

[0087]

[0088]

[0089] As shown in Table 1, if the current camera is detected to be in normal condition and can acquire a clear image, the current camera sensor status level can be determined as Level 1, with a predicted corner weight of 0.5 and a preview corner weight of 0.5. If a camera malfunction or failure is detected, the current camera sensor status level can be determined as Level 5, with a predicted corner weight of 0 and a preview corner weight of 1.

[0090] If the current camera is detected to be obstructed, the severity of the obstruction can be further analyzed, and the level of the current camera status can be determined based on the severity of the obstruction. Specifically, the severity of the obstruction can be determined based on the obstructed area of ​​the image captured by the camera.

[0091] For example, when rain obstructs the view, if the obstructed area of ​​the image captured by the camera is less than or equal to 25%, it does not affect the recognition of visual scene information in the image. Therefore, the camera's obstruction level is determined to be relatively minor, and the current camera sensor state is classified as Level 2, with a prediction angle weight of 0.4 and a pre-aiming angle weight of 0.6. When dust obstructs the view, if the obstructed area of ​​the image captured by the camera is greater than 25% but less than 55%, it may affect the recognition of visual scene information in the image. Therefore, the camera's obstruction level is determined to be relatively severe, and the current camera sensor state is classified as Level 3, with a prediction angle weight of 0.25 and a pre-aiming angle weight of 0.75. When mud obstructs the view, if the obstructed area of ​​the image captured by the camera is greater than or equal to 55%, it may be difficult to recognize visual scene information in the image. Therefore, the camera sensor state level is determined to be Level 4, with a prediction angle weight of 0.1 and a pre-aiming angle weight of 0.9.

[0092] As an example, the correspondence between the weather condition level and the pre-aiming angle weight interval and the predicted angle weight interval can be shown in Table 2.

[0093] Table 2

[0094]

[0095] As shown in Table 2, if the current weather conditions are detected as good, such as sunny or cloudy, the current weather condition level can be determined as Level 1, with a predicted turning angle weight range of 0.5-0.6 and a pre-aiming turning angle weight range of 0.4-0.5. If the current weather conditions are detected as poor, such as damp, foggy, dusty, smoky, or light rain, the current weather condition level can be determined as Level 2, with a predicted turning angle weight range of 0.3-0.4 and a pre-aiming turning angle weight range of 0.6-0.7. If the current weather conditions are detected as very poor, such as dense fog or heavy rain, the current weather condition level can be determined as Level 3, with a predicted turning angle weight range of 0.1-0.2 and a pre-aiming turning angle weight range of 0.8-0.9.

[0096] As an example, the correspondence between the road surface condition level and the pre-aiming angle weight interval and the predicted angle weight interval can be shown in Table 3.

[0097] Table 3

[0098]

[0099] As shown in Table 3, if the current road surface condition is detected as normal, for example, a dry and smooth road surface, the current road surface condition level can be determined as Level 1, with a predicted turning angle weight range of 0.5-0.6 and a pre-aimed turning angle weight range of 0.4-0.5. If the current road surface condition is detected as poor, for example, covered by rain or flooded, the current road surface condition level can be determined as Level 2, with a predicted turning angle weight range of 0.35-0.45 and a pre-aimed turning angle weight range of 0.55-0.65. If the current road surface condition is detected as very poor, for example, a muddy, potholed, or snow-covered road surface, the current road surface condition level can be determined as Level 3, with a predicted turning angle weight range of 0.1-0.35 and a pre-aimed turning angle weight range of 0.65-0.9.

[0100] As an example, the correspondence between the road condition level and the pre-aiming angle weight interval and the predicted angle weight interval can be shown in Table 4.

[0101] Table 4

[0102]

[0103] As shown in Table 4, if the lane lines or markings of the current road condition are clear, the current road condition level is determined to be Level 1, with a predicted turning angle weight range of 0.5-0.6 and a preview turning angle weight range of 0.4-0.5. If the lane lines or markings of the current road condition are blurred or partially missing, the current road condition level is determined to be Level 2, with a predicted turning angle weight range of 0.4-0.5 and a preview turning angle weight range of 0.5-0.6. If the lane lines or markings of the current road condition are missing, the current road condition level is determined to be Level 3, with a predicted turning angle weight range of 0.2-0.4 and a preview turning angle weight range of 0.6-0.8.

[0104] As an example, the correspondence between the lighting condition level and the pre-aiming angle weight interval and the predicted angle weight interval can be shown in Table 5.

[0105] Table 5

[0106]

[0107] As shown in Table 5, if the current lighting condition is detected as good, for example, daytime light intensity greater than 1750 lux, then the current lighting condition level is determined to be Level 1, with a predicted corner weight range of 0.5-0.55 and a pre-aiming corner weight range of 0.45-0.5. If the current lighting condition is detected as dim, for example, cloudy or evening, then the current lighting condition level is determined to be Level 2, with a predicted corner weight range of 0.4-0.5 and a pre-aiming corner weight range of 0.5-0.6. If the current lighting condition is detected as strong, for example, strong light, backlight, or reflection, then the current lighting condition level is determined to be Level 3, with a predicted corner weight range of 0.3-0.4 and a pre-aiming corner weight range of 0.6-0.7. If the current lighting condition is detected as weak, such as at night under streetlights, the current lighting condition level is determined to be 4, with a predicted turning angle weight range of 0.2-0.4 and a pre-aimed turning angle weight range of 0.6-0.8. If the current lighting condition is detected as insufficient, such as in a tunnel or at night without streetlights, the current lighting condition level is determined to be 5, with a predicted turning angle weight range of 0.1-0.2 and a pre-aimed turning angle weight range of 0.8-0.9.

[0108] It should be noted that the pre-aiming angle weight can be determined from the pre-aiming angle weight range, while the predicted angle weight can be determined from the predicted angle weight range by first determining the predicted angle weight based on the specific environmental conditions, and then determining the pre-aiming angle weight based on the determined predicted angle weight. Alternatively, the average value can be determined from both the pre-aiming angle weight range and the predicted angle weight range.

[0109] In addition, weights can be set for the aforementioned vehicle sensor status, weather conditions, road conditions, road conditions, and lighting conditions in the weighted fusion rules. This allows the vehicle to perform weighted fusion of the pre-aiming angle weight and the predicted angle weight based on the weights of the at least two conditions when it detects them, thereby determining the final pre-aiming angle weight and the predicted angle weight.

[0110] Figure 9 This is a flowchart illustrating a method for optimizing weighted fusion rules according to an embodiment of this application. The method for optimizing weighted fusion rules includes the following steps S210-S240.

[0111] Step S210: During model training, a training dataset is obtained. The training dataset includes a pre-aiming angle training set, a predicted angle training set, and an actual steering wheel angle dataset. The pre-aiming angles in the pre-aiming angle training set, the predicted angles in the predicted angle training set, and the actual steering wheel angles in the actual steering wheel angle dataset have a corresponding relationship.

[0112] The pre-aiming angle training set can be the set of pre-aiming angles output by the two-point pre-aiming visual models during training. The predicted angle training set can be the set of predicted angles output by the neural network models during training.

[0113] It should be noted that the two-point pre-aiming visual model, the neural network model, and the weighted fusion rule can be trained simultaneously, or the two-point pre-aiming visual model, the neural network model, and the weighted fusion rule can be trained separately, and then the trained models can be integrated together.

[0114] Step S220: The pre-aiming angle in the pre-aiming angle training set and the predicted angle in the predicted angle training set are weighted and fused using a preliminary weighted fusion rule to obtain the steering wheel angle training value.

[0115] The preliminary weighted fusion rule is a pre-set weighted fusion rule. Its specific content is similar to the weighted fusion rule after optimization and training. The only difference is that the pre-aiming corner weight, predicted corner weight or pre-aiming corner weight range and predicted corner weight range in the preliminary weighted fusion rule may be different from those in the weighted fusion rule after optimization and training.

[0116] For a detailed description of step S220, please refer to the aforementioned section on weighted fusion rule design, which will not be repeated here.

[0117] Step S230: Calculate the loss value based on the training value of the steering wheel angle and the corresponding actual steering wheel angle.

[0118] The training values ​​for steering wheel angles can be compared with the corresponding actual steering wheel angles to calculate the loss value. For example, the squared error or mean squared error between the two can be calculated as the loss value, which can then be used as a standard to measure the reasonableness of the weights. Generally, the smaller the loss value, the more reasonable the pre-aiming angle weight, the predicted steering angle weight, or the pre-aiming angle weight range and the predicted steering angle weight range are.

[0119] Step S240: Based on the training dataset and the loss value, the preliminary weighted fusion rule is optimized and trained to obtain the weighted fusion rule.

[0120] A pre-defined optimization algorithm can be used, with the loss value as the optimization metric. Based on the training dataset, the initial weighted fusion rules can be optimized and trained. Specifically, the pre-aiming corner weights, predicted corner weights, or the pre-aiming corner weight intervals and predicted corner weight intervals in the initial weighted fusion rules can be optimized and trained to obtain the final weighted fusion rules. The pre-defined optimization algorithm includes, but is not limited to, least squares and gradient descent. The optimal and most reasonable correspondence among the optimized weighted fusion rules is the one described in Tables 1-5. Applying the optimized weighted fusion rules to the pre-aiming corners and predicted corners can improve the accuracy and reasonableness of the weighted fusion results.

[0121] Figure 10 This is a schematic flowchart of a vehicle control method provided in an embodiment of this application. The vehicle control method can be applied to a vehicle. The vehicle control method may include the following steps S310-S340.

[0122] Step S310: Obtain the pre-aiming angle output by the two-point pre-aiming visual model. The two-point pre-aiming visual model is constructed based on the driver's visual behavior mechanism during driving.

[0123] Among them, the two-point pre-aiming visual model is a model that has been optimized and trained for near and far pre-aiming distances. Compared with the current two-point pre-aiming model, the output pre-aiming angle is more consistent with the driver's actual pre-aiming angle, which can improve the accuracy and rationality of the pre-aiming angle.

[0124] When the driver starts the vehicle, the driver's account is identified, a two-point pre-aiming visual model corresponding to the driver's account is obtained, and the pre-aiming angle output by the two-point pre-aiming visual model is obtained.

[0125] Step S320: Input visual scene information into the neural network model and obtain the predicted turning angle output by the neural network model. The neural network model is constructed based on the driver's visual attention mechanism.

[0126] As mentioned earlier, visual scene information can be image information captured by a camera. During vehicle operation, the onboard camera acquires visual scene information and uploads it to a neural network model, thereby obtaining the predicted turning angle output by the neural network model.

[0127] Step S330: Using a weighted fusion rule, the pre-aiming angle and the predicted angle are weighted and fused to obtain the target steering wheel angle.

[0128] The weighted fusion rules here are the trained and optimized weighted fusion rules. The weighted fusion rules include the correspondence between the level of the vehicle sensor status and the pre-aiming angle weight and the predicted angle weight, the correspondence between the level of weather conditions and the pre-aiming angle weight interval and the predicted angle weight interval, the correspondence between the level of road conditions and the pre-aiming angle weight interval and the predicted angle weight interval, the correspondence between the level of road conditions and the pre-aiming angle weight interval and the predicted angle weight interval, and the correspondence between the level of lighting conditions and the pre-aiming angle weight interval and the predicted angle weight interval, in order to cope with all visual detection situations that vehicles may detect.

[0129] The system can obtain the current visual detection status; based on the correspondence between the visual detection status and the pre-aiming angle weight and the predicted angle weight in the weighted fusion rules, it can determine the target pre-aiming angle weight and the target predicted angle weight corresponding to the current visual detection status; based on the target pre-aiming angle weight and the target predicted angle weight, it can perform weighted fusion processing on the pre-aiming angle and the predicted angle to obtain the target steering wheel angle.

[0130] As an example, the current visual detection situation includes the current vehicle sensor status, at which point the level of the current vehicle sensor status can be determined; based on the correspondence between the level of the vehicle sensor status and the pre-aiming angle weight and the predicted angle weight in the weighted fusion rules, the target pre-aiming angle weight and the target predicted angle weight corresponding to the level of the current vehicle sensor status are determined.

[0131] As an example, the current visual detection situation includes the current environmental situation, at which point the level of the current environmental situation can be determined; based on the correspondence between the level of the environmental situation and the pre-aiming corner weight interval and the predicted corner weight interval in the weighted fusion rule, the target pre-aiming corner weight interval and the target predicted corner weight interval corresponding to the level of the current environmental situation are determined; the target pre-aiming corner weight is determined from the target pre-aiming corner weight interval; and the target predicted corner weight is determined from the target predicted corner weight interval.

[0132] As an example, the current environmental conditions include the current weather conditions, where the level of the current weather conditions can be determined; based on the correspondence between the weather condition levels and the pre-aiming angle weight interval and the predicted angle weight interval in the weighted fusion rules, the target pre-aiming angle weight interval and the target predicted angle weight interval corresponding to the current weather condition level are determined; the target pre-aiming angle weight is determined from the target pre-aiming angle weight interval; and the target predicted angle weight is determined from the target predicted angle weight interval.

[0133] As an example, the current environmental condition is the current road surface condition. At this time, the level of the current road surface condition can be determined. Based on the correspondence between the road surface condition level and the pre-aiming turning angle weight interval and the predicted turning angle weight interval in the weighted fusion rules, the target pre-aiming turning angle weight interval and the target predicted turning angle weight interval corresponding to the current road surface condition level are determined. The target pre-aiming turning angle weight is determined from the target pre-aiming turning angle weight interval. The target predicted turning angle weight is determined from the target predicted turning angle weight interval.

[0134] As an example, the current environment is the current road condition. At this time, the level of the current road condition can be determined. Based on the correspondence between the road condition level and the pre-aiming turning weight interval and the predicted turning weight interval in the weighted fusion rules, the target pre-aiming turning weight interval and the target predicted turning weight interval corresponding to the current road condition level are determined. The target pre-aiming turning weight is determined from the target pre-aiming turning weight interval. The target predicted turning weight is determined from the target predicted turning weight interval.

[0135] As an example, the current environmental condition is the current lighting condition. At this time, the level of the current lighting condition can be determined. Based on the correspondence between the level of the lighting condition and the pre-aiming corner weight interval and the predicted corner weight interval in the weighted fusion rule, the target pre-aiming corner weight interval and the target predicted corner weight interval corresponding to the level of the current lighting condition are determined. The target pre-aiming corner weight is determined from the target pre-aiming corner weight interval. The target predicted corner weight is determined from the target predicted corner weight interval.

[0136] As an example, the current visual detection situation includes the current vehicle sensor status, current weather conditions, current road surface conditions, current road conditions, and current lighting conditions. Following a method similar to the previous example, the following can be determined: the first pre-aiming angle weight A1 and the first predicted angle weight A2 corresponding to the level of the current vehicle sensor status; the second pre-aiming angle weight B1 and the second predicted angle weight B2 corresponding to the level of the current weather conditions; the third pre-aiming angle weight C1 and the third predicted angle weight C2 corresponding to the level of the current road surface conditions; the fourth pre-aiming angle weight D1 and the fourth predicted angle weight D2 corresponding to the level of the current road conditions; and the fifth pre-aiming angle weight E1 and the fifth predicted angle weight E2 corresponding to the level of the current lighting conditions.

[0137] Furthermore, weights can be assigned to each of the following factors—vehicle sensor status, weather conditions, road surface conditions, road conditions, and lighting conditions—based on their influence on the predicted turning angle: A weight for vehicle sensor status (A), weight for the first predicted turning angle (A1), weight for weather conditions (B), weight for the second predicted turning angle (B1), weight for road surface conditions (C), weight for the third predicted turning angle (C1), weight for road conditions (D), weight for the fourth predicted turning angle (D1), weight for lighting conditions (E), and weight for the fifth predicted turning angle (E1). A weighted fusion is then performed to determine the final predicted turning angle weight as A1*A + B1*B + C1*C + D1*D + E1*E. The weights of the vehicle sensor status (A), the first predicted turning angle (A2), the weather status (B), the second predicted turning angle (B2), the road surface condition (C), the third predicted turning angle (C2), the road condition (D), the fourth predicted turning angle (D2), the lighting condition (E), and the fifth predicted turning angle (E2) are weighted and fused to determine the final predicted turning angle weight as A2*A+B2*B+C2*C+D2*D+E2*E.

[0138] By dynamically adjusting the pre-aiming and predicted steering angle weights based on various visual detection conditions, including the vehicle's sensor status, weather conditions, road surface conditions, and lighting conditions, the influence of different operating conditions and the vehicle's sensor status on the vehicle control effect can be considered, thereby improving the accuracy of the pre-aiming and predicted steering angle weights and enhancing the accuracy of vehicle control.

[0139] Step S340: The target steering wheel angle is sent to the steering wheel angle controller so that the steering wheel angle controller controls the steering wheel to follow the target steering wheel angle.

[0140] The target steering wheel angle is output to the steering wheel angle controller, which controls the steering wheel to begin turning until the angle control amount reaches the target steering wheel angle, thereby achieving lateral control of the vehicle. It should be noted that parts not described in detail in steps S310-S340 can be found in the aforementioned embodiments and will not be repeated here.

[0141] The vehicle control method provided in this application's embodiments outputs a pre-aiming steering angle based on a two-point pre-aiming visual model constructed from the driver's visual behavior mechanism during driving, and outputs a predicted steering angle based on the driver's visual attention mechanism according to visual scene information. The pre-aiming and predicted steering angles are then weighted and fused using a weighted fusion rule to obtain the final target steering wheel angle. This method considers the impact of the driver's driving style and behavior characteristics on vehicle control performance, improving the rationality and accuracy of vehicle control. Furthermore, by fusing the pre-aiming steering angle output from the two-point pre-aiming visual model and the predicted steering angle output from the neural network model using a weighted fusion rule to determine the target steering wheel angle, the method eliminates reliance on upper-level modules, reducing system redundancy and complexity.

[0142] Figure 11 This is a schematic flowchart of a vehicle control method provided in another embodiment of this application. The vehicle control method can be applied to a vehicle and specifically includes the following steps S410-S450.

[0143] Step S410: Input the optimized near-pre-aiming distance and far-pre-aiming distance, the lateral position deviation at the current near-pre-aiming point, the current vehicle heading angle, and the current vehicle turning radius into the two-point pre-aiming model to obtain the heading angle deviation at the near-pre-aiming point and the heading angle deviation at the far-pre-aiming point.

[0144] Among them, the two-point pre-aiming model is based on Figure 4 The model is constructed based on the two-point pre-aiming geometric relationship; please refer to the aforementioned [reference]. Figure 4 The relevant description section. In step S410, using the optimized near-pre-aiming distance and far-pre-aiming distance, which are bound to the driver's account, can improve the accuracy and human-likeness of the two-point pre-aiming model. For details on how to optimize and train the near-pre-aiming distance and far-pre-aiming distance, please refer to the aforementioned... Figure 5 The details and related descriptions will not be repeated here.

[0145] Step S420: Input the heading angle deviation at the near pre-aiming point and the heading angle deviation at the far pre-aiming point to the lower-level vision system to obtain the pre-aiming turning angle.

[0146] As mentioned above, the lower-level vision system includes a vision prediction module, a vision compensation module, and a signal transmission delay module. The specific implementation of step S420 includes the following steps: inputting the heading angle deviation at the far pre-aiming point to the vision prediction module to obtain a first heading angle deviation; inputting the heading angle deviation at the near pre-aiming point to the vision compensation module to obtain a second heading angle deviation; determining the preliminary pre-aiming angle based on the first and second heading angle deviations; and inputting the preliminary pre-aiming angle to the signal transmission delay module to obtain the pre-aiming angle.

[0147] For details on how the lower-level vision system determines the preview angle, please refer to the aforementioned implementation. Figure 6 And its related parts, which will not be elaborated here.

[0148] Step S430: Input visual scene information into the neural network model and obtain the predicted turning angle output by the neural network model.

[0149] As mentioned above, the neural network model includes a visual attention mechanism model, a long short-term memory network layer, and a fully connected layer. The specific implementation of step S430 includes the following steps: inputting visual scene information into the visual attention mechanism model to obtain a first visual scene feature map; inputting the first feature map into the long short-term memory network layer to obtain a second visual scene feature map; inputting the second feature map into the fully connected layer to obtain a third visual scene feature map; and determining the predicted turning angle corresponding to the third visual scene feature map based on the correspondence between the visual scene feature map and the predicted turning angle.

[0150] For details on how the neural network model determines the predicted rotation angle, please refer to the aforementioned implementation. Figure 7 Figure 8 and its related parts will not be described in detail here.

[0151] Step S440: Using a weighted fusion rule, the pre-aiming angle and the predicted angle are weighted and fused to obtain the target steering wheel angle. The target steering wheel angle is then sent to the steering wheel angle controller so that the steering wheel angle controller controls the steering wheel to follow the target steering wheel angle.

[0152] For step S440, please refer to the aforementioned steps S330-S340, which will not be repeated here.

[0153] Step S450: Calculate the loss value based on the predicted steering angle and the actual steering wheel angle, and input the loss value into the backpropagation training model so that the backpropagation training model updates the network weights of the neural network model.

[0154] The neural network model can calculate the error between the predicted steering angle and the actual steering wheel angle (e.g., squared error or root mean square error), and use this error as a loss value. This loss value is then input into the backpropagation training model so that the backpropagation training model updates the network weights of the neural network model.

[0155] For example, when the loss value is large, backpropagation training can reduce the step size of the average pooling layer and / or max pooling layer in the visual attention mechanism model to improve the extraction accuracy of important features, thereby improving prediction accuracy.

[0156] It should be noted that for the parts not described in detail in steps S410-S450, please refer to the relevant parts of the foregoing embodiments, and they will not be repeated here.

[0157] This application integrates the prediction, compensation, and signal transmission delay characteristics of a two-point pre-aiming visual model with the mapping and memory characteristics of a neural network model. It considers the impact of the driver's visual behavior and attention mechanisms on vehicle control performance, achieving good road following performance during autonomous driving systems such as steering obstacle avoidance, lane changing, and lane alignment. The vehicle control method provided in this application does not rely on planning and positioning information from upper-level modules, enabling module decoupling and reducing module complexity, system redundancy, and complexity. This application employs optimized near and far pre-aiming distances to improve the accuracy and rationality of pre-aiming turning angles. This application's embodiments adjust the network weights of the neural network through backpropagation training, dynamically adjusting the network weights to improve the accuracy of predicted turning angles.

[0158] Figure 12 This is a structural block diagram of a vehicle control device provided in an embodiment of this application. The vehicle control device 100 can be deployed in a vehicle. The vehicle control device 100 includes a pre-aiming angle acquisition module 110, a predicted angle acquisition module 120, a data weighted fusion module 130, and a data transmission module 140. The pre-aiming angle acquisition module 110 is used to acquire the pre-aiming angle output by a two-point pre-aiming visual model, which is constructed based on the driver's visual behavior mechanism during driving. The predicted angle acquisition module 120 is used to input visual scene information into a neural network model and acquire the predicted angle output by the neural network model, which is constructed based on the driver's visual attention mechanism. The data weighted fusion module 130 is used to perform weighted fusion processing on the pre-aiming angle and the predicted angle using weighted fusion rules to obtain a target steering wheel angle. The data transmission module 140 is used to send the target steering wheel angle to a steering wheel angle controller, so that the steering wheel angle controller controls the steering wheel to follow the target steering wheel angle.

[0159] In some implementations, the data weighted fusion module 130 is further configured to acquire the current visual detection status; determine the target pre-aiming corner weight and target predicted corner weight corresponding to the current visual detection status based on the correspondence between the visual detection status and the pre-aiming corner weight and the predicted corner weight in the weighted fusion rules; and perform weighted fusion processing on the pre-aiming corner and the predicted corner according to the target pre-aiming corner weight and the target predicted corner weight.

[0160] In some implementations, the data weighted fusion module 130 is further used to determine the level of the current vehicle sensor state; based on the correspondence between the level of the vehicle sensor state and the pre-aiming angle weight and the predicted angle weight in the weighted fusion rules, it determines the target pre-aiming angle weight and the target predicted angle weight corresponding to the level of the current vehicle sensor state.

[0161] In some implementations, the data weighted fusion module 130 is further configured to determine the level of the current environmental condition; based on the correspondence between the level of the environmental condition and the pre-aiming angle weight interval and the predicted angle weight interval in the weighted fusion rules, determine the target pre-aiming angle weight interval and the target predicted angle weight interval corresponding to the level of the current environmental condition; determine the target pre-aiming angle weight from the target pre-aiming angle weight interval; and determine the target predicted angle weight from the target predicted angle weight interval.

[0162] In some embodiments, the vehicle control device 100 further includes a first model training module. The first model training module is used to acquire a training dataset during model training. The training dataset includes a pre-aiming angle training set, a predicted angle training set, and an actual steering wheel angle dataset. The pre-aiming angles in the pre-aiming angle training set, the predicted angles in the predicted angle training set, and the actual steering wheel angles in the actual steering wheel angle dataset have a corresponding relationship. A preliminary weighted fusion rule is used to perform weighted fusion processing on the pre-aiming angles in the pre-aiming angle training set and the predicted angles in the predicted angle training set to obtain steering wheel angle training values. A loss value is calculated based on the steering wheel angle training values ​​and the corresponding actual steering wheel angles. Based on the training dataset and the loss value, the preliminary weighted fusion rule is optimized and trained to obtain the weighted fusion rule.

[0163] In some embodiments, the vehicle control device 100 further includes a second model training module. The second model training module is used to acquire, during the training of the two-point pre-aiming visual model, a near pre-aiming distance interval, a far pre-aiming distance interval, and a training dataset, the training dataset including actual steering wheel angles; determine a near pre-aiming distance from the near pre-aiming distance interval and a far pre-aiming distance from the far pre-aiming distance interval, the near pre-aiming distance being the distance from the vehicle's center of gravity to the near pre-aiming point, and the far pre-aiming distance being the distance from the vehicle's center of gravity to the far pre-aiming point; input the near pre-aiming distance and the far pre-aiming distance into the two-point pre-aiming visual model to obtain pre-aiming angle training values; calculate a loss value based on the pre-aiming angle training values ​​and the corresponding actual steering wheel angles; and optimize the near pre-aiming distance and far pre-aiming distance in the two-point pre-aiming visual model based on the near pre-aiming distance interval, the far pre-aiming distance interval, and the loss value. The near pre-aiming distance and far pre-aiming distance in the two-point pre-aiming visual model are bound to the driver's account.

[0164] In some implementations, the pre-aiming angle acquisition module 110 is also used to input the optimized training near pre-aiming distance and far pre-aiming distance, the lateral position deviation at the current near pre-aiming point, the current vehicle heading angle, and the current vehicle turning radius into the two-point pre-aiming model to obtain the heading angle deviation at the near pre-aiming point and the heading angle deviation at the far pre-aiming point; and input the heading angle deviation at the near pre-aiming point and the heading angle deviation at the far pre-aiming point into the lower-level vision system to obtain the pre-aiming angle.

[0165] In some embodiments, the pre-aiming angle acquisition module 110 is further configured to input the heading angle deviation at the far pre-aiming point to the visual prediction module to obtain a first heading angle deviation; input the heading angle deviation at the near pre-aiming point to the visual compensation module to obtain a second heading angle deviation; determine the preliminary pre-aiming angle based on the first heading angle deviation and the second heading angle deviation; and input the preliminary pre-aiming angle to the signal transmission delay module to obtain the pre-aiming angle.

[0166] In some implementations, the predicted turning angle acquisition module 120 is further configured to input visual scene information into the visual attention mechanism model to obtain a first visual scene feature map; input the first feature map into the long short-term memory network layer to obtain a second visual scene feature map; input the second feature map into the fully connected layer to obtain a third visual scene feature map; and determine the predicted turning angle corresponding to the third visual scene feature map based on the correspondence between the visual scene feature map and the predicted turning angle.

[0167] In some implementations, the predicted steering angle acquisition module 120 is further configured to calculate a loss value based on the predicted steering angle and the actual steering wheel angle; and input the loss value into the backpropagation training model so that the backpropagation training model updates the network weights of the neural network model.

[0168] Those skilled in the art will clearly understand that the vehicle control device 100 provided in the embodiments of this application can implement the vehicle control method provided in the embodiments of this application. The specific working process of the above-mentioned device and modules can be found in the process corresponding to the vehicle control method in the embodiments of this application, and will not be repeated here.

[0169] In the embodiments provided in this application, the coupling, direct coupling, or communication connection between the modules shown or discussed may be indirect coupling or communication coupling through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms. The embodiments of this application do not limit this.

[0170] Furthermore, the functional modules in the embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules, and this application embodiment does not impose any restrictions on this.

[0171] Figure 13 This is a structural block diagram of a vehicle provided in an embodiment of this application. The vehicle 200 may include one or more of the following components: a memory 210, one or more processors 220, and one or more application programs, wherein the one or more application programs may be stored in the memory 210 and configured to, when invoked by one or more processors 220, cause one or more processors 220 to execute the vehicle control method provided in the embodiment of this application.

[0172] The processor 220 may include one or more processing cores. The processor 220 uses various interfaces and lines to connect to various parts of the vehicle 200, and is used to run or execute instructions, programs, code sets or instruction sets stored in the memory 210, as well as to call and run or execute data stored in the memory 210, and perform various functions of the vehicle 200 and process data.

[0173] In some implementations, the processor 220 may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 220 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and Modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 220 and may be implemented using a separate communication chip.

[0174] The memory 210 may include random access memory (RAM) or read-only memory (ROM). The memory 210 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 210 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may store data created by the vehicle 200 during use.

[0175] Figure 14 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 300 stores program code 310, which is configured to cause the processor to execute the vehicle control method described above in an embodiment of this application when called by the processor.

[0176] The computer-readable storage medium 300 may be an electronic storage device such as flash memory, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), hard disk, or ROM.

[0177] In some embodiments, the computer-readable storage medium 300 includes a non-volatile computer-readable storage medium (Non-TCRSM). The computer-readable storage medium 300 has storage space for program code 310 that performs any of the method steps described above. This program code 310 can be read from or written to one or more computer program products. The program code 310 may be compressed in a suitable form.

[0178] In summary, this application provides a vehicle control method, device, vehicle, and storage medium, relating to the field of intelligent vehicle control technology. The method obtains the pre-aiming angle output by a two-point pre-aiming visual model constructed based on the driver's visual behavior mechanism during driving; inputs visual scene information into a neural network model to obtain the predicted angle output by a neural network model constructed based on the driver's visual attention mechanism; employs a weighted fusion rule to perform weighted fusion processing on the pre-aiming angle and the predicted angle to obtain the target steering wheel angle; and sends the target steering wheel angle to a steering wheel angle controller, so that the steering wheel angle controller controls the steering wheel to follow the target steering wheel angle. This method considers the impact of the driver's driving style and driving behavior characteristics on the vehicle control effect, improving the rationality and accuracy of vehicle control, eliminating reliance on upper-level modules, and reducing system redundancy and complexity.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A vehicle control method, characterized in that, include: The aiming angle output by the two-point aiming visual model is obtained, and the two-point aiming visual model is constructed based on the driver's visual behavior mechanism during driving. Input visual scene information into a neural network model and obtain the predicted turning angle output by the neural network model. The neural network model is constructed based on the driver's visual attention mechanism. A weighted fusion rule is used to perform weighted fusion processing on the pre-aiming angle and the predicted angle to obtain the target steering wheel angle; The target steering wheel angle is sent to the steering wheel angle controller, so that the steering wheel angle controller controls the steering wheel to follow the target steering wheel angle. The step of employing a weighted fusion rule to perform weighted fusion processing on the pre-aiming angle and the predicted angle includes: Obtain the current visual detection status; Based on the correspondence between visual detection status and pre-aiming corner weights and predicted corner weights in the weighted fusion rules, the target pre-aiming corner weights and target predicted corner weights corresponding to the current visual detection status are determined. Based on the target pre-aiming angle weight and the target predicted angle weight, the pre-aiming angle and the predicted angle are weighted and fused.

2. The method according to claim 1, characterized in that, The current visual detection status includes the current state of the vehicle's sensors. The determination of the target pre-aiming angle weight and target predicted angle weight corresponding to the current visual detection status, based on the correspondence between the visual detection status and the pre-aiming angle weight and predicted angle weight in the weighted fusion rules, includes: Determine the level of the current vehicle sensor status; Based on the correspondence between the vehicle sensor status level and the pre-aiming angle weight and the predicted angle weight in the weighted fusion rules, the target pre-aiming angle weight and the target predicted angle weight corresponding to the current vehicle sensor status level are determined.

3. The method according to claim 1, characterized in that, The current visual detection situation includes the current environmental situation. The determination of the target pre-aiming corner weight and target predicted corner weight corresponding to the current visual detection situation, based on the correspondence between the visual detection situation and the pre-aiming corner weight and predicted corner weight in the weighted fusion rules, includes: Determine the level of the current environmental condition; Based on the correspondence between the environmental condition level and the pre-aiming angle weight interval and the predicted angle weight interval in the weighted fusion rule, the target pre-aiming angle weight interval and the target predicted angle weight interval corresponding to the current environmental condition level are determined. The target aiming angle weight is determined from the target aiming angle weight range; The target predicted angle weight is determined from the target predicted angle weight interval.

4. The method according to claim 3, characterized in that, The current environmental conditions include one or more of the following: current weather conditions, current road surface conditions, current road conditions, and current lighting conditions.

5. The method according to claim 1, characterized in that, The method further includes: During model training, a training dataset is acquired, which includes a pre-aiming angle training set, a predicted angle training set, and an actual steering wheel angle dataset. The pre-aiming angles in the pre-aiming angle training set, the predicted angles in the predicted angle training set, and the actual steering wheel angles in the actual steering wheel angle dataset have a corresponding relationship. The pre-aiming angles in the pre-aiming angle training set and the predicted angles in the predicted angle training set are weighted and fused using a preliminary weighted fusion rule to obtain the steering wheel angle training value. The loss value is calculated based on the training value of the steering wheel angle and the corresponding actual steering wheel angle. Based on the training dataset and the loss value, the preliminary weighted fusion rule is optimized and trained to obtain the weighted fusion rule.

6. The method according to claim 1, characterized in that, The method further includes: During the training of the two-point pre-aiming visual model, the near pre-aiming distance interval, the far pre-aiming distance interval, and the training dataset are obtained, and the training dataset includes the actual steering wheel angle. The near-pre-aiming distance is determined from the near-pre-aiming distance range, and the far-pre-aiming distance is determined from the far-pre-aiming distance range. The near-pre-aiming distance is the distance from the vehicle's center of gravity to the near-pre-aiming point, and the far-pre-aiming distance is the distance from the vehicle's center of gravity to the far-pre-aiming point. Input the near pre-aiming distance and the far pre-aiming distance into the two-point pre-aiming visual model to obtain the pre-aiming angle training value; The loss value is calculated based on the pre-aiming angle training value and the corresponding actual steering wheel angle. Based on the near-pre-aiming distance interval, the far-pre-aiming distance interval, and the loss value, the near-pre-aiming distance and the far-pre-aiming distance in the two-point pre-aiming visual model are optimized and trained.

7. The method according to claim 6, characterized in that, The near-pre-aiming distance and far-pre-aiming distance in the two-point pre-aiming visual model are linked to the driver's account.

8. The method according to claim 6 or 7, characterized in that, The two-point pre-aiming visual model includes a two-point pre-aiming model and a lower-level vision system. Obtaining the pre-aiming angle output by the two-point pre-aiming visual model includes: Input the optimized near and far aiming distances, the lateral position deviation at the current near aiming point, the current vehicle heading angle, and the current vehicle turning radius into the two-point aiming model to obtain the heading angle deviation at the near aiming point and the heading angle deviation at the far aiming point. Input the heading angle deviation at the near pre-aiming point and the heading angle deviation at the far pre-aiming point into the lower-level vision system to obtain the pre-aiming turning angle.

9. The method according to claim 8, characterized in that, The lower-level vision system includes a visual prediction module, a visual compensation module, and a signal transmission delay module. The process of inputting the heading angle deviation at the near pre-aiming point and the heading angle deviation at the far pre-aiming point to the lower-level vision system to obtain the pre-aiming angle includes: Input the heading angle deviation at the far aiming point into the visual prediction module to obtain the first heading angle deviation; Input the heading angle deviation at the near-pre-aiming point to the visual compensation module to obtain the second heading angle deviation; The initial aiming angle is determined based on the first heading angle deviation and the second heading angle deviation; The preliminary aiming angle is input into the signal transmission delay module to obtain the aiming angle.

10. The method according to claim 1, characterized in that, The neural network model includes a visual attention mechanism model, a long short-term memory network layer, and a fully connected layer. The input visual scene information is fed into the neural network model, and the predicted turning angle output by the neural network model is obtained, including: The visual scene information is input into the visual attention mechanism model to obtain the first visual scene feature map; The first visual scene feature map is input into the long short-term memory network layer to obtain the second visual scene feature map; The second visual scene feature map is input into the fully connected layer to obtain the third visual scene feature map; Based on the correspondence between the visual scene feature map and the predicted turning angle, the predicted turning angle corresponding to the third visual scene feature map is determined.

11. The method according to claim 10, characterized in that, The neural network model further includes a backpropagation training model, and the method further includes: The loss value is calculated based on the predicted steering angle and the actual steering wheel angle. The loss value is input into the backpropagation training model so that the backpropagation training model updates the network weights of the neural network model.

12. A vehicle control device, characterized in that, include: The pre-aiming angle acquisition module is used to acquire the pre-aiming angle output by the two-point pre-aiming visual model, which is constructed based on the driver's visual behavior mechanism during driving. The predicted turning angle acquisition module is used to input visual scene information into a neural network model and obtain the predicted turning angle output by the neural network model. The neural network model is constructed based on the driver's visual attention mechanism. The data weighted fusion module is used to perform weighted fusion processing on the pre-aiming angle and the predicted angle using weighted fusion rules to obtain the target steering wheel angle. The data transmission module is used to send the target steering wheel angle to the steering wheel angle controller, so that the steering wheel angle controller controls the steering wheel to follow the target steering wheel angle. The step of employing a weighted fusion rule to perform weighted fusion processing on the pre-aiming angle and the predicted angle includes: Obtain the current visual detection status; Based on the correspondence between visual detection status and pre-aiming corner weights and predicted corner weights in the weighted fusion rules, the target pre-aiming corner weights and target predicted corner weights corresponding to the current visual detection status are determined. Based on the target pre-aiming angle weight and the target predicted angle weight, the pre-aiming angle and the predicted angle are weighted and fused.

13. A vehicle, characterized in that, include: Memory; One or more processors; One or more applications, wherein the one or more applications are stored in the memory and configured to, when invoked by the one or more processors, cause the one or more processors to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that, when invoked by a processor, causes the processor to perform the method as described in any one of claims 1-11.

Images