A method, device, electronic device and storage medium for determining vehicle attitude angle

Abstract

The present disclosure provides a method, device, electronic device and storage medium for determining a vehicle attitude angle. The method comprises: obtaining an AVM image at a target moment collected during vehicle driving; inputting the AVM image into a trained machine learning model, and having the machine learning model calculate a first predicted vehicle attitude parameter at the target moment based on the AVM image; determining the vehicle attitude parameter at the target moment based at least on a Kalman filter algorithm, the vehicle attitude parameter at the previous moment of the target moment, and the first predicted vehicle attitude parameter. Accordingly, the present disclosure enables a vehicle to obtain a relatively accurate vehicle attitude angle by collecting the AVM image at this time and then applying a pre-trained machine learning model and Kalman filter when driving to a changing section of a road with a sensor of general accuracy, thereby avoiding the use of sensors of general accuracy to calculate and obtain the vehicle attitude angle, thereby helping vehicle navigation and path planning.

Description

Technical Field

[0001] One or more embodiments of the present disclosure relate to the field of data processing technology, and in particular, to a method, device, electronic device, and storage medium for determining a vehicle attitude angle. Background Art

[0002] In intelligent driving systems, the use of on-board IMU (Inertial Measurement Unit) sensors can measure and report the vehicle's tilt, pitch, and yaw angles, as well as the vehicle's acceleration and angular velocity in three spatial dimensions, thereby helping to determine the vehicle's motion state, such as acceleration, deceleration, and turning. It can be seen that in intelligent driving systems, on-board IMU sensors help vehicles perceive their position and state in the environment. In order to accurately obtain the vehicle's posture changes, high requirements are placed on the accuracy and performance of the on-board IMU sensors.

[0003] However, in real life, due to factors such as cost and manufacturing process, the accuracy of mass-produced vehicle-mounted IMU sensors is average under large-scale production. When the vehicle passes through a stable road section, the performance of the vehicle-mounted IMU sensor is acceptable. However, the vehicle will encounter various situations during actual driving, such as bumps, speed bumps, entering underground garages, turning, etc., which may cause large errors in the vehicle's attitude angle prediction, causing deviations in the visual positioning of the intelligent driving system, thereby affecting navigation and path planning. Summary of the invention

[0004] The present disclosure provides a method for determining a vehicle attitude angle, the method comprising:

[0005] Acquire the AVM image at the target time collected during the vehicle driving process;

[0006] Inputting the AVM image into a trained machine learning model, and having the machine learning model calculate a first predicted vehicle posture parameter at the target moment based on the AVM image;

[0007] The vehicle posture parameter at the target moment is determined based at least on a Kalman filter algorithm, the vehicle posture parameter at a moment before the target moment, and the first predicted vehicle posture parameter.

[0008] Optionally, the vehicle posture parameter includes a vehicle rotation angle;

[0009] The vehicle rotation angle includes one or more of the following: pitch angle; roll angle; yaw angle; angular velocity.

[0010] Optionally, the machine learning model includes a feature extraction layer and at least one fully connected layer; wherein one fully connected layer is used to calculate a type of vehicle rotation angle;

[0011] The step of inputting the AVM image into a trained machine learning model, and having the machine learning model calculate a first predicted vehicle posture parameter at the target time based on the AVM image, comprises:

[0012] The AVM image is input into the trained machine learning model, the feature extraction layer performs feature extraction on the AVM image to obtain feature data corresponding to the AVM image, and each fully connected layer calculates various types of vehicle rotation angles based on the feature data.

[0013] Optionally, the method further includes:

[0014] Acquiring sensor data at the target time point collected during vehicle driving;

[0015] Calculating a second predicted vehicle posture parameter at the target time based on the sensor data;

[0016] The method of determining the vehicle posture parameter at the target moment based on the Kalman filter algorithm, the vehicle posture parameter at the previous moment of the target moment, and the first predicted vehicle posture parameter comprises:

[0017] The vehicle posture parameter at the target moment is determined based on a Kalman filter algorithm, the vehicle posture parameter at a moment before the target moment, and the first predicted vehicle posture parameter and the second predicted vehicle posture parameter.

[0018] Optionally, determining the vehicle posture parameter at the target moment based on a Kalman filter algorithm, the vehicle posture parameter at a previous moment of the target moment, and the first predicted vehicle posture parameter and the second predicted vehicle posture parameter includes:

[0019] Defining a state transfer equation of a Kalman filter algorithm based on an integral model of the sensor data;

[0020] Based on the first predicted vehicle posture parameter and the second predicted vehicle posture parameter, two observation equations of a Kalman filter algorithm are defined respectively, and the two observation equations are combined into one observation equation by weighting;

[0021] Based on the defined Kalman filter algorithm, the vehicle posture parameters at the previous moment of the target moment, and the first predicted vehicle posture parameters and the second predicted vehicle posture parameters, the vehicle posture parameters at the target moment are determined.

[0022] Optionally, the machine learning model includes a residual network.

[0023] Optionally, the method further includes:

[0024] Acquire an AVM image sample marked with a corresponding label; wherein the AVM image sample is an AVM image of a vehicle when it is traveling on a road section where the road condition has changed, and the label is a vehicle posture parameter;

[0025] The machine learning model is supervisedly trained based on the AVM image samples.

[0026] The present disclosure also provides a vehicle attitude angle determination device, the device comprising:

[0027] An acquisition unit, used for acquiring an AVM image at a target time point collected during the driving process of the vehicle;

[0028] An input unit, used for inputting the AVM image into a trained machine learning model, so that the machine learning model calculates a first predicted vehicle posture parameter at the target moment based on the AVM image;

[0029] A determination unit is used to determine the vehicle posture parameter at the target moment based on a Kalman filter algorithm, the vehicle posture parameter at a moment before the target moment, and the first predicted vehicle posture parameter.

[0030] The present disclosure also provides an electronic device, comprising a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0031] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0032] The present disclosure also provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the above method is implemented.

[0033] Through the embodiments of the present disclosure, an AVM image at a target moment collected during the driving process of the vehicle is obtained, and the AVM image is input into a pre-trained machine learning model to obtain vehicle posture parameters at the same moment as the AVM image calculated by the machine learning model for the AVM image, and then Kalman filtering is performed on the vehicle posture parameters calculated by the machine learning model to eliminate noise interference in the vehicle posture parameters at the target moment calculated by the machine learning model, so that the vehicle posture parameters are more accurate.

[0034] Through the above method, the present invention enables a vehicle equipped with sensors of general accuracy to obtain a more accurate vehicle attitude angle when it travels on a changing road section by collecting the AVM image at this time, and then passing it through a pre-trained machine learning model and Kalman filter, thereby avoiding the use of sensors of general accuracy to calculate and obtain the vehicle attitude angle, thereby helping vehicle navigation and path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 is a flow chart of a method for determining a vehicle attitude angle shown in an exemplary embodiment;

[0037] Figure 2 is a schematic diagram of a method for optimizing vehicle attitude angle estimation shown in an exemplary embodiment;

[0038] Figure 3 is a schematic diagram of a method for predicting vehicle attitude angle parameters shown in an exemplary embodiment;

[0039] Figure 4 is a hardware structure diagram of an electronic device shown in an exemplary embodiment;

[0040] Figure 5 It is a block diagram of a vehicle attitude angle determination device shown in an exemplary embodiment. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present disclosure.

[0042] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this disclosure. In some other embodiments, the steps included in the method may be more or less than those described in this disclosure. In addition, a single step described in this disclosure may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this disclosure may also be combined into a single step for description in other embodiments.

[0043] In intelligent driving systems, the use of on-board IMU (Inertial Measurement Unit) sensors can measure and report the vehicle's tilt, pitch, and yaw angles, as well as the vehicle's acceleration and angular velocity in three spatial dimensions, thereby helping to determine the vehicle's motion state, such as acceleration, deceleration, and turning. It can be seen that in intelligent driving systems, on-board IMU sensors help vehicles perceive their position and state in the environment. In order to accurately obtain the vehicle's posture changes, high requirements are placed on the accuracy and performance of the on-board IMU sensors.

[0044] However, in real life, due to factors such as cost and manufacturing process, the accuracy of mass-produced vehicle-mounted IMU sensors is average under large-scale production. When the vehicle passes through a stable road section, the performance of the vehicle-mounted IMU sensor is acceptable. However, the vehicle will encounter various situations during actual driving, such as bumps, speed bumps, entering underground garages, turning, etc., which may cause large errors in the vehicle's attitude angle prediction, causing deviations in the visual positioning of the intelligent driving system, thereby affecting navigation and path planning.

[0045] In view of this, the present disclosure aims to propose a technical solution that can predict the vehicle attitude angle at the corresponding moment based on the AVM image (Around View Monitor, panoramic parking imaging system) during vehicle driving, so as to obtain accurate vehicle attitude angle when the IMU sensor accuracy is general.

[0046] In some embodiments, an AVM image at a target moment captured during vehicle driving is first acquired; then, the AVM image is input into a trained machine learning model, and the machine learning model calculates a first predicted vehicle posture parameter at the target moment based on the AVM image; further, the vehicle posture parameters at the target moment are determined based at least on a Kalman filtering algorithm, the vehicle posture parameters at a previous moment before the target moment, and the first predicted vehicle posture parameters.

[0047] For example, when the vehicle A driven by the user arrives at the turning point of the parking lot exit at 10 am, the AVM image of the vehicle collected at this time is first obtained. The AVM image includes a 360° panoramic view around the vehicle body, that is, it includes the front image of the front of the vehicle, the images of the two sides of the vehicle body, and the image of the rear of the vehicle body when the vehicle A arrives at the turning point of the parking lot exit. Then, the acquired vehicle AVM image is input into the trained machine learning model, and the attitude parameters of the vehicle A at 10 am calculated based on the input AVM image output by the machine learning model can be obtained, including attitude angle and angular velocity, wherein the attitude angle includes pitch angle, roll angle and yaw angle. Finally, the attitude parameters of the vehicle A at 10 am output by the machine learning model are Kalman filtered. The specific process is to perform Kalman filtering algorithm calculation based on the vehicle attitude parameters at the previous moment of 10 am (e.g., 9:59:59 am) and the attitude parameters of the vehicle A at 10 am output by the machine learning model, so as to obtain more accurate attitude parameters of the vehicle A at 10 am.

[0048] It can be seen that in the technical solution disclosed in the present invention, by acquiring the AVM image at the target moment collected during the driving process of the vehicle, the AVM image is input into a pre-trained machine learning model to obtain the vehicle posture parameters at the same moment as the AVM image calculated by the machine learning model for the AVM image, and then the vehicle posture parameters calculated by the machine learning model are subjected to Kalman filtering, thereby eliminating the noise interference of the vehicle posture parameters at the target moment calculated by the machine learning model, so that the vehicle posture parameters are more accurate.

[0049] Through the above method, the present invention enables a vehicle equipped with sensors of general accuracy to obtain a more accurate vehicle attitude angle when it travels on a changing road section by collecting the AVM image at this time, and then passing it through a pre-trained machine learning model and Kalman filter, thereby avoiding the use of sensors of general accuracy to calculate and obtain the vehicle attitude angle, thereby helping vehicle navigation and path planning.

[0050] The present disclosure is described below through specific embodiments in combination with specific application scenarios.

[0051] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for determining a vehicle attitude angle according to an exemplary embodiment. The method may perform the following steps:

[0052] Step 102: Acquire the AVM image at the target time collected during the vehicle driving process.

[0053] For example, when vehicle A driven by a user arrives at the turning point of a parking lot exit at 10 a.m., an AVM image of the vehicle collected at this time is obtained. The AVM image includes a 360° panoramic view around the vehicle body, that is, it includes the front image of the vehicle, the images of the two sides of the vehicle body, and the rear image of the vehicle body when vehicle A arrives at the turning point of the parking lot exit.

[0054] Among them, AVM (Around View Monitor, panoramic parking imaging system) is often used in intelligent driving systems. It captures images of the vehicle's surroundings by shooting images through multiple ultra-wide-angle fisheye lenses installed on the front, back, left and right sides of the vehicle body, and then stitches these images into a panoramic image to provide the driver with a 360° bird's-eye view. The fisheye lens is an ultra-wide-angle lens that can capture a wider field of view than traditional lenses. In the AVM system, the fisheye lens is used to capture the environment around the vehicle in order to create a panoramic image. Image stitching technology is used to seamlessly combine images captured from different cameras to form a continuous panoramic image. The process may involve steps such as image correction, feature matching, image transformation and fusion. As for how to stitch together the AVM image, the present disclosure does not limit this.

[0055] Step 104: Input the AVM image into a trained machine learning model, and the machine learning model calculates the first predicted vehicle posture parameter at the target moment based on the AVM image.

[0056] For example, by inputting the acquired vehicle AVM image into the trained machine learning model, the posture parameters of vehicle A at 10 am calculated based on the input AVM image output by the machine learning model can be obtained, including posture angle and angular velocity, where the posture angle includes pitch angle, roll angle and yaw angle.

[0057] Among them, the machine learning model can learn from the data and make decisions. In the present disclosure, the machine learning model is trained to understand and interpret the AVM image and extract the posture parameters of the vehicle therefrom. The trained machine learning model has been trained with a large amount of data, which includes AVM images and their corresponding vehicle posture parameters. During the training process, the machine learning model will learn how to identify and predict posture parameters from the AVM image. The first predicted vehicle posture parameter refers to the preliminary prediction result calculated by the model based on the input image. The posture parameter describes the parameters of the direction and position of the vehicle in space. The output of the machine learning model needs to be interpreted as the posture parameters of the vehicle, which may involve converting the angle value output by the model into an actual physical angle and performing necessary post-processing. The present disclosure does not limit the specific method of post-processing.

[0058] Step 106: Determine the vehicle posture parameter at the target moment based at least on a Kalman filter algorithm, the vehicle posture parameter at a moment before the target moment, and the first predicted vehicle posture parameter.

[0059] For example, Kalman filtering is performed on the posture parameters of vehicle A at 10 a.m. output by the machine learning model. The specific process is to perform Kalman filtering algorithm calculation based on the vehicle posture parameters at the previous moment of 10 a.m. (e.g., 9:59:59 a.m.) and the posture parameters of vehicle A at 10 a.m. output by the machine learning model, so as to obtain more accurate posture parameters of vehicle A at 10 a.m.

[0060] Among them, Kalman filtering is a recursive estimation algorithm used for state estimation of linear dynamic systems. It can estimate the state of a dynamic system from a series of measurement data in the presence of noise. The vehicle posture parameters at the previous moment of the target moment refer to the vehicle posture parameters recorded or estimated at the time point before the target moment. In Kalman filtering, the vehicle posture parameters at the previous moment of the target moment are used as prior information to predict the state at the current moment. The first predicted vehicle posture parameter refers to the preliminary estimate of the vehicle posture parameters at the current moment obtained by the machine learning model (i.e., the current observation). Kalman filtering combines prior information and current observations to provide an optimal estimate, thereby determining the posture parameters of the vehicle at the target moment.

[0061] In this embodiment, by using the Kalman filter algorithm, historical data and real-time prediction can be effectively combined to obtain a smoother and more reliable vehicle posture parameter estimation.

[0062] In one embodiment shown, the vehicle posture parameter includes a vehicle rotation angle; the vehicle rotation angle includes one or more of the following: pitch angle; roll angle; yaw angle; angular velocity.

[0063] For example, the attitude parameters of vehicle A include the pitch angle, roll angle, yaw angle and angular velocity of vehicle A. For example, the pitch angle may be +5° (the front end of the vehicle is tilted upward) or -3° (the front end of the vehicle is tilted downward). The roll angle may be +2° (the vehicle is tilted to the left) or -2° (the vehicle is tilted to the right). The yaw angle may be 15° (the vehicle is yawed to the right relative to the north) or -10° (the vehicle is yawed to the left relative to the north). The angular velocity may be: pitch angular velocity: 0.5° / s (the front end of the vehicle tilts upward 0.5° per second), roll angular velocity: -0.3° / s (the vehicle tilts to the right 0.3° per second), yaw angular velocity: 1.2° / s (the vehicle yaws 1.2° to the right relative to the north per second).

[0064] Among them, the vehicle rotation angle can also be called the vehicle attitude angle. In the embodiments of the present disclosure, the two can be interchanged. The vehicle attitude angle includes the pitch angle (i.e., the rotation angle of the vehicle around the x-axis, that is, the inclination of the vehicle relative to the horizontal plane), the roll angle (i.e., the rotation angle of the vehicle around the y-axis, that is, the left and right inclination of the vehicle), and the yaw angle (i.e., the rotation angle of the vehicle around the z-axis, that is, the angle between the vehicle's forward direction and the reference direction (such as geographic north)). The vehicle angular velocity describes the rate at which the vehicle attitude angle changes over time, that is, the speed at which the vehicle rotates. The above values ​​are only examples, and the actual vehicle attitude parameters will vary depending on the actual movement of the vehicle and the environment in which it is located. In actual applications, these parameters can be measured by sensors (such as gyroscopes, accelerometers, magnetometers, etc.), and fused and optimized through data processing algorithms (such as Kalman filtering) to obtain more accurate and reliable estimates.

[0065] In one embodiment shown, the machine learning model includes a feature extraction layer and at least one fully connected layer; wherein, a fully connected layer is used to calculate a type of vehicle rotation angle; the AVM image is input into the trained machine learning model, and the machine learning model calculates the first predicted vehicle posture parameter at the target moment based on the AVM image, including: inputting the AVM image into the trained machine learning model, the feature extraction layer performs feature extraction on the AVM image to obtain feature data corresponding to the AVM image, and each fully connected layer calculates various types of vehicle rotation angles based on the feature data.

[0066] For example, when a user drives vehicle A to a complex intersection at 10 a.m., he needs to accurately estimate the vehicle's posture parameters to make the right driving decision. The vehicle's AVM image is collected at this time, and the AVM image is input into the trained machine learning model. The machine learning model structurally includes a feature extraction layer and a fully connected layer. The features extracted by the feature extraction layer encode useful information in the vehicle's AVM image, such as road edges, traffic signs, the positions of other vehicles, and other feature data related to the vehicle's posture. After the feature extraction layer, there are one or more fully connected layers used to convert the feature data into prediction results. Each fully connected layer may be dedicated to calculating a type of posture parameter. For example, the first fully connected layer may calculate the pitch angle and output a value, such as +2°, indicating that the front end of the vehicle is tilted 2 degrees upward. The second fully connected layer may calculate the roll angle and output -1°, indicating that the vehicle is tilted 1 degree to the right. The third fully connected layer may calculate the yaw angle and output 3°, indicating that the vehicle is yawed 3 degrees to the right relative to the north. The fourth fully connected layer may calculate the angular velocity, and the output values ​​​​may be 0.2° / s for pitch, -0.1° / s for roll, and 0.3° / s for yaw.

[0067] Among them, the feature extraction layer is usually based on a convolutional neural network (CNN) to extract visual features from the input AVM image. For example, the first convolution layer may focus on extracting edge information, while subsequent convolution layers may extract more complex shapes or patterns. Each fully connected layer independently calculates a vehicle rotation angle or angular velocity based on the feature data.

[0068] In one embodiment shown, the method also includes: acquiring sensor data at the target moment collected during vehicle driving; calculating a second predicted vehicle posture parameter at the target moment based on the sensor data; determining the vehicle posture parameter at the target moment based on the Kalman filter algorithm, the vehicle posture parameter at the previous moment of the target moment, and the first predicted vehicle posture parameter, including: determining the vehicle posture parameter at the target moment based on the Kalman filter algorithm, the vehicle posture parameter at the previous moment of the target moment, the first predicted vehicle posture parameter and the second predicted vehicle posture parameter.

[0069] For example, vehicle A is undergoing an autonomous driving test, and vehicle A is equipped with an onboard IMU (Inertial Measurement Unit) sensor. During the vehicle's driving, sensor data at the target time (for example, 10 a.m.) is collected, and the sensor data is used to make a preliminary estimate of the attitude parameters of vehicle A. Since the accuracy of the IMU sensor is average, the data collected may contain noise and deviation, and the Kalman filter algorithm is also used to combine the attitude parameters of vehicle A at the previous moment and the IMU data at the current moment to optimize the estimation of the attitude parameters of vehicle A.

[0070] See also Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a method for optimizing the estimation of a vehicle attitude angle according to an exemplary embodiment. Figure 2 As shown, an AVM image of vehicle A at a target moment during driving is collected, and the AVM image is input into a pre-trained machine learning model to obtain a first predicted vehicle posture parameter learned and predicted from the AVM image output by the machine learning model; at the same time, on-board sensor data of vehicle A at the target moment during driving is collected, and a second predicted vehicle posture parameter is calculated based on the sensor data; the first predicted vehicle posture parameter and the second predicted vehicle posture parameter are subjected to Kalman filtering to obtain a smoother and more accurate vehicle posture parameter estimate after noise is filtered out.

[0071] Among them, the on-board sensor is usually an on-board IMU (Inertial Measurement Unit) sensor. The IMU sensor data includes but is not limited to angular velocity (from a gyroscope), acceleration (from an accelerometer), and possible magnetic field data (if the IMU includes a magnetometer, there is magnetic field data). The key components of Kalman filtering include state estimation, estimation error covariance, observation model, system model, Kalman gain, etc. After Kalman filtering, the attitude parameters of vehicle A are optimized, reducing the errors caused by sensor noise and bias.

[0072] In one embodiment shown, the vehicle posture parameters at the target moment are determined based on the Kalman filter algorithm, the vehicle posture parameters at the previous moment, and the first predicted vehicle posture parameters and the second predicted vehicle posture parameters, including: defining the state transfer equation of the Kalman filter algorithm based on the integral model of the sensor data; defining two observation equations of the Kalman filter algorithm based on the first predicted vehicle posture parameters and the second predicted vehicle posture parameters, respectively, and merging the two observation equations into one observation equation by weighting; determining the vehicle posture parameters at the target moment based on the defined Kalman filter algorithm, the vehicle posture parameters at the previous moment, and the first predicted vehicle posture parameters and the second predicted vehicle posture parameters.

[0073] For example, the IMU sensor mounted on vehicle A provides the following initial angular velocity data at 10 am: pitch velocity: 0.5° / s, roll velocity: 0.3° / s, yaw velocity: 0.2° / s; the initial attitude angle is 0 degrees. In the next second, we integrate these angular velocities to obtain new attitude angle changes. These changes are added to the initial attitude angle to obtain new pitch angle = 0 + 0.5 degrees, new roll angle = 0 + 0.3 degrees, and new yaw angle = 0 + 0.2 degrees. The Kalman filter uses this new state as the state for the next moment of prediction. During the driving process of vehicle A, there are two ways to predict the attitude parameters of vehicle A: the first prediction is based on the prediction of the camera system (AVM image analysis), and the second prediction is based on the prediction of IMU sensor data. For example, the current pitch angle of the vehicle obtained by AVM image analysis is 2°, and the current pitch angle of the vehicle obtained by IMU sensor data analysis is 1.5°. In Kalman filtering, an observation equation is defined for each of the two prediction methods. The observation equation is a mathematical expression used to describe how to get the predicted value from the actual measurement. In order to get a more accurate prediction, a weight is assigned to each prediction method. These weights are based on our trust in each prediction method. For example, if we trust AVM image analysis more, then we give it a larger weight, and finally combine the two predictions to form a comprehensive prediction and obtain the combined observation equation. For example, the weight given to the AVM image analysis prediction is 0.6, and the weight given to the IMU sensor data prediction is 0.4, then the comprehensive predicted pitch angle will be: Finally, based on the Kalman filter algorithm that defines the state transfer equation and the merged observation equation, the vehicle posture parameters at the previous moment of the target moment, the vehicle posture parameters obtained by AVM image analysis at the target moment, and the vehicle posture parameters obtained by IMU sensor data analysis at the target moment are referred to determine the accurate and reliable vehicle posture parameters at the target moment.

[0074] Among them, the attitude state of the vehicle is usually represented by three angles: pitch angle, roll angle and yaw angle. In Kalman filtering, the change of these angles over time is the state of the vehicle attitude parameter that we want to estimate. The IMU sensor provides angular velocity data, which is a measurement of the rotation rate of the vehicle around each axis. By integrating the angular velocity data (i.e., accumulating over time), the change in attitude angle can be obtained. In Kalman filtering, the state transfer equation describes the evolution of the system state from one time step to the next time step, which can be simply understood as "adding the change obtained by integrating the attitude angle at the previous moment". Since IMU data is collected continuously and the change in attitude angle is usually obtained by integration, the state transfer equation can be based on the integral model of IMU data. Kalman filtering uses the state transfer equation to predict the attitude state at the next moment, and when new IMU data arrives, it fuses this data into the prediction to update and correct the attitude estimate.

[0075] In one illustrated embodiment, the machine learning model includes a residual network.

[0076] For example, the machine learning model used in this embodiment is ResNet-18, which is a type of residual network.

[0077] Among them, ResNet-18 is a deep learning architecture, a specific version in the residual network family, with a depth of 18 layers, and uses residual connections to improve the training efficiency and performance of deep networks. The core idea of ​​the residual network is to introduce the concept of "residual learning". By adding "residual connections", the signal is allowed to directly skip one or more layers, thereby solving the problem of increasing training difficulty as the network depth increases. This design enables the network to maintain the flow of gradients during the learning process, so that even very deep networks can be effectively trained. ResNet-18 contains multiple residual blocks, and residual connections are implemented in each residual block, allowing the input to be directly added to the output of the block, so that the network can learn the identity mapping during training (that is, directly passing the input to the output), thereby avoiding the gradient disappearance or gradient explosion problems in deep network training.

[0078] In one embodiment shown, the method also includes: obtaining AVM image samples marked with corresponding labels; wherein the AVM image samples are AVM images of the vehicle when it is driving on a road section where road conditions have changed, and the labels are vehicle posture parameters; and performing supervised training on the machine learning model based on the AVM image samples.

[0079] For example, obtain several groups of AVM images of vehicle A when it is driving on a curved section and the true values ​​of the vehicle posture parameters at this time (the accuracy will be more accurate than the vehicle posture parameters calculated by general on-board IMU sensor data), obtain several groups of AVM images of vehicle A when it is driving on a bumpy section and the true values ​​of the vehicle posture parameters at this time, obtain several groups of AVM images of vehicle A when it is driving into an underground garage section and the true values ​​of the vehicle posture parameters at this time, and perform supervised training on the machine learning model based on the AVM images of each group and the corresponding true values ​​of the vehicle posture parameters.

[0080] Among them, in machine learning, labels refer to known output results associated with input data. In the scenario disclosed in this disclosure, vehicle posture parameters are labels, which correspond to AVM image samples and are used to train the model to understand the relationship between images and vehicle posture. Supervised training refers to providing a data set with correct answers (i.e., labels) when training a machine learning model. The model learns the association between input data (AVM images) and outputs (vehicle posture parameters), with the goal of being able to make accurate predictions when the model encounters new data without labels in the future.

[0081] In this embodiment, the true value of the vehicle attitude parameter is usually obtained in the following ways: the first is to obtain it by fusing data from multiple sensors, and its accuracy is usually higher than that of the vehicle attitude parameter obtained by solving the data of a single IMU sensor. The second is to use the IMU sensor data with higher accuracy to solve the true value of the vehicle attitude parameter. The third is to calculate the distance between the vehicle and the surrounding objects by emitting a laser beam and measuring the reflection time of the laser beam, and then estimate the attitude angle of the vehicle. This method has high measurement accuracy and is suitable for complex environments, but the equipment cost is high and the data processing is complex, so this method is rarely used. The present disclosure does not limit the specific method of obtaining the true value of the vehicle attitude angle during driving.

[0082] In order to facilitate the understanding of the technical solution of the present disclosure by those skilled in the art, an example of the process of predicting vehicle attitude angle parameters using a trained machine learning model is given below:

[0083] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a method for predicting vehicle attitude angle parameters according to an exemplary embodiment. Figure 3As shown in the figure, first, a 500x500 resolution vehicle AVM image is input to the trained machine learning model. The machine learning model adopts the ResNet-18 architecture, which is a classic deep residual network, usually used for image classification tasks. Its output layer is usually a fully connected layer, which outputs the probability of classification. In order to extract useful feature maps, the standard ResNet model needs to be modified in advance. Usually, the feature maps of the deeper layers in the network should be retained, such as retaining the feature maps of 30x30 resolution, to contain important information of the input image, which is used for the machine learning model to learn the attitude angle parameters. After the feature map, the machine learning model design contains multiple fully connected layers, which are used to learn attitude parameters from the feature map, such as rotation angle (pitch angle, roll angle, yaw angle) and angular velocity. Each fully connected layer corresponds to a physical quantity. In order to train the model, a suitable loss function (such as mean square error) is also selected to measure the difference between the predicted value and the actual value. Secondly, the input data is forward passed through the network to obtain the predicted output. Then, the loss function is used to calculate the difference between the predicted output and the true label. According to the gradient of the loss function, the network weights are updated through the network back propagation. Finally, the optimization algorithm is used to update the network parameters based on the gradient obtained by back propagation. In order to improve the generalization ability of the model and cover more scenarios, data enhancement technology can also be used to increase the diversity of training data.

[0084] Among them, the fully connected layer is used to identify and classify the input content. In essence, it takes an angle value or angular velocity value as a category, and then the machine learning model determines which category the input AVM image belongs to. The angle value or angular velocity value corresponding to this category is the vehicle posture parameter value corresponding to the AVM image. Therefore, in the machine learning model, after the AVM image is processed by the fully connected layer, it must be processed by the Softmax function (an activation function used for multi-classification problems) to calculate the probability that the AVM image belongs to each category, and the category with the highest probability is taken as the category to which the AVM image belongs.

[0085] Corresponding to the above-mentioned embodiment of the vehicle attitude angle determination method, the present disclosure also provides an embodiment of a vehicle attitude angle determination device.

[0086] See also Figure 4 , Figure 4It is a hardware structure diagram of an electronic device shown in an exemplary embodiment. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of the present disclosure can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present disclosure do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0087] See also Figure 5 , Figure 5 FIG. 5 is a block diagram of a vehicle attitude angle determination device according to an exemplary embodiment. The vehicle attitude angle determination device 500 can be applied to Figure 4 In the electronic device shown, the technical solution of the present disclosure is implemented. The device includes:

[0088] The first acquisition unit 502 is used to acquire the AVM image at the target time collected during the vehicle driving process;

[0089] An input unit 504 is used to input the AVM image into a trained machine learning model, and the machine learning model calculates a first predicted vehicle posture parameter at the target moment based on the AVM image;

[0090] The determination unit 506 is used to determine the vehicle posture parameter at the target moment based on at least a Kalman filter algorithm, the vehicle posture parameter at a previous moment of the target moment, and the first predicted vehicle posture parameter.

[0091] In some embodiments, the vehicle posture parameter includes a vehicle rotation angle;

[0092] The vehicle rotation angle includes one or more of the following: pitch angle; roll angle; yaw angle; angular velocity.

[0093] In some embodiments, the machine learning model includes a feature extraction layer and at least one fully connected layer; wherein one fully connected layer is used to calculate a type of vehicle rotation angle; the input unit includes:

[0094] The input subunit is used to input the AVM image into the trained machine learning model, and the feature extraction layer performs feature extraction on the AVM image to obtain feature data corresponding to the AVM image, and each fully connected layer calculates various types of vehicle rotation angles based on the feature data.

[0095] In some embodiments, the apparatus further comprises:

[0096] A second acquisition unit, used to acquire sensor data at the target time collected during the vehicle's driving process;

[0097] a calculation unit, configured to calculate a second predicted vehicle posture parameter at the target time based on the sensor data;

[0098] The determining unit comprises:

[0099] A determination subunit is used to determine the vehicle posture parameter at the target moment based on a Kalman filter algorithm, the vehicle posture parameter at the previous moment of the target moment, and the first predicted vehicle posture parameter and the second predicted vehicle posture parameter.

[0100] In some embodiments, the determining subunit is specifically used to:

[0101] Defining a state transfer equation of a Kalman filter algorithm based on an integral model of the sensor data;

[0102] Based on the first predicted vehicle posture parameter and the second predicted vehicle posture parameter, two observation equations of a Kalman filter algorithm are defined respectively, and the two observation equations are combined into one observation equation by weighting;

[0103] Based on the defined Kalman filter algorithm, the vehicle posture parameters at the previous moment of the target moment, and the first predicted vehicle posture parameters and the second predicted vehicle posture parameters, the vehicle posture parameters at the target moment are determined.

[0104] In some embodiments, the machine learning model comprises a residual network.

[0105] In some embodiments, the apparatus further comprises:

[0106] A third acquisition unit is used to acquire an AVM image sample marked with a corresponding label; wherein the AVM image sample is an AVM image of a vehicle when it is traveling on a road section with a changed road condition, and the label is a vehicle posture parameter;

[0107] A training unit is used to perform supervised training on the machine learning model based on the AVM image samples.

[0108] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0109] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0110] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0111] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0113] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0114] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0115] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0116] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] The terms used in one or more embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present disclosure. The singular forms of "a", "said" and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0118] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0119] The above description is only a preferred embodiment of one or more embodiments of the present disclosure and is not intended to limit one or more embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present disclosure shall be included in the scope of protection of one or more embodiments of the present disclosure.

Claims

1. A method for determining a vehicle attitude angle, characterized in that: The method comprises: Acquire the AVM image and sensor data at the target time collected during the vehicle driving process; Inputting the AVM image into a trained machine learning model, and having the machine learning model calculate a first predicted vehicle posture parameter at the target moment based on the AVM image; Calculating a second predicted vehicle posture parameter at the target time based on the sensor data; Determining the vehicle posture parameter at the target moment based at least on a Kalman filter algorithm, the vehicle posture parameter at a moment before the target moment, and the first predicted vehicle posture parameter; Wherein, determining the vehicle posture parameter at the target moment based at least on a Kalman filter algorithm, the vehicle posture parameter at a previous moment of the target moment, and the first predicted vehicle posture parameter comprises: The vehicle posture parameter at the target moment is determined based on a Kalman filter algorithm, the vehicle posture parameter at a moment before the target moment, and the first predicted vehicle posture parameter and the second predicted vehicle posture parameter.

2. The method according to claim 1, characterized in that The vehicle posture parameters include a vehicle rotation angle; The vehicle rotation angle includes one or more of the following: pitch angle; roll angle; yaw angle; angular velocity.

3. The method according to claim 2, characterized in that The machine learning model includes a feature extraction layer and at least one fully connected layer; wherein one fully connected layer is used to calculate a type of vehicle rotation angle; The step of inputting the AVM image into a trained machine learning model, and having the machine learning model calculate a first predicted vehicle posture parameter at the target time based on the AVM image, comprises: The AVM image is input into the trained machine learning model, the feature extraction layer performs feature extraction on the AVM image to obtain feature data corresponding to the AVM image, and each fully connected layer calculates various types of vehicle rotation angles based on the feature data.

4. The method according to claim 1, characterized in that: The method of determining the vehicle posture parameter at the target moment based on the Kalman filter algorithm, the vehicle posture parameter at the previous moment of the target moment, the first predicted vehicle posture parameter and the second predicted vehicle posture parameter comprises: Defining a state transfer equation of a Kalman filter algorithm based on an integral model of the sensor data; Based on the first predicted vehicle posture parameter and the second predicted vehicle posture parameter, two observation equations of a Kalman filter algorithm are defined respectively, and the two observation equations are combined into one observation equation by weighting; Based on the defined Kalman filter algorithm, the vehicle posture parameters at the previous moment of the target moment, and the first predicted vehicle posture parameters and the second predicted vehicle posture parameters, the vehicle posture parameters at the target moment are determined.

5. The method according to claim 1, characterized in that The machine learning model includes a residual network.

6. The method according to claim 1, characterized in that The method further comprises: Acquire an AVM image sample marked with a corresponding label; wherein the AVM image sample is an AVM image of a vehicle when it is traveling on a road section where the road condition has changed, and the label is a vehicle posture parameter; The machine learning model is supervisedly trained based on the AVM image samples.

7. A vehicle attitude angle determination device, characterized in that: The device comprises: A first acquisition unit is used to acquire the AVM image and sensor data at a target time collected during the driving process of the vehicle; An input unit, used for inputting the AVM image into a trained machine learning model, so that the machine learning model calculates a first predicted vehicle posture parameter at the target moment based on the AVM image; a calculation unit, configured to calculate a second predicted vehicle posture parameter at the target time based on the sensor data; A determination unit, used to determine the vehicle posture parameters at the target moment based at least on a Kalman filter algorithm, the vehicle posture parameters at the previous moment of the target moment, and the first predicted vehicle posture parameters; wherein, the determination of the vehicle posture parameters at the target moment based at least on a Kalman filter algorithm, the vehicle posture parameters at the previous moment of the target moment, and the first predicted vehicle posture parameters comprises: determining the vehicle posture parameters at the target moment based on a Kalman filter algorithm, the vehicle posture parameters at the previous moment of the target moment, the first predicted vehicle posture parameters and the second predicted vehicle posture parameters.

8. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 6 by calling the machine-readable instructions.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images