Vehicle positioning method, electronic equipment and vehicle

Abstract

The embodiment of the invention provides a vehicle positioning method, electronic equipment and a vehicle, and the method comprises the steps: collecting the state information of the vehicle, and the state information comprises the rotating speed data of a driving motor of the vehicle; and determining the positioning information of the vehicle according to the state information of the vehicle and the pose information of the vehicle. According to the embodiment of the invention, accurate positioning can be realized according to the state information and the pose information of the vehicle, the limitation of positioning based on information of a single sensor is effectively overcome, and more accurate positioning information is provided for the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of positioning technology, and in particular to a vehicle positioning method, an electronic device, a vehicle, a computer program product, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of autonomous driving technology, achieving precise vehicle positioning has become crucial.

[0003] However, current vehicle positioning technologies struggle to provide accurate positioning results. For example, a common method for vehicle positioning relies on vehicle speed, which is primarily determined by wheel speed data. However, wheel speed data can be inaccurate due to tire wear, changes in tire pressure, or different road conditions, leading to insufficient accuracy in vehicle positioning.

[0004] Therefore, how to provide more accurate vehicle location information has become an urgent problem to be solved. Summary of the Invention

[0005] In view of the above problems, a vehicle positioning method, electronic device, and vehicle are proposed to overcome or at least partially solve the above problems. The specific technical solution is as follows:

[0006] In a first aspect of the present invention, a vehicle positioning method is provided, the method comprising:

[0007] Collect vehicle status information, including the vehicle's drive motor speed data;

[0008] The vehicle's positioning information is determined based on the vehicle's status information and its pose information.

[0009] In a second aspect of the invention, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform any of the vehicle positioning methods described above.

[0010] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the vehicle positioning methods described above.

[0011] In another aspect of the present invention, a vehicle is also provided, which implements the vehicle positioning method described in any of the preceding claims.

[0012] Compared with related technologies, the embodiments of the present invention have at least the following advantages:

[0013] In this embodiment of the invention, vehicle status information is collected, which may include at least the vehicle's drive motor speed data. Based on the vehicle's status information and pose information, the vehicle's positioning information is determined, achieving precise vehicle positioning. This embodiment of the invention can achieve precise positioning based on the vehicle's status and pose information, effectively overcoming the limitations of positioning based on single-source information from a single sensor, and providing more accurate positioning information for the vehicle. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0015] Figure 1 This is a flowchart illustrating the steps of a vehicle positioning method provided in an embodiment of the present invention;

[0016] Figure 2 This is a system framework diagram of vehicle positioning based on multi-source information fusion provided in an embodiment of the present invention;

[0017] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0019] Given that current vehicle-based single-sensor positioning often fails to provide ideal positioning results in complex environments, and that the fusion method of multi-source information directly affects the accuracy and stability of positioning, this invention proposes a vehicle positioning method based on multi-source information fusion. Specifically, by optimizing the multi-source information fusion strategy, the limitations of a single sensor are effectively overcome, providing more accurate and stable positioning information for the vehicle.

[0020] Reference Figure 1 The above is a flowchart of the steps of a vehicle positioning method provided in an embodiment of the present invention, as follows: Figure 1 As shown, the method may specifically include the following steps:

[0021] Step 101: Collect vehicle status information, including the vehicle's drive motor speed data.

[0022] In specific implementations, embodiments of the present invention can be applied to vehicle-mounted terminals. Of course, embodiments of the present invention can also be implemented using other devices, and the present invention does not impose any limitations on this.

[0023] In this embodiment of the invention, when locating a vehicle, status information collected by at least one sensor of the vehicle can be obtained. The status information may include the speed data of the vehicle's drive motor. Specifically, the speed data of the drive motor may include the speed data of the left and right motors.

[0024] Step 102: Determine the vehicle's positioning information based on the vehicle's status information and its pose information.

[0025] The vehicle's pose information may include information such as the position and orientation of targets within the vehicle.

[0026] In this embodiment of the invention, the vehicle can be located based on its state information. However, since precise positioning cannot be achieved based on state information collected from a single sensor, in this embodiment of the invention, the vehicle's positioning information can be determined jointly based on the vehicle's pose information and coarse positioning information.

[0027] In this embodiment of the invention, the vehicle's positioning information (precise positioning information) is determined by fusing state information and pose information. In this way, the limitations of positioning based on information collected by a single sensor can be overcome, and accurate positioning of the vehicle can be achieved.

[0028] In the above vehicle positioning method, vehicle status information is collected, which may include at least the vehicle's drive motor speed data. Based on the vehicle's status information and pose information, the vehicle's positioning information is determined, achieving precise vehicle positioning. This embodiment of the invention can achieve precise positioning based on the vehicle's status and pose information, effectively overcoming the limitations of positioning based on single-source information from a single sensor, and providing more accurate positioning information for the vehicle.

[0029] In one embodiment of the present invention, step 102, determining the vehicle's positioning information based on the vehicle's state information and the vehicle's pose information, includes:

[0030] The coarse positioning information of the vehicle is determined based on the vehicle's status information;

[0031] The vehicle's positioning information is determined based on the vehicle's status information and its pose information.

[0032] In this embodiment of the invention, coarse positioning information of the vehicle can be determined based on the vehicle's state information. While the vehicle can be located based on this coarse positioning information, the presence of bias and noise in the coarse positioning information affects the accuracy of vehicle positioning. Therefore, in this embodiment of the invention, the vehicle's positioning information can be jointly determined based on the vehicle's pose information and the coarse positioning information to improve the accuracy of vehicle positioning.

[0033] In one embodiment of the present invention, the method further includes:

[0034] Visual information is collected simultaneously with the vehicle's state information, and the vehicle's pose information is determined based on the visual information and the vehicle's state information.

[0035] The visual information refers to the images (camera images) captured by the vehicle's visual sensors (such as cameras). The visual information can typically include multiple targets, such as vehicles, pedestrians, and traffic signs.

[0036] In this embodiment of the invention, visual information is collected simultaneously with the state information, and the vehicle's pose information is determined based on the visual information. The pose information may include information such as the position and orientation of the target in the vehicle. The pose information is obtained by matching the feature points of the target in the visual information, which has high accuracy. However, the accuracy of vehicle positioning is affected when the light intensity of the visual information is poor.

[0037] Therefore, the embodiments of the present invention determine the vehicle's positioning information by fusing coarse positioning information and pose information, thereby overcoming the limitations of positioning based on information collected by a single sensor and achieving accurate positioning of the vehicle.

[0038] In one embodiment of the present invention, the coarse positioning information of the vehicle is determined based on the vehicle's state information:

[0039] The drive motor speed data is input into the machine learning model to obtain the vehicle speed data and steering angle data output by the machine learning model.

[0040] The coarse positioning information of the vehicle is determined based on the speed data and the turning angle data.

[0041] In this embodiment of the invention, a trained machine learning model can be pre-deployed in the vehicle. The machine learning model can be used to predict the vehicle's speed and turning angle data based on the drive motor speed data, and then determine the vehicle's coarse positioning information based on the speed and turning angle data.

[0042] In one embodiment of the present invention, the machine learning model can be trained and generated in the following manner:

[0043] Obtain the first dataset of the vehicle; the first dataset includes a first training set, a first validation set, and a first test set; the first dataset includes pre-collected sample left and right motor speed data, sample speed data, and sample turning angle data of the vehicle;

[0044] The machine learning model to be trained is trained based on the first training set;

[0045] The trained machine learning model is adjusted based on the first validation set;

[0046] The adjusted machine learning model is evaluated based on the first test set;

[0047] After the adjusted machine learning model passes the evaluation, the trained machine learning model is obtained.

[0048] In this embodiment of the invention, sample rotation speed data of the left and right motors of the vehicle are collected in advance, and sample speed data and sample turning angle data of the vehicle are recorded simultaneously to construct a dataset (first dataset), wherein the dataset may include a training set (first training set), a validation set (first validation set) and a test set (first test set).

[0049] The training set is input into the machine learning model for training. A validation set is used to evaluate the model's performance and adjust its parameters, resulting in a trained and optimized machine learning model. A test set is then used to evaluate the performance of this optimized model. Once the machine learning model passes the evaluation, the trained model is complete. If the evaluation fails, the model can be further trained, optimized, and evaluated using the same dataset until a model capable of accurate predictions is obtained. The trained machine learning model can then be deployed on in-vehicle terminals to predict vehicle speed and turning angle data.

[0050] Machine learning models include, but are not limited to, regression models, such as linear regression, decision trees, random forests, or neural networks.

[0051] In one embodiment of the present invention, the vehicle's state information further includes the vehicle's acceleration data and angular velocity data; determining the vehicle's coarse positioning information based on the velocity data and the turning angle data may include:

[0052] Perform a pre-integration operation on the acceleration data and the angular velocity data to obtain relative velocity change data and relative angle change data;

[0053] The speed data and the rotation angle data are used as state variables;

[0054] The relative velocity change data and the relative angle change data are used as observations;

[0055] Kalman filtering is performed on the state variables and the observations to obtain coarse positioning information for the vehicle; wherein, the coarse positioning information includes target speed data and target turning angle data.

[0056] In this embodiment of the invention, the state information may include acceleration data and angular velocity data of the vehicle's inertial measurement unit (IMU).

[0057] The coarse positioning information can include target speed data and target turning angle data. Based on these data, the vehicle's pose information can be determined, and then vehicle positioning can be performed to obtain the vehicle's location information. However, the vehicle pose information determined from the target speed and turning angle data contains bias and noise, which can affect the accuracy of vehicle positioning.

[0058] In this embodiment of the invention, the rotational speed data of the left and right motors, the acceleration data and angular velocity data of the IMU, and the visual information of the camera are collected synchronously and their timestamps are aligned. The collected rotational speed data of the left and right motors is input into a trained machine learning model for prediction to obtain the vehicle's speed and turning angle data. The collected acceleration and angular velocity data of the IMU are pre-integrated to estimate the relative velocity change data and relative angle change data between two key frames (time points). The two key frames are taken as two adjacent frames of motor rotational speed data, because the IMU's frame rate is significantly higher than the motor rotational speed frame rate. Since the vehicle's speed and turning angle data in this embodiment of the invention are obtained by inputting the motor rotational speed data into the machine learning model for prediction, their frequencies are the same.

[0059] In related technical solutions, traditional vehicle speed estimation methods mainly rely on wheel speed data, or motor speed and transmission ratio. However, this method has some obvious drawbacks. On the one hand, wheel speed data may be inaccurate due to differences in tire wear, air pressure changes, or road conditions. On the other hand, simple conversion of motor speed and transmission ratio may also lead to inaccurate vehicle speed estimation due to factors such as the nonlinear characteristics of the transmission system and friction losses. To improve the accuracy of estimation, this embodiment of the invention uses the rotational speed data of the left and right motors and uses a machine learning model to predict wheel speed data (speed data) and steering angle data. The motor speed data can more accurately reflect the actual motion state of the vehicle and effectively reduce motion errors caused by tire slippage.

[0060] It should be noted that the inertial measurement unit (IMU) is rigidly connected to the vehicle. A rigid connection means that the two objects are not allowed to move or deform relative to each other. Therefore, the relative velocity change data and relative angle change data of the inertial measurement unit can be equated with the relative velocity change data and relative angle change data of the vehicle.

[0061] Then, the vehicle's speed and turning angle data can be used as state variables, and the vehicle's relative speed change data and relative angle change data can be used as observations. Kalman filtering can be performed based on the state variables and observations to obtain the vehicle's target speed data and target turning angle data, i.e., the vehicle's coarse positioning information.

[0062] In one embodiment of the present invention, the acceleration data and the angular velocity data include a bias b and noise n; before performing a pre-integration operation on the acceleration data and the angular velocity data to obtain relative velocity change data and relative angle change data, the method further includes:

[0063] Obtain the bias and noise corresponding to the angular velocity data;

[0064] The angular velocity data is processed based on the bias and noise corresponding to the angular velocity data to obtain the processed angular velocity data.

[0065] In addition, the bias and noise corresponding to the acceleration data are obtained;

[0066] The acceleration data is processed based on the bias and noise corresponding to the acceleration data to obtain the processed acceleration data.

[0067] In some specific embodiments, the acceleration data and the angular velocity data can be processed according to the inertial measurement unit model to obtain the processed acceleration data and the angular velocity data;

[0068] The formulas for the inertial measurement unit model may include an angular velocity measurement model and an acceleration measurement model:

[0069] The angular velocity measurement model can be:

[0070]

[0071] The acceleration measurement model can be:

[0072]

[0073] in, This represents the angular velocity data before processing; This represents the acceleration data before processing; ω b This represents the processed angular velocity data; a b The processed acceleration data; b ω Indicates the bias of angular velocity data; n ω Noise in angular velocity data; q bw Represents a rotation quaternion, from the world coordinate system to the inertial measurement unit (IMU) coordinate system; g wb represents the component of gravitational acceleration in the world coordinate system; a Indicates the bias of acceleration data; n a This indicates noise in the acceleration data; the superscript w indicates the world coordinate system, and the superscript b indicates the inertial measurement unit coordinate system (IMU coordinate system).

[0074] The acceleration and angular velocity data before processing include the measured values ​​of bias b and noise n. Therefore, before performing pre-integration on the acceleration and angular velocity data, the bias b and noise n can be eliminated using the inertial measurement unit model to obtain processed acceleration and angular velocity data with bias b and noise n eliminated. The processed acceleration and angular velocity data are closer to the true values. Therefore, vehicle positioning based on the processed acceleration and angular velocity data can improve the accuracy of vehicle positioning.

[0075] In one embodiment of the present invention, a pre-integration operation is performed on the acceleration data and the angular velocity data to obtain relative velocity change data and relative angle change data, including:

[0076] The processed acceleration data is pre-integrated to obtain relative velocity change data;

[0077] The processed angular velocity data is pre-integrated to obtain relative angle change data;

[0078] In some specific embodiments, the processed acceleration data and angular velocity data can be pre-integrated using the following formulas to obtain the relative velocity change data and relative angle change data between time i and time j:

[0079]

[0080] in, This represents data on relative velocity changes; This represents data on relative angle changes.

[0081] In one embodiment of the present invention, the method may further include:

[0082] Based on the relative velocity change data and the relative angle change data, the relative position change data is obtained.

[0083] In some specific embodiments, the relative velocity change data and relative angle change data can be pre-integrated using the following formula to obtain the relative position change data between time i and time j:

[0084]

[0085] in, This represents data on relative positional changes.

[0086] In this embodiment of the invention, the processed IMU data (processed acceleration data and angular velocity data) within two keyframes are pre-integrated to obtain the measurement constraints of the IMU between times i and j, i.e., the pre-integrated quantity, which may specifically include position change data. Relative velocity change data and relative angle change data

[0087] In one embodiment of the present invention, the relative velocity change data and the relative angle change data are used as observations, including:

[0088] The relative velocity change data and the relative angle change data are used as observations;

[0089] Alternatively, the location change data can be used as an observation.

[0090] Among them, location change data Based on relative velocity change data and relative angle change data The result is obtained by integration. In this embodiment of the invention, when performing Kalman filtering, the relative velocity change data can be... and relative angle change data As an observation, location change data can also be selected. As an observation, the embodiments of the present invention need not impose any limitations on this.

[0091] The Kalman filtering process can include two steps: prediction and update. In one embodiment of the invention, Kalman filtering is performed based on the state variables and the observations to obtain coarse positioning information of the vehicle, including:

[0092] Determine the state variables for the next time step based on the current state variables and the state transition matrix.

[0093] Based on the noise covariance matrix, state transition matrix, and process noise covariance matrix at the current moment, determine the noise covariance matrix at the next moment.

[0094] The Kalman gain matrix is ​​determined based on the noise covariance matrix, observation matrix, and observation noise covariance matrix at the next time step.

[0095] The coarse positioning information of the vehicle is determined based on the state quantity at the next time step, the Kalman gain matrix, the observation quantity, and the observation matrix.

[0096] In some specific embodiments, the present invention can obtain coarse vehicle positioning information by performing Kalman filtering based on state variables and observations using the following formula:

[0097] The prediction process may include state prediction and the transformation of the noise covariance matrix:

[0098] State prediction:

[0099] Transition of the noise covariance matrix: P(k+1|k)=ΦP(k|k)Φ T +Q

[0100] The update process may include Kalman gain matrix and state update:

[0101] Kalman gain matrix: K(k+1)=P(k+1|k)H T [HP(k+1|k)H T +R] -1

[0102] Status Update:

[0103] in, This represents the predicted state quantity at time k+1; Let represent the state variables at time k; Φ represent the state transition matrix, describing how the system state changes over time; P(k+1|k) represent the noise covariance matrix of the predicted state variables at time k+1; P(k|k) represent the noise covariance matrix of the state variables at time k; Q represent the process noise covariance matrix, describing the influence of system noise on the state; T represents the transpose; K(k+1) represents the Kalman gain matrix; H represents the observation matrix, describing the relationship between the observations and the state; R represents the observation noise covariance matrix, describing the influence of observation noise on the data; -1 represents the inverse matrix. Y(k+1) represents coarse positioning information, including target velocity data and target rotation angle data; Y(k+1) represents the observed data, i.e., relative velocity change data. and relative angle change data Or, relative position change data

[0104] In one embodiment of the present invention, after performing Kalman filtering based on the state variables and the observations to obtain coarse positioning information of the vehicle, the method further includes:

[0105] Update the noise covariance matrix.

[0106] In one embodiment of the present invention, updating the noise covariance matrix includes:

[0107] The noise covariance matrix is ​​updated based on the unit matrix, Kalman gain matrix, observation matrix, and noise covariance matrix at the next time step.

[0108] In some specific embodiments, the noise covariance matrix can be updated using the following formula:

[0109] P(k+1|k+1)=[I n -K(k+1)H]P(k+1|k)

[0110] Where P(k+1|k+1) represents the updated noise covariance matrix; I n Represents a unit matrix.

[0111] In this embodiment of the invention, after obtaining coarse localization information according to the formula corresponding to Kalman filtering, the noise covariance matrix in the Kalman filtering formula can be updated, and the updated noise covariance matrix can be used for the next Kalman filtering process.

[0112] In one embodiment of the present invention, the camera and the vehicle are rigidly connected; the pose change information of the camera can be equivalent to the pose change information of the vehicle; visual information is acquired simultaneously with the state information of the vehicle, and the pose information of the vehicle is determined based on the visual information and the state information of the vehicle, including:

[0113] The visual information is input into a deep neural network model to obtain the target category and detection bounding box output by the deep neural network model.

[0114] Feature points of the target are extracted within the detection bounding box according to the target category; the feature points include key points and descriptors; the key points are the target state information of the feature points in the visual information; the descriptors are vectors describing the information of pixels surrounding the key points; the target state information includes at least the position, orientation, and size of the target;

[0115] The feature points of adjacent visual information are matched, and the pose change information of the camera is calculated based on the matched feature points.

[0116] The pose change information of the camera is used as the pose information of the vehicle.

[0117] In one embodiment of the present invention, the deep neural network model is trained and generated in the following manner:

[0118] Obtain a second dataset for the vehicle; the second dataset includes a second training set, a second validation set, and a second test set; the second dataset includes pre-collected sample visual information of the vehicle during its driving process, the sample visual information including labeled targets; the targets include at least vehicles, pedestrians, and traffic signs;

[0119] The deep neural network model to be trained is trained based on the second training set;

[0120] The trained deep neural network model is adjusted based on the second validation set;

[0121] The adjusted deep neural network model is evaluated based on the second test set;

[0122] After the adjusted deep neural network model passes the evaluation, the trained deep neural network model is obtained.

[0123] In this embodiment of the invention, sample visual information of vehicles during their driving process is collected in advance. The targets in the sample visual information have been labeled with their corresponding categories, such as vehicles, pedestrians, and traffic signs. This constructs a dataset (second dataset), which may include a training set (second training set), a validation set (second validation set), and a test set (second test set).

[0124] The training set is input into the deep neural network model for training. A validation set is used to evaluate the model's performance and adjust its parameters, resulting in a trained and optimized deep neural network model. A test set is then used to evaluate the performance of this optimized model. Once the deep neural network model passes the evaluation, the trained model is complete. If the evaluation fails, the model can be further trained, optimized, and evaluated using the same dataset until a deep neural network model capable of accurate predictions is obtained. The trained deep neural network model can then be deployed on in-vehicle terminals to predict vehicle pose information.

[0125] Deep neural network models include, but are not limited to, Faster R-CNN and YOLO models.

[0126] In this embodiment of the invention, the acquired visual information is preprocessed and then input into a trained deep neural network model for target detection, resulting in the target category and detection bounding box. Then, feature points of the target are extracted within the detection bounding box. Since different target categories have different contour features, this embodiment of the invention can extract feature points of the target within the detection bounding box based on the target category. Each feature point consists of keypoints and descriptors. Keypoints refer to the target state information of the feature point in the image, such as its position, orientation, and size. A descriptor is a vector that describes the information of the pixels surrounding the keypoint.

[0127] In some embodiments, feature points include ORB (Oriented Fast and Rotated BRIEF) features. ORB feature extraction involves two steps: extracting Fast corner points and calculating the BRIEF descriptor for each keypoint. Feature points in adjacent visual information are matched using a feature matching algorithm. Based on the matched feature points, the camera's pose change information is calculated. Specifically, feature points detected in the current visual information are matched with known feature points in previous visual information, and the camera's pose change information is calculated based on the matched feature point pairs. In some specific embodiments, the feature matching algorithm includes, but is not limited to, brute-force matching algorithms and fast approximate nearest neighbor algorithms.

[0128] In related technical solutions, feature point extraction and matching of the target within the entire visual information can lead to low efficiency. However, the embodiments of the present invention perform feature point extraction and matching within the detection bounding box of the target in the visual information. The detection bounding box is obtained by a neural network model for target detection in the visual information. Since the embodiments of the present invention perform feature point extraction and matching within the detection bounding box rather than the entire visual information, the processing area is reduced, thereby reducing the amount of computation and making the feature point extraction and matching more efficient. At the same time, focusing only on the detection bounding box of the target can eliminate background and noise in the visual information, which helps to extract more accurate feature points and thus improve the accuracy of vehicle positioning.

[0129] In one embodiment of the present invention, the feature points of adjacent visual information are matched, and pose change information is calculated based on the matched feature points, including:

[0130] Determine the camera type of the camera that acquires the visual information;

[0131] The feature points of adjacent visual information are matched according to the camera type, and the pose change information of the camera is calculated based on the matched feature points.

[0132] In specific implementations, cameras have corresponding camera types, such as monocular cameras, binocular cameras, RGB-D cameras, 2D cameras, and 3D cameras. In this embodiment of the invention, a corresponding pose estimation algorithm can be selected according to the camera type. After matching the feature points of adjacent visual information to obtain matching feature points, the pose change information of the camera is calculated based on the matching feature points according to the pose estimation algorithm corresponding to the camera type.

[0133] For example, when calculating the pose change information of the camera, if the camera used is a monocular camera, the pose estimation algorithm of the epipolar geometry method is used; if the camera used is a stereo camera or an RGB-D camera, the pose estimation algorithm of ICP (Iterative Closest Point) is used; if there is one 2D camera and one 3D camera, the pose estimation algorithm of PnP (Perspective-n-Point) is used.

[0134] In one embodiment of the present invention, determining the vehicle's positioning information based on the vehicle's state information and the vehicle's pose information includes:

[0135] Obtain real-time environmental data for the vehicle;

[0136] Weighting factors are assigned to the state information and the pose information based on the real-time environmental data.

[0137] The vehicle's positioning information is determined based on the state information, the pose information, the weighting factor of the state information, and the weighting factor of the pose information.

[0138] In this embodiment of the invention, after obtaining the vehicle's state information and pose information, coarse positioning information can be determined based on the state information. Then, the coarse positioning information and pose information are weighted and fused to obtain the vehicle's fine positioning information (positioning information). The key to the weighted fusion is to dynamically adjust the weight factors of the two information sources, coarse positioning information and pose information, when generating positioning information based on the vehicle's real-time environmental data.

[0139] In one embodiment of the present invention, assigning weighting factors to the state information and the pose information based on the real-time environmental data includes:

[0140] The real-time light intensity is determined based on the real-time environmental data.

[0141] When the real-time illumination intensity is higher than or equal to the preset illumination intensity, a first weighting factor is assigned to the state information and a second weighting factor is assigned to the pose information; wherein, the first weighting factor is less than the second weighting factor;

[0142] When the real-time illumination intensity is lower than the preset illumination intensity, a third weighting factor is assigned to the state information and a fourth weighting factor is assigned to the pose information; wherein the third weighting factor is greater than the fourth weighting factor.

[0143] In this embodiment of the invention, the real-time illumination intensity of the vehicle can be determined based on the vehicle's real-time environmental data. Under good lighting conditions, i.e., when the real-time illumination intensity is higher than or equal to a preset illumination intensity, a larger weighting factor (second weighting factor) can be assigned to the pose information, and a smaller weighting factor (first weighting factor) can be assigned to the coarse positioning information. Under poor lighting conditions, i.e., when the real-time illumination intensity is lower than the preset illumination intensity, a smaller weighting factor (fourth weighting factor) can be assigned to the pose information, and a larger weighting factor (third weighting factor) can be assigned to the coarse positioning information.

[0144] In practical implementation, the tightly coupled strategy is a fusion method that highly relies on the data and states between sensors. While it can significantly improve the accuracy and reliability of the fusion results, tightly coupled measurements involve high computational complexity and complex system design. In related technical solutions, a tightly coupled strategy is applied to data collected by all vehicle sensors. If any module fails, it will affect the stability and accuracy of the entire positioning system. However, in this embodiment of the invention, a dynamic weighted fusion mode is selected in the final fusion stage. This loosely coupled design ensures that even if one module (e.g., the coarse positioning information processing module) fails, another module (e.g., the pose information processing module) can still provide positioning information briefly, ensuring the continuity of positioning information and providing good compatibility with different types of positioning modules. Specifically, in the final fusion stage of this embodiment, IMU data and motor speed data are first fused to obtain the vehicle's coarse positioning information. Then, dynamic weighted fusion is performed with the pose information obtained from the visual sensor. Different weighting factors are dynamically selected according to different real-time environments to finally obtain the vehicle's fine positioning information.

[0145] To enable those skilled in the art to better understand the embodiments of the present invention, a complete example is described below. (Refer to...) Figure 2 This is a system framework diagram of vehicle positioning based on multi-source information fusion provided in an embodiment of the present invention. The specific steps based on this system framework diagram may include:

[0146] S01: Pre-collect left and right motor speed data, and simultaneously record vehicle speed data and steering angle data to build a dataset, including training set, validation set and test set.

[0147] S02: Input the training set of S01 into the machine learning model for training, use the validation set of S01 to evaluate the performance of the machine learning model and adjust the parameters to obtain the trained and optimized machine learning model, and use the test set of S01 to evaluate the performance of the trained and optimized machine learning model.

[0148] S03: Collect visual information during vehicle movement in advance, label the targets in the visual information, such as vehicles, pedestrians, traffic signs, etc., and divide the dataset into training set, validation set and test set.

[0149] S04: Input the training set of S03 into the deep neural network model for training, use the validation set of S03 to evaluate the performance of the deep neural network model and adjust the parameters to obtain the trained and optimized deep neural network model, and use the test set of S03 to evaluate the performance of the trained and optimized deep neural network model.

[0150] S05: Synchronously acquire left and right motor speed data, IMU data (IMU acceleration and angular velocity data), and camera images, and align the timestamps.

[0151] S06: Input the left and right motor speed data collected in S05 into the machine learning model trained and optimized in S02 for prediction, and obtain the vehicle speed data and steering angle data.

[0152] S07: The acceleration and angular velocity data of the IMU acquired in S05 are pre-integrated to estimate the relative velocity and relative angle changes between two keyframes.

[0153] In this embodiment, the two key frames are taken as adjacent frames of motor speed, because the IMU's frame rate is significantly higher than the motor speed acquisition frequency. Since the vehicle's speed and angle data are predicted from the motor speed, their frequencies are the same. In this embodiment, the IMU and the vehicle are rigidly connected, so the IMU's relative speed change data and relative angle change data are equivalent to the vehicle's relative speed change data and relative angle change data.

[0154] S08: Using the vehicle speed and turning angle data from S06 as state variables, and the vehicle relative speed change data and relative angle change data from S07 as observation variables, the data are fused using Kalman filtering to obtain the vehicle's coarse positioning information.

[0155] S09: After preprocessing the visual information obtained in S05, it is input into the deep neural network model trained and optimized in S04 to perform target detection, and obtain the target category and detection bounding box (target box).

[0156] S10: Extract feature points of the target within the detection bounding box.

[0157] A feature point consists of keypoints and descriptors. A keypoint refers to the feature point's location, orientation, size, and other information within the image. A descriptor is a vector that describes information about the pixels surrounding the keypoint.

[0158] Among them, feature points can include ORB features. Extracting ORB features is divided into two steps: extracting FAST corner points and calculating the BRIEF descriptor for each key point.

[0159] S11: The feature points detected in the current visual information are matched with the known feature points in the previous visual information using a feature matching algorithm, and the pose change information of the camera is calculated based on the matched feature points.

[0160] When calculating camera pose change information, the epipolar geometry method is used if the camera is a monocular camera, and the ICP method is used if the camera is a stereo camera or an RGB-D camera. If the camera is a 2D camera and a 3D camera, the PnP method is used.

[0161] Since the camera and the vehicle are rigidly connected, the camera's pose change information can be equated with the vehicle's pose change information.

[0162] S12: The coarse positioning information from S08 and the vehicle pose information from S11 are weighted and fused to obtain the fine positioning information of the vehicle.

[0163] The key to weighted fusion lies in dynamically adjusting the weight factors of the two information sources based on real-time ambient lighting conditions. Under good lighting conditions, a larger weight factor is assigned to the vehicle pose information of S11. Under poor lighting conditions, a larger weight factor is assigned to the coarse positioning information of S08.

[0164] As can be seen, by dynamically adjusting the weighting factors of the two information sources, pose information and coarse positioning information, the embodiments of the present invention can achieve a vehicle positioning system with stronger robustness and better adaptability.

[0165] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0166] This invention also provides an electronic device, such as... Figure 3As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0167] Memory 303 is used to store computer programs;

[0168] When the processor 301 executes the program stored in the memory 303, it implements any of the vehicle positioning methods described in the above embodiments:

[0169] Collect vehicle status information, including the vehicle's drive motor speed data;

[0170] The vehicle's positioning information is determined based on the vehicle's status information and its pose information.

[0171] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0172] The communication interface is used for communication between the aforementioned terminal and other devices.

[0173] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0174] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0175] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the vehicle positioning methods described in the above embodiments.

[0176] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the vehicle positioning methods described in the above embodiments.

[0177] In another embodiment of the present invention, a vehicle is also provided, which implements any of the vehicle positioning methods described above.

[0178] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0179] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0180] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0181] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A vehicle positioning method characterized by comprising: The method includes: Collect vehicle status information, including the vehicle's drive motor speed data; The vehicle's positioning information is determined based on the vehicle's status information and its pose information.

2. The method of claim 1, wherein, Determining the vehicle's positioning information based on the vehicle's state information and pose information includes: The coarse positioning information of the vehicle is determined based on the vehicle's status information; The vehicle's positioning information is determined based on the vehicle's status information and its pose information.

3. The method of claim 1, wherein, The method further includes: Visual information is collected simultaneously with the vehicle's state information, and the vehicle's pose information is determined based on the visual information and the vehicle's state information.

4. The method of claim 2, wherein, Determining the coarse positioning information of the vehicle based on the vehicle's status information includes: The drive motor speed data is input into the machine learning model to obtain the vehicle speed data and steering angle data output by the machine learning model. The coarse positioning information of the vehicle is determined based on the speed data and the turning angle data.

5. The method of claim 4, wherein, The vehicle's status information also includes the vehicle's acceleration data and angular velocity data; The coarse positioning information of the vehicle is determined based on the speed data and the turning angle data, including: Perform a pre-integration operation on the acceleration data and the angular velocity data to obtain relative velocity change data and relative angle change data; The speed data and the rotation angle data are used as state variables; The relative velocity change data and the relative angle change data are used as observations; Kalman filtering is performed on the state variables and the observations to obtain coarse positioning information for the vehicle; wherein, the coarse positioning information includes target speed data and target turning angle data.

6. The method of claim 5, wherein, The angular velocity data includes bias and noise; before performing a pre-integration operation on the acceleration data and the angular velocity data to obtain the relative velocity change data and the relative angle change data, the method further includes: Obtain the bias and noise corresponding to the angular velocity data; The angular velocity data is processed based on the bias and noise corresponding to the angular velocity data to obtain the processed angular velocity data.

7. The method of claim 5, wherein, The acceleration data includes bias and noise; before performing a pre-integration operation on the acceleration data and the angular velocity data to obtain the relative velocity change data and the relative angle change data, the method further includes: Obtain the bias and noise corresponding to the acceleration data; The acceleration data is processed based on the bias and noise corresponding to the acceleration data to obtain the processed acceleration data.

8. The method according to claim 5 or 6 or 7, characterized in that, Perform a pre-integration operation on the acceleration data and the angular velocity data to obtain relative velocity change data and relative angle change data, including: The processed acceleration data is pre-integrated to obtain relative velocity change data; The processed angular velocity data is pre-integrated to obtain relative angle change data.

9. The method of claim 8, wherein, The method further includes: Based on the relative velocity change data and the relative angle change data, the relative position change data is obtained.

10. The method of claim 9, wherein, Using the relative velocity change data and the relative angle change data as observations, including: The relative speed change data and the relative angle change data are taken as observations; Or, the position change data is taken as an observation.

11. The method of claim 5, wherein, Performing Kalman filtering processing according to the state quantity and the observation quantity to obtain coarse positioning information of the vehicle, including: Determining a state quantity at a next time according to a state quantity at a current time and a state transition matrix; Determining a noise covariance matrix at the next time according to a noise covariance matrix at the current time, the state transition matrix and a process noise covariance matrix; Determining a Kalman gain matrix according to the noise covariance matrix at the next time, an observation matrix and an observation noise covariance matrix; Determining the coarse positioning information of the vehicle according to the state quantity at the next time, the Kalman gain matrix, the observation quantity and the observation matrix.

12. The method of claim 11, wherein, After performing Kalman filtering processing according to the state quantity and the observation quantity to obtain coarse positioning information of the vehicle, the method further includes: Updating the noise covariance matrix.

13. The method of claim 12, wherein, Updating the noise covariance matrix, including: Updating the noise covariance matrix according to a unit matrix, the Kalman gain matrix, the observation matrix and the noise covariance matrix at the next time.

14. The method of claim 2, wherein, Collecting visual information at the same time as the state information of the vehicle, and determining pose information of the vehicle according to the visual information and the state information of the vehicle, including: Inputting the visual information into a deep neural network model to obtain a category of a target and a detection bounding box output by the deep neural network model; Extracting feature points of the target in the detection bounding box according to the category of the target; the feature points include key points and descriptors; the key points are target state information of the feature points in the visual information; the descriptors are vectors describing information of pixels around the key points; the target state information at least includes a position, an orientation and a size of the target; Matching the feature points of adjacent visual information to calculate pose change information according to the matched feature points; Taking the pose change information as the pose information of the vehicle.

15. The method of claim 14, wherein, Matching the feature points of adjacent visual information according to the camera type to calculate pose change information according to the matched feature points. Determining positioning information of the vehicle according to the state information of the vehicle and the pose information of the vehicle, including: Obtaining real-time environmental data of the vehicle; 16. The method of claim 1, wherein, Assigning weight factors to the state information and the pose information according to the real-time environmental data; Determining the positioning information of the vehicle according to the state information, the pose information, the weight factor of the state information and the weight factor of the pose information. Assigning weight factors to the state information and the pose information according to the real-time environmental data, including: Determining a real-time light intensity according to the real-time environmental data; 17. The method of claim 16, wherein, When the real-time light intensity is higher than or equal to a preset light intensity, assigning a first weight factor to the state information and a second weight factor to the pose information; wherein the first weight factor is smaller than the second weight factor; ​ ​ When the real-time light intensity is lower than the preset light intensity, a third weight factor is assigned to the state information and a fourth weight factor is assigned to the pose information; wherein the third weight factor is greater than the fourth weight factor.

18. The method of claim 4, wherein, The machine learning model is trained by the following way: A first data set of the vehicle is acquired; the first data set includes a first training set, a first validation set and a first test set; the first data set includes pre-acquired sample left and right motor speed data, sample speed data and sample turning angle data of the vehicle; The machine learning model to be trained is trained according to the first training set; The trained machine learning model is adjusted according to the first validation set; The adjusted machine learning model is evaluated according to the first test set; After the evaluation of the adjusted machine learning model passes, the trained machine learning model is obtained.

19. The method of claim 14, wherein, The deep neural network model is trained by the following way: A second data set of the vehicle is acquired; the second data set includes a second training set, a second validation set and a second test set; the second data set includes pre-acquired sample visual information of the vehicle in the vehicle driving process, and the sample visual information includes labeled targets; the targets at least include vehicles, pedestrians and traffic signs; The deep neural network model to be trained is trained according to the second training set; The trained deep neural network model is adjusted according to the second validation set; The adjusted deep neural network model is evaluated according to the second test set; After the evaluation of the adjusted deep neural network model passes, the trained deep neural network model is obtained.

20. An electronic device, comprising: The electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; The memory is used for storing a computer program; The processor is used for executing the program stored on the memory to realize the method steps of any one of claims 1-19.

21. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the method of any one of claims 1-19.

22. A vehicle characterized by The vehicle comprises the electronic device of claim 20. The vehicle comprises the electronic device of claim 20.

Images