Multi-sensor fusion positioning method and device and electronic equipment

Abstract

The embodiment of the invention discloses a multi-sensor fusion positioning method and device and electronic equipment, and the method comprises the steps: obtaining multi-sensor data, inputting the multi-sensor data into a pre-trained multi-sensor fusion model, carrying out the processing, obtaining fusion positioning attribute information, determining a target positioning attribute according to the fusion positioning attribute information, and carrying out the positioning according to the target positioning attribute. Wherein the multi-sensor data comprises original information and auxiliary information of a plurality of sensors, and the auxiliary information is used for representing differences between different sensor data corresponding to the same motion attribute, so that feature fusion can be performed through the original information and the auxiliary information of the plurality of sensors; and the accuracy of fusion positioning is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a multi-sensor fusion positioning method, device and electronic equipment. Background Art

[0002] In positioning tasks, multiple sensors are often used to observe the movement of the body, including position, speed, attitude and other information. The sensors involved often include: GNSS (Global Navigation Satellite System), accelerometer, gyroscope, magnetometer, Lidar (radar), camera, odometer, etc. These different sensor observations need to be fused to obtain accurate estimates of position, speed and attitude. Currently, the uncertainty of sensor observation is usually used to achieve multi-sensor fusion, and the uncertainty of sensor observation often changes with scenes, working conditions, etc., which in turn affects the accuracy of sensor fusion positioning. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a multi-sensor fusion positioning method, device and electronic device to perform feature fusion through original information and auxiliary information of multiple sensors, thereby improving the accuracy of fusion positioning.

[0004] In a first aspect, an embodiment of the present invention provides a multi-sensor fusion positioning method, the method comprising:

[0005] Acquire multi-sensor data, where the multi-sensor data includes original information and auxiliary information of multiple sensors, where the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute;

[0006] Inputting the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain fused positioning attribute information;

[0007] The target positioning attribute is determined according to the fused positioning attribute information.

[0008] Optionally, the fused positioning attribute information includes the uncertainty of each of the sensors;

[0009] Determining the target positioning attribute according to the fused positioning attribute information includes:

[0010] The original information of each sensor is subjected to feature fusion processing based on the fused positioning attribute information to obtain the target positioning attribute.

[0011] Optionally, the uncertainty is characterized by a covariance or variance of a motion attribute of a corresponding sensor.

[0012] Optionally, performing feature fusion processing on the original information of each sensor and the fused positioning attribute information to obtain the target positioning attribute includes:

[0013] Adjusting the fusion weight of each sensor according to the uncertainty of each sensor;

[0014] Data of each sensor is fused according to the fusion weight of each sensor to obtain the target positioning attribute.

[0015] Optionally, the fused positioning attribute information includes a fusion weight of each of the sensors;

[0016] Determining the target positioning attribute according to the fused positioning attribute information includes:

[0017] Data of each sensor is fused according to the fusion weight of each sensor to obtain the target positioning attribute.

[0018] Optionally, the fused positioning attribute information includes positioning information, and the positioning information includes actual positioning attribute information or measurement value deviation;

[0019] Determining the target positioning attribute according to the fused positioning attribute information includes:

[0020] The target positioning attribute is determined according to the positioning information.

[0021] Optionally, the motion attribute includes a position attribute, a speed attribute, and / or a heading angle attribute;

[0022] The auxiliary information includes a difference between a position observation of a position sensor and an integral of a velocity observation of a velocity sensor, a difference between a velocity observation of the velocity sensor and an integral of an acceleration observation of an acceleration sensor, and / or a difference between a heading angle observation of a heading sensor and a direction of displacement calculation.

[0023] Optionally, the acquiring of multi-sensor data includes:

[0024] Acquiring raw data collected by each of the sensors;

[0025] Preprocessing is performed on each of the raw data to obtain the multi-sensor data, and the preprocessing method includes position encoding and / or data standardization.

[0026] Optionally, the multi-sensor fusion model includes a feature perception network, a temporal feature extraction network, and an output network;

[0027] Inputting the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain fused positioning attribute information includes:

[0028] Inputting the multi-sensor data into the feature perception network to extract data association features and obtain association feature information;

[0029] Inputting the associated feature information into the temporal feature extraction network to extract the temporal features and obtain target feature information;

[0030] The target feature information is input into the middle output network for processing to obtain the fused positioning attribute information.

[0031] Optionally, the multi-sensor data also includes a signal characterizing the positioning quality of each of the sensors.

[0032] Optionally, the loss function of the multi-sensor fusion model may include a combination of one or more of the following: a mean square error loss function, a Gaussian distribution loss function, a regularization loss function, and a second-order derivative loss function.

[0033] In a second aspect, an embodiment of the present invention provides a multi-sensor fusion positioning device, the device comprising:

[0034] A data acquisition unit is configured to acquire multi-sensor data, wherein the multi-sensor data includes original information and auxiliary information of multiple sensors, wherein the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute;

[0035] A data processing unit is configured to input the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain fused positioning attribute information;

[0036] The positioning unit is configured to determine the target positioning attribute according to the fused positioning attribute information.

[0037] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described above.

[0038] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0039] In a fifth aspect, an embodiment of the present invention provides a computer program product, which, when executed on a computer, enables the computer to execute the method described above.

[0040] The embodiment of the present invention obtains multi-sensor data, inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing, obtains fused positioning attribute information, and determines the target positioning attribute according to the fused positioning attribute information, wherein the multi-sensor data includes original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fused positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0042] Figure 1 is a flow chart of a multi-sensor fusion positioning method according to an embodiment of the present invention;

[0043] Figure 2 is a flow chart of a data processing method of a multi-sensor fusion model according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a multi-sensor fusion positioning process according to an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of another multi-sensor fusion positioning process according to an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of another multi-sensor fusion positioning process according to an embodiment of the present invention;

[0047] Figure 6 is a schematic diagram of a multi-sensor fusion positioning device according to an embodiment of the present invention;

[0048] Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0050] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0051] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0052] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0053] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0054] In related technologies, commonly used technologies for multi-sensor fusion are:

[0055] a. Multi-sensor fusion technology based on Bayesian filtering principle, such as Kalman filter.

[0056] b. Multi-sensor fusion technology based on nonlinear optimization principles, such as factor graph optimization.

[0057] These methods all require an estimate of the uncertainty of sensor observations to determine the proportion of different observations in state estimation.

[0058] However, the uncertainty of sensor observations is difficult to estimate, especially in real-time systems. It is difficult to estimate the uncertainty of current sensor observations. Moreover, different sensors have different measurement principles and observation errors, making it difficult to achieve a balance in the observations of different sensors.

[0059] There are two commonly used methods for uncertainty estimation:

[0060] 1) Based on the uncertainty signal output by the sensor, such as the observation variance, the uncertainty of the observation is used. However, these observations of the sensor are of reference value but not absolutely accurate, and the variance is often too large or too small.

[0061] 2) Based on experience and sensor accuracy data, a fixed uncertainty is assigned to each sensor observation to determine the weights of different weights in the fusion process.

[0062] However, the uncertainty of sensor observation often varies with scenes and working conditions. For example, in degraded scenes lacking geometric features, the uncertainty of positioning observations provided by lidar increases. In single scenes lacking textures, the quality of camera-based positioning will also decrease. GNSS positioning observations will also change with factors such as satellite visibility and signal occlusion. Therefore, the above two uncertainty estimation methods both have a certain impact on positioning accuracy. Based on this, an embodiment of the present invention provides a multi-sensor fusion positioning method, device, and electronic device, which obtains multi-sensor data, inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing, obtains fused positioning attribute information, and determines the target positioning attribute according to the fused positioning attribute information, wherein the multi-sensor data includes original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fusion positioning. The embodiment of the present invention can be applied to any positioning scenario of an object carrying multiple positioning sensors, such as a vehicle positioning scenario, a robot positioning scenario, a drone positioning scenario, etc., and the positioning scenario is not limited here.

[0063] It should be understood that the positioning representation of this embodiment estimates the state of the target object, such as position, speed and / or posture.

[0064] Figure 1 FIG. 1 is a flow chart of a multi-sensor fusion positioning method according to an embodiment of the present invention. Figure 1 As shown, the multi-sensor fusion positioning method of this embodiment includes the following steps:

[0065] Step S110, acquiring multi-sensor data, wherein the multi-sensor data includes original information and auxiliary information of multiple sensors, wherein the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute.

[0066] In an optional implementation, the multiple sensors of this embodiment include any two or more of the following combinations: position sensors (such as various GNSS sensors, etc.), various radar sensors, various camera sensors, speed sensors, acceleration sensors, gyroscopes, magnetometers, odometers, IMU (Inertial Measurement Unit) sensors, etc. It should be understood that the composition of the multi-sensor data can be determined according to the type and number of positioning sensors carried or started on the target device, and is not limited here.

[0067] In an optional implementation, this embodiment obtains multi-sensor data by obtaining raw data collected by multiple sensors and preprocessing the raw data corresponding to each sensor.

[0068] Further optionally, the preprocessing method of this embodiment includes position encoding and / or data standardization.

[0069] Optionally, this embodiment may use trigonometry encoding to encode each raw data, for example, the coordinates of the input position may be encoded into a multi-dimensional (32, 64, etc.) vector. It should be understood that this embodiment does not limit the position encoding method, and it can standardize the representation of the corresponding position. For example, other position encodings, such as absolute / relative position encoding, sinusoidal position encoding, and learning position encoding can all be applied to this embodiment.

[0070] Therefore, this embodiment can use position coding to achieve a unified way to characterize complex spatial features, such as motion trajectories, etc., and can process periodic data (such as heading angle, roll angle, pitch angle, etc.) through trigonometric position coding, thereby solving the unevenness caused by the jump from 360 degrees to 0 degrees. At the same time, when describing the position, some earth coordinate systems will use very large values. For example, when using the UTM coordinate system, large values ​​such as (630084m, 4833438m) will appear, which is not conducive to the convergence of model training. These problems can be well solved through position coding, making model training easier to converge and improving model training efficiency.

[0071] Optionally, this embodiment can standardize the data by taking the logarithm of the data collected by the sensor, or normalizing based on the maximum and minimum values ​​collected, or achieving standard normal distribution of the data by calculating the mean and variance, so as to make the model training easier to converge and obtain more generalized results.

[0072] Furthermore, in this embodiment, the original information in the multi-sensor data may include various types of original data after position encoding or data standardization.

[0073] In this embodiment, the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Optionally, the motion attribute may include a position attribute, a speed attribute and / or a heading angle attribute.

[0074] It is easy to understand that the target device (such as a vehicle, robot, drone, etc.) follows some inherent kinematic laws during movement. For example, the relationship between the position observation and the velocity observation integral (both of which represent the position attribute), that is, when the observations of the position sensor and the velocity sensor are accurate, the position difference and the velocity integral of a certain period of time (such as 1 second) traveling in the same direction should be the same or basically the same. If the position sensor changes by 10 meters in a time interval of 1 second, and the velocity integral of this 1 second is 0 meters, or the velocity is 0m / s, then it is obvious that at least one of the observations between the position sensor and the velocity sensor is wrong. For example, the relationship between the velocity observation and the acceleration observation integral (both of which represent the velocity attribute), and the relationship between the heading angle observation and the direction of the displacement calculation (both of which represent the heading angle attribute), etc.

[0075] Therefore, the auxiliary information of this embodiment includes the difference between the position observation of the position sensor and the integral of the speed observation of the speed sensor, the difference between the speed observation of the speed sensor and the integral of the acceleration observation of the acceleration sensor, and / or the difference between the heading angle observation of the heading sensor and the direction of the displacement calculation.

[0076] In some optional implementations, these kinematic laws can be learned directly through the positioning state of the target device (i.e., the original information of multiple sensors) through the relevant model. In other optional implementations, the above-mentioned motion state quantities (i.e., auxiliary information) can also be directly calculated, and the above-mentioned auxiliary information can be input into the relevant model, which can make the relevant model (such as the multi-sensor fusion model of this embodiment) more accurately and easily capture the kinematic laws between these state quantities, so that the model can determine whether the relevant sensor observations are reliable by whether the relationship between the motion state quantities conforms to the kinematic laws. Compared with the embodiment of directly learning kinematic laws through the original information of multiple sensors, this reduces the model complexity while improving the accuracy of the model kinematic law learning.

[0077] In an optional implementation, the multi-sensor data also includes signals that characterize the positioning quality of each sensor. For example, the number of GNSS satellites involved in the solution, the acceleration observation covariance matrix output by the acceleration sensor, the matching results of the radar point cloud and the map, etc. These signals are correlated with the observation quality of the corresponding sensors, but the noise is relatively large and the accuracy is relatively low. Factors such as the environment in which the target device is located and the operating conditions of each sensor will directly affect the positioning observation quality. During the training process, the relevant model can learn the measurement principles of various sensors, the relationship between operating conditions and positioning quality, etc. based on these signals, which further improves the accuracy of the trained model.

[0078] Step S120, inputting the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain fused positioning attribute information.

[0079] In an optional implementation, the multi-sensor fusion model of this embodiment includes a feature perception network, a time series feature extraction network and an output network. The feature perception network is used to capture the relationship between observations of different sensors at the same time, the time series feature extraction network is used to capture the time series characteristics of the signal and establish the relationship between signals at different times, and the output network is used to convert the network features into fused positioning attribute information.

[0080] In an optional implementation, the feature perception network of this embodiment can use a multi-layer perceptron, the temporal feature extraction network can use a temporal convolutional neural network, a transformer, a long short-term memory network LSTM (long short-term memory networks) and other networks, and the output network can use a fully-connected layer, a lightweight head network, etc. It should be understood that this embodiment does not limit the types of feature perception networks, temporal feature extraction networks and output networks, and it only needs to be able to extract associated features and temporal features and convert features.

[0081] Figure 2 is a flow chart of a data processing method of a multi-sensor fusion model according to an embodiment of the present invention. Figure 2 As shown, in this embodiment, the multi-sensor data is input into a pre-trained multi-sensor fusion model for processing, and obtaining fused positioning attribute information includes the following steps:

[0082] Step S121, input the multi-sensor data into the feature perception network to extract data association features and obtain associated feature information. Different motion state quantities are interrelated. In this embodiment, the multi-sensor data (i.e., the original positioning signal) can be abstracted into high-dimensional features (i.e., associated feature information) through the feature perception network to obtain the relationship between the motion state quantities observed by different sensors at the same time.

[0083] Step S122: input the associated feature information into a time series feature extraction network to extract the time series features and obtain target feature information, wherein the target feature information includes the associated feature information and the time series feature information.

[0084] This embodiment uses a time series feature extraction network to extract time series features from associated feature information and establish the relationship between signals at different times. It is easy to understand that the motion state quantity is continuous and smooth in time, and establishing time series features helps to model the uncertainty of the current signal with the help of historical motion state quantities. For example, if the position signal calculated by radar has been jumping significantly in the past 10 seconds, and the position trajectory is jagged on the time axis, then the radar sensor observation is likely to be unreliable and has a large uncertainty.

[0085] Step S123, input the target feature information into the output network for processing to obtain fused positioning attribute information.

[0086] In an optional implementation manner, the fused positioning attribute information of this embodiment may include any one or a combination of multiple items of the uncertainty of each sensor, the fusion weight of each sensor, and the positioning information of the target device.

[0087] Further optionally, when the fused positioning attribute information includes a combination of multiple items including the uncertainty of each sensor, the fusion weight of each sensor, and the positioning information of the target device, the multi-sensor fusion model of this embodiment may include multiple output layers to respectively output corresponding fused positioning attribute information. Correspondingly, the multi-sensor fusion model, i.e., the multi-task model, may be trained in a multi-task joint training manner during the training process.

[0088] Step S130, determining the target positioning attribute according to the fused positioning attribute information, wherein the target positioning attribute includes motion attribute information of the target device such as position, speed, and / or posture.

[0089] In an optional implementation, when the fused positioning attribute information includes the uncertainty of each sensor, this embodiment performs feature fusion processing on the original information of each sensor based on the fused positioning attribute information to obtain the target positioning attribute.

[0090] Further optionally, this embodiment adjusts the fusion weight of each sensor according to the uncertainty of each sensor, performs data fusion on each sensor according to the fusion weight of each sensor, and obtains the target positioning attribute.

[0091] Further optionally, the present embodiment may also realize data fusion of each sensor according to the uncertainty of each sensor and multi-sensor fusion technology based on the Bayesian filtering principle (such as Kalman filter, etc.), or may realize data fusion of each sensor according to multi-sensor fusion technology based on the nonlinear optimization principle (such as factor graph optimization, etc.), and the present embodiment is not limited to this.

[0092] In an optional implementation, if there is a sensor whose uncertainty exceeds a threshold, such as a camera sensor in a scene lacking texture, the data of the sensor can be filtered out during the multi-sensor data fusion process, or the fusion weight of the sensor can be adjusted to 0 to avoid the influence of the sensor with higher uncertainty on the positioning result, thereby further improving the positioning accuracy.

[0093] In an optional implementation, the uncertainty of this embodiment is characterized by the covariance or variance of the motion attribute of the corresponding sensor.

[0094] Further optionally, the present embodiment can also input the original information of the corresponding sensor and the corresponding covariance or variance (such as the position information of the GNSS and the covariance matrix of the position information, the heading information and the variance of the heading information), and the original information of other sensors into the Kalman filter for fusion processing to obtain the target positioning attribute. Therefore, when the measurement accuracy of some sensors is poor, the multi-sensor fusion model of the present embodiment can output a larger covariance matrix to characterize that the corresponding measurement has a larger uncertainty, so that when the observation is fused through the Kalman filter, the weight of the measurement value can be reduced to further improve the accuracy and stability of the fusion positioning.

[0095] Therefore, this embodiment can process the original information, auxiliary information and signals characterizing the positioning quality of each sensor through a multi-sensor fusion model to obtain the uncertainty of each sensor, improve the accuracy of the uncertainty, and further improve the accuracy of subsequent fusion positioning.

[0096] In another optional implementation, the fused positioning attribute information includes the fusion weight of each sensor. In this case, the present embodiment performs data fusion on each sensor according to the fusion weight of each sensor to obtain the target positioning attribute.

[0097] Further optionally, the present embodiment can calculate the weighted sum of the measurement values ​​of the corresponding sensors according to the fusion weights of each sensor to obtain the corresponding target positioning attribute. For example, the multi-sensor fusion model can output the fusion weights w1 and w2 of the position sensor and the speed sensor for calculating the speed attribute. Assuming that the speeds measured by the position sensor and the speed sensor are v1 and v2 respectively, the speed of the target device v = v1*w1+v2*w2. In other optional implementations, the present embodiment can also realize the data fusion of each sensor according to the fusion weights of each sensor, using a multi-sensor fusion technology based on the Bayesian filtering principle (such as Kalman filtering, etc.) or a multi-sensor fusion technology based on the nonlinear optimization principle (such as factor graph optimization, etc.), and the present embodiment is not limited to this.

[0098] That is to say, the multi-sensor fusion model of this embodiment can process the original information, auxiliary information and signals characterizing the positioning quality of each sensor of each sensor to obtain the fusion weight of each sensor, thereby improving the rationality and accuracy of the fusion weight of each sensor and further improving the accuracy of subsequent fusion positioning. It should be understood that the fusion weight of each sensor corresponds to the uncertainty of its observation.

[0099] In another optional implementation, the fused positioning attribute information of this embodiment includes positioning information. Further optionally, the positioning information may be the actual positioning attribute information of the target device, or may be a deviation relative to a measured value (i.e., a measured value deviation). When the positioning information is the actual positioning attribute information of the target device, this embodiment may directly determine the positioning information as the target positioning attribute. When the positioning information is a deviation relative to a measured value, the measured value may be corrected based on the measured value deviation to obtain the target positioning attribute.

[0100] That is to say, the multi-sensor fusion model of this embodiment can process the original information, auxiliary information and signals characterizing the positioning quality of each sensor, and can directly output the positioning attribute information of the target device after multi-sensor fusion or output the measurement value deviation.

[0101] In other optional implementations, the multi-sensor fusion model of this embodiment may include multiple output networks that output multiple types of fused positioning attribute information, such as an output network that outputs the uncertainty of each sensor, and an output network that outputs positioning information, etc. Therefore, this embodiment can also determine the target positioning attributes respectively through the outputs of different output networks to verify each other, or determine the confidence of different results, etc., so as to further improve the accuracy of the positioning attribute information.

[0102] In an optional implementation, the multi-sensor fusion model of this embodiment can be trained based on the acquired training data, wherein the training data may include the historical measurement data of each sensor (or the measurement data of the historical test) and the corresponding actual positioning information. Furthermore, this embodiment inputs the training data into the multi-sensor fusion model after position encoding and data standardization, and adjusts the parameters based on the corresponding loss function until the loss meets the predetermined condition or the number of training times reaches the predetermined number of times, thereby obtaining the trained multi-sensor fusion model.

[0103] In an optional implementation, the loss function of the multi-sensor fusion model of this embodiment may include a combination of one or more of the following: a mean square error loss function, a Gaussian distribution loss function, a regularization loss function, and a second-order derivative loss function.

[0104] Further, if the output of the multi-sensor fusion model is positioning information (actual positioning attribute information or deviation from the measured value), the loss function can use the mean square error MSE (Mean-Square Error). If the output of the multi-sensor fusion model is the uncertainty of the sensor or the fusion weight, the loss function that maximizes the probability of Gaussian distribution or its variant can be used, for example:

[0105] L=e T ∑ -1 e+α|∑|

[0106] Among them, L is the loss, ∑ represents the covariance matrix to be estimated, which can represent the uncertainty of the corresponding sensor observation, e represents the error matrix between the observed value (that is, the measured value) and the true value, and α represents the coefficient.

[0107] Since it is difficult to determine the true value of uncertainty, it is difficult for the model to directly regress uncertainty using the true value of uncertainty. Therefore, this embodiment describes the Gaussian distribution of the observation value by estimating the covariance matrix. If the probability of the actual observation value error in the distribution is small, it indicates that the multi-sensor fusion model is not accurate in prediction and the training loss is large. Therefore, this embodiment can guide model training by the size of the Gaussian probability to obtain accurate uncertainty estimation.

[0108] Further optionally, the multi-sensor fusion model of this embodiment can also use additional regularization loss function and second-order derivative loss function to constrain the smoothness of the output positioning state. Since the changes of various motion attributes of the target device during actual motion are continuous and smooth, adding smoothness constraints can make the output of the multi-sensor fusion model more consistent with the actual motion situation, which can further improve the accuracy of the model.

[0109] It should be understood that this embodiment does not limit the loss function of the multi-sensor fusion model, as long as it can achieve the regression of the corresponding fusion positioning attribute information.

[0110] The embodiment of the present invention obtains multi-sensor data, inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing, obtains fused positioning attribute information, and determines the target positioning attribute according to the fused positioning attribute information, wherein the multi-sensor data includes original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fused positioning.

[0111] Figure 3 FIG. 1 is a schematic diagram of a multi-sensor fusion positioning process according to an embodiment of the present invention. Figure 3As shown, this embodiment obtains the raw data d1-dn collected by each sensor. Among them, n is greater than or equal to 1, representing the number of sensors, d1 represents the raw data collected by the first sensor, and dn represents the raw data collected by the nth sensor. This embodiment preprocesses the raw data d1-dn to obtain multi-sensor data d. Optionally, the raw data d1-dn include unpreprocessed raw information, auxiliary information, and a signal representing the positioning quality of each sensor, and the multi-sensor data d includes the preprocessed raw information of each sensor, auxiliary information, and a signal representing the positioning quality of each sensor. In other optional implementations, the auxiliary information can also be determined based on the preprocessed raw information, and this embodiment is not limited to this.

[0112] Furthermore, in this embodiment, the multi-sensor data d is input into the multi-sensor fusion model 31, so as to extract the associated features of the multi-sensor data d through the feature perception network 311 to obtain the associated feature information, and extract the time series features of the associated feature information through the time series feature extraction network 312 to obtain the target feature information including the associated features and the time series features, and convert the target feature information through the output network 313 to obtain the uncertainty Σ of the corresponding sensor. Optionally, the uncertainty Σ of this embodiment can be characterized by the covariance or variance of the motion attribute of the corresponding sensor.

[0113] Furthermore, this embodiment outputs the uncertainty Σ of the corresponding sensor to the Kalman filter 32 for fusion processing to obtain the target positioning attribute P. It should be understood that this embodiment can also use other multi-sensor fusion technologies based on the Bayesian filtering principle or multi-sensor fusion technologies based on the nonlinear optimization principle for data fusion, and this embodiment is not limited to this.

[0114] The embodiment of the present invention obtains multi-sensor data, inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing, obtains the uncertainty of the corresponding sensor, and fuses the data of each sensor based on the uncertainty and a pre-determined multi-sensor fusion technology to obtain the corresponding target positioning attribute. The multi-sensor data includes the original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fusion positioning.

[0115] Figure 4 FIG. 1 is a schematic diagram of another multi-sensor fusion positioning process according to an embodiment of the present invention. Figure 4As shown, this embodiment obtains the raw data d1'-dn' collected by each sensor. Among them, n' is greater than or equal to 1, representing the number of sensors, d1' represents the raw data collected by the 1st sensor, and dn' represents the raw data collected by the n'th sensor. This embodiment preprocesses the raw data d1'-dn' to obtain multi-sensor data d'. Optionally, the raw data d1'-dn' includes unpreprocessed raw information, auxiliary information, and a signal representing the positioning quality of each sensor, and the multi-sensor data d' includes the preprocessed raw information of each sensor, auxiliary information, and a signal representing the positioning quality of each sensor. In other optional implementations, the auxiliary information can also be determined based on the preprocessed raw information, and this embodiment is not limited to this.

[0116] Furthermore, this embodiment inputs the multi-sensor data d' into the multi-sensor fusion model 41, so as to perform correlation feature extraction on the multi-sensor data d' through the feature perception network 411 to obtain correlation feature information, and perform time series feature extraction on the correlation feature information through the time series feature extraction network 412 to obtain target feature information including correlation features and time series features, and convert the target feature information through the output network 413 to obtain the fusion weight W of the corresponding sensor.

[0117] Furthermore, this embodiment performs weighted sum based on the fusion weight W of each corresponding sensor and the measurement value of each sensor to obtain the target positioning attribute P'.

[0118] The embodiment of the present invention obtains multi-sensor data, inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing, obtains the fusion weight of the corresponding sensor, and performs weighted sum based on the fusion weight and measurement value of each sensor to obtain the corresponding target positioning attribute. The multi-sensor data includes original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fusion positioning.

[0119] Figure 5 FIG. 1 is a schematic diagram of another multi-sensor fusion positioning process according to an embodiment of the present invention. Figure 5As shown, this embodiment obtains the raw data d1"-dn". collected by each sensor, wherein n" is greater than or equal to 1, representing the number of sensors, d1" represents the raw data collected by the first sensor, and dn" represents the raw data collected by the n"th sensor. This embodiment preprocesses the raw data d1"-dn" to obtain multi-sensor data d". Optionally, the raw data d1"-dn" includes unpreprocessed raw information, auxiliary information, and a signal representing the positioning quality of each sensor, and the multi-sensor data d" includes the preprocessed raw information of each sensor, auxiliary information, and a signal representing the positioning quality of each sensor. In other optional implementations, the auxiliary information can also be determined based on the preprocessed raw information, and this embodiment is not limited to this.

[0120] Furthermore, this embodiment inputs the multi-sensor data d" into the multi-sensor fusion model 51, so as to extract the associated features of the multi-sensor data d" through the feature perception network 511 to obtain the associated feature information, and extract the time series features of the associated feature information through the time series feature extraction network 512 to obtain the target feature information including the associated features and the time series features, and convert the target feature information through the output network 513 to obtain the positioning information. Optionally, the positioning information includes the actual positioning attribute information or the measurement value deviation.

[0121] Furthermore, when the positioning information is the actual positioning attribute information of the target device, this embodiment can directly determine the positioning information as the target positioning attribute. When the positioning information is a deviation relative to the measurement value, the measurement value can be corrected based on the measurement value deviation to obtain the target positioning attribute.

[0122] The embodiment of the present invention obtains multi-sensor data and inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain corresponding target positioning attributes. The multi-sensor data includes original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fusion positioning.

[0123] The following takes estimating the GNSS position, heading value and corresponding uncertainty, and using the estimated GNSS position, heading value and corresponding uncertainty as input to the Kalman filter to perform multi-sensor fusion as an example to further describe the multi-sensor fusion method of the embodiment of the present invention. The details are as follows:

[0124] This embodiment obtains the original data of GNSS, which may specifically include the coordinate values ​​(x, y) and heading values ​​yaw measured by GNSS, track angle measurement values, speed measurement values ​​(that is, original information), the variance of x, y, and yaw measurements provided by GNSS, the number of satellites involved in the solution, RTK (Real-time kinematic, real-time differential positioning) solution status (that is, a signal characterizing the positioning quality), and the consistency of the xy position and speed (for example, the speed measurement value of a speed sensor, etc.) (that is, auxiliary information).

[0125] Furthermore, this embodiment performs position encoding or data standardization on the above-mentioned original data to obtain multi-sensor data, inputs the multi-sensor data after data initialization into a multi-layer perceptron (e.g., a 2-layer perceptron) for feature encoding to obtain associated feature information, and inputs the associated feature information into a time series feature extraction network (e.g., a 4-layer LSTM) to extract time series features to obtain target feature information, and inputs the target feature information into an output network (e.g., a fully connected layer) to regress 7 quantities, including x, y, yaw, variance of x, variance of y, covariance of xy, and variance of yaw (which can characterize the uncertainty of GNSS).

[0126] Furthermore, this embodiment replaces the corresponding original GNSS measurement values ​​with the measurement values ​​of x, y, the xy covariance matrix, and the measurement values ​​and variance of yaw, and inputs them into the Kalman filter for multi-sensor fusion together with the position observation of the radar, the acceleration and angular velocity measurement values ​​of the IMU, and outputs the final target positioning attributes (position, velocity, attitude).

[0127] In this embodiment, when the xy measurement accuracy of GNSS is poor, this embodiment can output a larger xy covariance matrix to indicate that the currently measured xy has a larger uncertainty, thereby reducing the weight of the GNSS xy observation when the Kalman filter fuses the observations, making the final positioning attributes after fusion more accurate and stable.

[0128] It should be understood that this embodiment only uses the uncertainty estimation of GNSS as an example, and other sensors may also use a multi-sensor fusion model to estimate the corresponding uncertainty before multi-sensor data fusion, which will not be described in detail here.

[0129] The embodiment of the present invention obtains multi-sensor data, inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing, obtains the uncertainty of the corresponding sensor, and fuses the data of each sensor based on the uncertainty and a pre-determined multi-sensor fusion technology to obtain the corresponding target positioning attribute. The multi-sensor data includes the original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fusion positioning.

[0130] The following describes the multi-sensor fusion positioning method of the embodiment of the present invention by taking the fusion of GNSS and Lidar as an example. The position of the target device in the map is estimated by matching the Lidar point cloud with the positioning elements in the high-precision map to obtain the Lidar position measurement. The details are as follows:

[0131] This embodiment obtains the raw data of GNSS and lidar, which may include: lidar position measurement, the number of effective matching point clouds between lidar and map, the final error in the lidar position optimization process, GNSS position measurement, GNSS speed measurement, the number of GNSS satellites involved in the solution, and the GNSS positioning solution status.

[0132] Furthermore, this embodiment performs position encoding or data standardization processing on the above-mentioned raw data (for example, trigonometric encoding of Lidar and GNSS position measurements) to obtain multi-sensor data, inputs the multi-sensor data after data initialization into a multi-layer perceptron (for example, a 2-layer perceptron) for feature encoding to obtain associated feature information, and inputs the associated feature information into a temporal feature extraction network (for example, TCNN and Transformer with a convolution kernel size of 3) to extract temporal features to obtain target feature information, and inputs the target feature information into an output network (for example, a 2-layer fully connected layer) to regress the position of xy.

[0133] Furthermore, this embodiment obtains the true xy coordinates by de-normalizing the xy estimated by the multi-sensor fusion model.

[0134] In the process of model reasoning, the embodiment of the present invention compares the uncertainty of different sensors (Lidar, GNSS) through signal input, thereby determining the degree of influence of different sensors on the final fusion result, and implicitly fuses them in the model network structure. For example, assuming that the current scene is a degraded scene lacking effective positioning features, Lidar cannot accurately position, then the final fusion output result will be more biased towards the positioning result of GNSS.

[0135] The embodiment of the present invention obtains multi-sensor data and inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain corresponding target positioning attributes. The multi-sensor data includes original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fusion positioning.

[0136] Figure 6 Schematic diagram of a multi-sensor fusion positioning device according to an embodiment of the present invention. Figure 6 As shown, the multi-sensor fusion positioning device 6 of the embodiment of the present invention includes a data acquisition unit 61, a data processing unit 62 and a positioning unit 63.

[0137] The data acquisition unit 61 is configured to acquire multi-sensor data, wherein the multi-sensor data includes original information and auxiliary information of multiple sensors, wherein the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. The data processing unit 62 is configured to input the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain fused positioning attribute information. The positioning unit 63 is configured to determine the target positioning attribute according to the fused positioning attribute information.

[0138] In an optional implementation, the fused positioning attribute information includes the uncertainty of each of the sensors. The positioning unit 63 is further configured to perform feature fusion processing on the original information of each of the sensors based on the fused positioning attribute information to obtain the target positioning attribute.

[0139] In an optional implementation, the uncertainty is characterized by a covariance or variance of a motion attribute of the corresponding sensor.

[0140] In an optional implementation, the positioning unit 63 is further configured to adjust the fusion weight of each sensor according to the uncertainty of each sensor, perform data fusion on each sensor according to the fusion weight of each sensor, and obtain the target positioning attribute.

[0141] In an optional implementation, the fused positioning attribute information includes a fusion weight of each of the sensors. The positioning unit 63 is further configured to perform data fusion on each of the sensors according to the fusion weight of each of the sensors to obtain the target positioning attribute.

[0142] In an optional implementation, the fused positioning attribute information includes positioning information. The positioning information includes actual positioning attribute information or measurement value deviation. The positioning unit 63 is further configured to determine the target positioning attribute according to the positioning information.

[0143] In an optional implementation, the motion attribute includes a position attribute, a speed attribute, and / or a heading angle attribute;

[0144] The auxiliary information includes a difference between a position observation of a position sensor and an integral of a velocity observation of a velocity sensor, a difference between a velocity observation of the velocity sensor and an integral of an acceleration observation of an acceleration sensor, and / or a difference between a heading angle observation of a heading sensor and a direction of displacement calculation.

[0145] In an optional implementation, the data acquisition unit 61 is further configured to acquire the raw data collected by each of the sensors, preprocess each of the raw data, and acquire the multi-sensor data, wherein the preprocessing method includes position encoding and / or data standardization.

[0146] In an optional implementation, the multi-sensor fusion model includes a feature perception network, a time series feature extraction network and an output network. The data processing unit 62 is further configured to input the multi-sensor data into the feature perception network to extract data association features, obtain association feature information, input the association feature information into the time series feature extraction network to extract time series features, obtain target feature information, input the target feature information into the output network for processing, and obtain the fusion positioning attribute information.

[0147] In an optional implementation, the multi-sensor data also includes a signal characterizing the positioning quality of each of the sensors.

[0148] In an optional implementation, the loss function of the multi-sensor fusion model may include a combination of one or more of the following: a mean square error loss function, a Gaussian distribution loss function, a regularization loss function, and a second-order derivative loss function.

[0149] The embodiment of the present invention obtains multi-sensor data, inputs the multi-sensor data into a pre-trained multi-sensor fusion model for processing, obtains fused positioning attribute information, and determines the target positioning attribute according to the fused positioning attribute information, wherein the multi-sensor data includes original information and auxiliary information of multiple sensors, and the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute. Therefore, the embodiment of the present invention can perform feature fusion through the original information and auxiliary information of multiple sensors, thereby improving the accuracy of fused positioning.

[0150] Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 7 As shown, the server 7 is a general data processing device, which includes a general computer hardware structure, which at least includes a processor 71 and a memory 72. The processor 71 and the memory 72 are connected via a bus 73. The memory 72 is suitable for storing instructions or programs executable by the processor 71. The processor 71 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 71 executes the instructions stored in the memory 72, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 73 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 74 and the display device and the input / output (I / O) device 75. The input / output (I / O) device 75 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 75 is connected to the system via an input / output (I / O) controller 76.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0153] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0154] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0155] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0156] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0157] Another embodiment of the present invention also relates to a target device, which may be a vehicle, a robot, a drone or other device. The target device includes at least two types of positioning sensors, a processor and a memory. Furthermore, the positioning sensor may collect positioning data and store it in the memory. The memory also stores one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement any of the above embodiments to improve the positioning accuracy of the device.

[0158] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-sensor fusion positioning method, characterized in that: The method comprises: Acquire multi-sensor data, where the multi-sensor data includes original information and auxiliary information of multiple sensors, where the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute; Inputting the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain fused positioning attribute information; The target positioning attribute is determined according to the fused positioning attribute information.

2. The method according to claim 1, characterized in that The fused positioning attribute information includes the uncertainty of each of the sensors; Determining the target positioning attribute according to the fused positioning attribute information includes: The original information of each sensor is subjected to feature fusion processing based on the fused positioning attribute information to obtain the target positioning attribute.

3. The method according to claim 2, characterized in that The uncertainty is characterized by the covariance or variance of the motion properties of the corresponding sensor.

4. The method according to claim 2, characterized in that: The performing feature fusion processing on the original information of each sensor and the fused positioning attribute information to obtain the target positioning attribute includes: Adjusting the fusion weight of each sensor according to the uncertainty of each sensor; Data of each sensor is fused according to the fusion weight of each sensor to obtain the target positioning attribute.

5. The method according to claim 1, characterized in that The fused positioning attribute information includes the fusion weight of each of the sensors; Determining the target positioning attribute according to the fused positioning attribute information includes: Data of each sensor is fused according to the fusion weight of each sensor to obtain the target positioning attribute.

6. The method according to claim 1, characterized in that The fused positioning attribute information includes positioning information, and the positioning information includes actual positioning attribute information or measurement value deviation; Determining the target positioning attribute according to the fused positioning attribute information includes: The target positioning attribute is determined according to the positioning information.

7. The method according to claim 1, characterized in that The motion attributes include position attributes, speed attributes, and / or heading angle attributes; The auxiliary information includes a difference between a position observation of a position sensor and an integral of a velocity observation of a velocity sensor, a difference between a velocity observation of the velocity sensor and an integral of an acceleration observation of an acceleration sensor, and / or a difference between a heading angle observation of a heading sensor and a direction of displacement calculation.

8. The method according to claim 1, characterized in that The acquiring of multi-sensor data comprises: Acquiring raw data collected by each of the sensors; Preprocessing is performed on each of the raw data to obtain the multi-sensor data, and the preprocessing method includes position encoding and / or data standardization.

9. The method according to claim 1, characterized in that: The multi-sensor fusion model includes a feature perception network, a temporal feature extraction network, and an output network; Inputting the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain fused positioning attribute information includes: Inputting the multi-sensor data into the feature perception network to extract data association features and obtain association feature information; Inputting the associated feature information into the temporal feature extraction network to extract the temporal features and obtain target feature information; The target feature information is input into the middle output network for processing to obtain the fused positioning attribute information.

10. The method according to claim 1, characterized in that The multi-sensor data also includes a signal representing the positioning quality of each of the sensors.

11. The method according to claim 1, characterized in that: The loss function of the multi-sensor fusion model may include a combination of one or more of the following: a mean square error loss function, a Gaussian distribution loss function, a regularization loss function, and a second-order derivative loss function.

12. A multi-sensor fusion positioning device, characterized in that: The device comprises: A data acquisition unit is configured to acquire multi-sensor data, wherein the multi-sensor data includes original information and auxiliary information of multiple sensors, wherein the auxiliary information is used to characterize the difference between different sensor data corresponding to the same motion attribute; A data processing unit is configured to input the multi-sensor data into a pre-trained multi-sensor fusion model for processing to obtain fused positioning attribute information; The positioning unit is configured to determine the target positioning attribute according to the fused positioning attribute information.

13. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

Images