Data processing method and device, autonomous exploration system and storage medium

Abstract

The invention provides a data processing method and device, an autonomous exploration system and a storage medium, the data processing method is applied to a mobile carrier, the mobile carrier is provided with a main measurement device and an observation device, the main measurement device is configured to measure the motion state of the main measurement device to obtain first measurement data, and the observation device is configured to observe the first measurement data; the observation device is configured to observe the environment to obtain observation data; the method comprises the following steps: determining pose state information of the main measurement device based on first measurement data of the main measurement device; and updating the pose state information of the main measurement device in an error state based on the observation data of the observation device so as to generate the positioning information of the mobile carrier. According to the data processing method provided by the invention, accumulative errors can be reduced, the data processing efficiency is improved, and the accuracy of positioning information of the mobile carrier is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of autonomous exploration, and particularly to a data processing method and apparatus, an autonomous exploration system, and a storage medium. Background Art

[0002] In recent years, the autonomous exploration technology of mobile carriers (such as cars, robots, drones, etc.) has developed rapidly. The decisions made by mobile carriers during autonomous exploration rely on their positioning information, and the positioning information of mobile carriers is usually determined by measurement data obtained by a measurement device measuring its own motion state.

[0003] However, due to measurement noise in the measurement device, the measurement noise will accumulate over time to form a large cumulative error, reducing the accuracy of the positioning information provided by the measurement device for the mobile carrier. Summary of the Invention

[0004] In view of this, the present disclosure provides a data processing method and apparatus, an autonomous exploration system, and a storage medium, which can reduce the cumulative error, improve the data processing efficiency, and improve the accuracy of the positioning information of the mobile carrier.

[0005] The present disclosure provides a data processing method applied to a mobile carrier. The mobile carrier is provided with a main measurement device and an observation device. The main measurement device is configured to measure its own motion state to obtain first measurement data, and the observation device is configured to observe the environment to obtain observation data. The method includes:

[0006] Determine the pose state information of the main measurement device based on the first measurement data of the main measurement device;

[0007] Update the pose state information of the main measurement device in an error state based on the observation data of the observation device for generating the positioning information of the mobile carrier.

[0008] Optionally, the observation device includes a first external observer and / or a second external observer. The first external observer is configured to observe the environment to obtain first observation data, and the second external observer is configured to observe the environment to obtain second observation data;

[0009] The updating of the pose state information of the main measurement device in an error state based on the observation data of the observation device includes at least one of the following:

[0010] Update the pose state information of the main measurement device in a relative error state based on the first observation data;

[0011] Update the pose state information of the main measurement device in an absolute error state based on the second observation data.

[0012] Optionally, updating the pose state information of the main measurement device in the relative error state based on the first observation data includes at least one of the following:

[0013] Based on multiple first observation data respectively obtained by the first external observer in multiple first observation frames, determining observation perturbation information of the same first observation feature among the multiple first observation data, and converting the observation perturbation information into pose error information to update the pose state information of the main measurement device in the multiple first observation frames in the relative error state;

[0014] Based on multiple first observation data respectively obtained by the first external observer in multiple first observation frames, using the state augmentation information of the first external observer in the current first observation frame, determining second observation residual information of the same first observation feature among the multiple first observation data to update the pose state information of the main measurement device in the multiple first observation frames in the relative error state, where the state augmentation information includes: error state information of the main measurement device in the current first observation frame and error state information of the first external observer in multiple first observation frames.

[0015] Optionally, before updating the pose state information of the main measurement device in the relative error state based on the first observation data, it further includes:

[0016] Based on the first measurement data of the main measurement device in each observation frame, determining the error state information of the main measurement device in each first observation frame;

[0017] Based on the error state information of the main measurement device in each observation frame and the calibration parameters between the main measurement device and the first external sensor, determining the error state information of the first external observer in each first observation frame;

[0018] Based on the error state information of the main measurement device in each first observation frame and the error state information of the first external observer in each first observation frame, performing state augmentation processing to determine the state augmentation information of the main measurement device in each observation frame.

[0019] Optionally, determining the observation perturbation information of the same first observation feature among the multiple first observation data based on the multiple first observation data respectively obtained by the first external observer in multiple first observation frames includes:

[0020] Among the multiple first observation data, determining the key observation data corresponding to the key frame and the reference observation data corresponding to the reference frame;

[0021] Based on the pose state information of the first external observer at the key frame and the pose state information at the reference frame, perform feature matching on the key observation data and the reference observation data to determine the observation perturbation information of the same first observation feature in the reference observation data relative to the key observation data.

[0022] Optionally, before determining the observation perturbation information of the same first observation feature among the multiple first observation data respectively obtained based on the first external observer at multiple first observation frames, it further includes:

[0023] Based on the first measurement data of the main measurement device at the key frame and the first measurement data at the reference frame, as well as the calibration parameters between the main measurement device and the first external observer, determine the pose state information of the first external observer at the key frame and the pose state information at the reference frame.

[0024] Optionally, the performing feature matching on the key observation data and the reference observation data according to the pose state information of the first external observer at the key frame and the pose state information at the reference frame includes:

[0025] Based on the reference observation data, determine the corresponding first observation feature of the reference frame;

[0026] Based on the pose state information of the first external observer at the key frame and the pose state information at the reference frame, perform optical flow tracking on the corresponding first observation feature of the reference frame in the key observation data.

[0027] Optionally, the determining the corresponding first observation feature of the reference frame based on the reference observation data includes at least one of the following:

[0028] Perform feature extraction on the reference observation data to determine the first observation feature;

[0029] Obtain object point data in the environment and project the object point data onto the reference observation data to determine the first observation feature.

[0030] Optionally, before obtaining the object point data in the environment, it further includes:

[0031] Generate map data in the global coordinate system based on the second observation data, and determine object point data based on the map data.

[0032] Optionally, the first observation data is suitable for providing color information, and the method further includes:

[0033] Based on the projection relationship between the first observation feature and the object point data, project the color information of the first observation feature into the object point data for map rendering.

[0034] Optionally, before performing optical flow tracking on the first observation feature corresponding to the reference frame in the key observation data based on the pose state information of the first external observer in the key frame and the pose state information in the reference frame, it further includes:

[0035] Obtain the spatial information of the first observation feature corresponding to the reference frame.

[0036] Optionally, the obtaining of the spatial information of the first observation feature corresponding to the reference frame includes at least one of the following:

[0037] Perform feature extraction on the key observation data to determine the first observation feature corresponding to the key frame, and based on the position information of the first observation feature in the key observation data and the position information in the reference observation data, perform triangulation processing to determine the spatial information of the first observation feature corresponding to the reference frame;

[0038] Obtain the spatial information of the object point data projected into the reference observation data, and determine the spatial information of the first observation feature with a projection relationship.

[0039] Optionally, the converting the observation perturbation information into the pose error information includes:

[0040] Convert the observation perturbation information into the pose error information according to the perturbation error conversion matrix.

[0041] Optionally, before converting the observation perturbation information into the pose error information, it further includes:

[0042] Based on the pose state information of the first external observer in the key frame and the pose state information in the reference frame, determine the key rotation information and the reference rotation information;

[0043] Based on the key rotation information and the reference rotation information, determine the perturbation error conversion matrix.

[0044] Optionally, the method further includes: correcting the state augmentation information based on the pose error information.

[0045] Optionally, before updating the pose state information of the main measurement device in the absolute error state based on the second observation data of the second external observer, it further includes:

[0046] Based on the first measurement data collected by the main measurement device at the start time of each second observation frame of the second external observer and the first measurement data collected at the end time of each second observation frame, perform motion compensation on the second observation data of each second observation frame.

[0047] Optionally, the motion compensation is performed by a non-uniform B-spline algorithm.

[0048] Optionally, the number of the first external observers is multiple;

[0049] Before updating the pose state information of the main measurement device in the relative error state based on the first observation data, it further includes:

[0050] Select one of the multiple first external observers as a reference first external observer, and convert the first observation data of the remaining first external observers to the coordinate system of the reference first external observer.

[0051] Optionally, the number of the second external observers is multiple;

[0052] Before updating the pose state information of the main measurement device in the absolute error state based on the second observation data, it further includes:

[0053] Based on the first measurement data collected by the main measurement device at the start time of the second observation frame of each second external observer and the first measurement data collected at the end time of the second observation frame, perform motion compensation on the second observation data of each second external observer respectively;

[0054] Select one of the multiple second external observers as a reference second external observer, and convert the second observation data after motion compensation of the remaining second external observers to the coordinate system of the reference second external observer.

[0055] Optionally, the mobile carrier is further provided with an auxiliary measurement device adapted to measure its own motion state to obtain second measurement data;

[0056] Before updating the pose state information of the main measurement device in the relative error state based on the first observation data, it further includes:

[0057] Based on the second measurement data of the auxiliary measurement device, determine the relative state information of the auxiliary measurement device, which is used to be combined with the first observation data to update the pose state information of the main measurement device in the relative error state.

[0058] Optionally, the auxiliary measurement device includes at least one of an inertial sensor, a wheel speed sensor, and an encoder.

[0059] Optionally, the first external observer is an image sensor or a lidar sensor; the second external observer is one of a lidar sensor, a millimeter-wave radar sensor, an infrared radar sensor, and an ultrasonic radar sensor.

[0060] Optionally, the main measurement device includes at least one of an inertial sensor and a wheel speed sensor.

[0061] The present disclosure also provides a data processing device, which is respectively connected to the main measurement device and the observation device. The main measurement device is configured to measure its own motion state to obtain first measurement data, and the observation device is configured to observe the environment to obtain observation data; the data processing device, the main measurement device, and the observation device are all assembled on a mobile carrier;

[0062] The data processing device includes:

[0063] An information acquisition module, configured to determine the pose state information of the main measurement device based on the first measurement data of the main measurement device;

[0064] An information update module, configured to update the pose state information of the main measurement device in an error state based on the observation data of the observation device, so as to generate the positioning information of the mobile carrier.

[0065] Optionally, the data processing device is respectively connected to the first external observer and / or the second external observer in the observation device, wherein the first external observer is configured to observe the environment to obtain first observation data, and the second external observer is configured to observe the environment to obtain second observation data;

[0066] The information update module includes at least one of the following:

[0067] A first information update unit, configured to update the pose state information of the main measurement device in a relative error state based on the first observation data;

[0068] A second information update unit, configured to update the pose state information of the main measurement device in an absolute error state based on the second observation data.

[0069] The present disclosure also provides an autonomous exploration system, which is assembled on a mobile carrier. The autonomous exploration system includes:

[0070] A main measurement device, configured to measure its own motion state to obtain first measurement data;

[0071] An observation device, configured to observe the environment to obtain observation data;

[0072] A data processing device is configured to determine the pose state information of the main measurement device based on the first measurement data of the main measurement device, and update the pose state information of the main measurement device in an error state based on the observation data of the observation device, so as to generate the positioning information of the mobile carrier.

[0073] The present disclosure also provides a data processing device, including: a memory and a processor. A computer instruction capable of running on the processor is stored on the memory, and when the processor runs the computer instruction, it executes the method steps disclosed in any one of the above embodiments.

[0074] The present disclosure also provides a readable storage medium, on which a computer instruction is stored, and when the computer instruction runs, it executes the method steps disclosed in any one of the above embodiments.

[0075] By using the data processing method provided by the present disclosure, the pose state information of the main measurement device is determined based on the first measurement data obtained by the main measurement device disposed on the mobile carrier to measure its own motion state, and the pose state information of the main measurement device is updated in an error state based on the observation data obtained by the observation device disposed on the mobile carrier to observe the environment, so as to generate the positioning information of the mobile carrier. Thus, by combining the first measurement data and the observation data, the cumulative error can be reduced, the data processing efficiency can be improved, and the accuracy of the positioning information of the mobile carrier can be improved. Description of the Drawings

[0076] In order to more clearly illustrate the technical solutions of the present disclosure, the drawings required to be used in the present disclosure will be introduced below. The following described drawings are only some exemplary embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0077] Figure 1 A flowchart showing an exemplary data processing method provided by the present disclosure is shown.

[0078] Figure 2 A flowchart showing another exemplary data processing method provided by an embodiment of the present disclosure is shown.

[0079] Figure 3 A flowchart showing an exemplary process of determining state augmentation information provided by an embodiment of the present disclosure is shown.

[0080] Figure 4 A flowchart showing an exemplary process of observing disturbance information provided by an embodiment of the present disclosure is shown.

[0081] Figure 5Shows a schematic flow chart of an exemplary feature matching provided by an embodiment of the present disclosure.

[0082] Figure 6 Shows a schematic flow chart of an exemplary process for determining a perturbation error conversion matrix provided by an embodiment of the present disclosure.

[0083] Figure 7 Shows a schematic flow chart of an exemplary process for standardizing and processing second observation data provided by an embodiment of the present disclosure.

[0084] Figure 8 Shows a schematic flow chart of an exemplary application scenario of a data processing method provided by an embodiment of the present disclosure.

[0085] Figure 9 Shows a structural block diagram of a data processing device provided by an embodiment of the present disclosure. Detailed implementation manners

[0086] Based on the above background technology, it can be known that due to the measurement noise of the measurement device, the measurement noise will accumulate over time to form a large cumulative error. Therefore, there is a problem that the accuracy of the measurement device providing positioning information for the mobile carrier needs to be improved.

[0087] To solve the above problems, the present disclosure provides a data processing solution. The pose state information of the main measurement device is determined through the first measurement data obtained by the main measurement device disposed on the mobile carrier for measuring its own motion state, and the pose state information of the main measurement device is updated in the error state through the observation data obtained by the observation device disposed on the mobile carrier for observing the environment, so as to generate the positioning information of the mobile carrier. Thus, by jointly using the first measurement data and the observation data to locate the mobile carrier, the accuracy and data processing efficiency of the mobile carrier positioning information can be improved.

[0088] To enable those skilled in the art to more clearly understand the concept, implementation manner and advantages of the data processing solution, the following will be described in detail with reference to the accompanying drawings.

[0089] Refer to Figure 1 , which shows a schematic flow chart of an exemplary data processing method provided by the present disclosure. In this example, the data processing method can be applied to a mobile carrier, where the mobile carrier can be provided with a main measurement device and an observation device. The main measurement device is configured to measure its own motion state to obtain first measurement data, and the observation device is configured to observe the environment to obtain observation data.

[0090] In some embodiments, the data processing method may include the following steps:

[0091] S11. Determine the pose state information of the main measurement device based on the first measurement data of the main measurement device.

[0092] The first measurement data of the main measurement device may include at least one of first linear acceleration data and first angular velocity data. Optionally, the first measurement data may further include first bias data, where the first bias data may be pre-set data, data generated by an algorithm, or data transmitted through software and hardware, etc.

[0093] The pose state information of the main measurement device may include: first displacement information and first rotation information. Among them, the first displacement information can be obtained by performing double integration on the first acceleration information in the first measurement data; the first rotation information can be obtained by performing single integration on the first angular velocity in the first measurement data.

[0094] In some embodiments, the pose state information of the main measurement device may further include at least one of first velocity information, first bias information, and first gravitational acceleration information. Among them, the first velocity information can be obtained by performing single integration on the first linear acceleration information in the first measurement data; the first bias information can be determined by the first bias data; the first gravitational acceleration information can be pre-set, generated by an algorithm, or information transmitted through software and hardware, etc. By increasing the number of dimensions of the pose state information, the accuracy of subsequent data processing can be improved.

[0095] In some embodiments, for the convenience of calculation and expression, a pose state prediction equation of the main measurement device in the manifold space can be constructed according to the motion state of the main measurement device, and the pose state prediction equation can be related to time. Based on the first measurement data of the main measurement device and the pose state prediction equation, determine the pose state information of the main measurement device in the manifold space.

[0096] S12. Update the pose state information of the main measurement device in the error state based on the observation data of the observation device to generate the positioning information of the mobile carrier.

[0097] In some embodiments, the observation device can capture useful information in the environment, and the observation device and the main measurement device can have data complementarity. Using the observation data of the observation device, the pose state information of the main measurement device can be updated. When updating the pose state information of the main measurement device, based on the observation data of the observation device, the error state information between the pose state information of the main measurement device and the true pose state of the main measurement device can be corrected, and based on the corrected error state information, it is superimposed with the pose state information to achieve the update of the pose state information of the main measurement device.

[0098] Among them, the error state information of the main measurement device may include at least one of displacement error information and rotation error information. Optionally, the error state information of the main measurement device may further include at least one of velocity error information, bias error information, and gravitational acceleration error information, and the bias error information and the gravitational acceleration error information may be information set in advance, generated by an algorithm, or transmitted by software and hardware.

[0099] By correcting the error state information of the main measurement device through the observation data of the observation device, the data volume can be reduced. Assuming that the error follows a Gaussian distribution, the mean of the error covariance is zero, that is, the error is distributed with the numerical zero as the middle value. Therefore, when correcting the error state information of the main measurement device, the middle value (i.e., the numerical zero) can be accurately located, and the error state information with high reliability and accuracy can be obtained, which can improve the accuracy of the pose state information update and ensure that the updated pose state information is closer to the true pose state.

[0100] In some embodiments, for the convenience of calculation and expression, an error state prediction equation related to time in the manifold space of the main measurement device may be constructed according to the motion state of the main measurement device. Through the first measurement data of the main measurement device and the error state prediction equation, the error state information of the main measurement device in the manifold space is determined.

[0101] Optionally, when updating the pose state information of the main measurement device in the error state and obtaining the updated pose state information, the positioning information of the mobile carrier may be generated according to the first displacement information and the first rotation information in the updated pose state information. Optionally, according to the coordinate system conversion parameters between the main measurement device and the global coordinate system, the first displacement information and the first rotation information in the updated pose state information may be converted from the main measurement device coordinate system to the global coordinate system, and the first displacement information and the first rotation information in the global coordinate system are used as the positioning information of the mobile carrier to represent the pose of the mobile carrier in the global coordinate system.

[0102] Among them, the global coordinate system may be set according to specific situations and requirements. For example, the global coordinate system may be the coordinate system where the main measurement device is located when it first acquires the first measurement frame. For another example, if the mobile carrier is provided with a positioning device, the coordinate system of the positioning device may be set as the global coordinate system. Among them, if the positioning device can use the global positioning system (GPS), the coordinate system of the positioning device may be the ENU (east-north-up) coordinate system. For still another example, if there is prior map data, the coordinate system where the prior map data is located may also be set as the global coordinate system.

[0103] In some embodiments, for the convenience of calculation and expression, an error state observation equation related to time of the main measurement device in the popular space can be constructed according to the observation method of the observation device. Based on the pose state information, the observation data, and the error state observation equation, the error state information of the main measurement device is corrected, and the pose state information of the main measurement device is updated.

[0104] In summary, by combining the first measurement data and the observation data, the cumulative error caused by the measurement noise of the main measurement device can be reduced, the data processing efficiency can be improved, and the accuracy of the positioning information of the mobile carrier can be improved.

[0105] It can be understood that, for the convenience of data processing, the data and information in the embodiments of the present disclosure can be represented in a specified mathematical form, such as in the form of matrices and / or quaternions. The present disclosure does not make specific limitations on this.

[0106] In practical applications, the main measurement device can measure its own motion state at the first measurement acquisition frequency. According to the time of each measurement of its own motion state and the obtained first measurement data, it can be recorded as the first measurement data of a first measurement frame.

[0107] The observation device can observe the environment at the observation acquisition frequency. According to the time of each observation of the environment and the obtained observation data, it can be recorded as the observation data of an observation frame.

[0108] In some embodiments, the measurement acquisition frequency of the main measurement device and the observation acquisition frequency of the observation device may be different, resulting in a time inconsistency between the first measurement data of each first measurement frame and the observation data of each observation frame. When the measurement acquisition frequency of the main measurement device and the observation acquisition frequency of the observation device are different, time alignment can be performed on each first measurement frame and each observation frame. It can be understood that if the measurement acquisition frequency of the main measurement device and the observation acquisition frequency of the observation device are the same, time alignment can also be performed on the time of the first measurement data of each first measurement frame and the observation data of each observation frame.

[0109] Among them, in order to reduce the computational load, time alignment can be performed after obtaining a frame of data in the one with the lower frequency among the main measurement device and the observation device. For example, if the observation acquisition frequency of the observation device is less than the first measurement acquisition frequency of the main measurement device, when the observation device acquires the observation data of an observation frame, after finding the first measurement frame closest in time to the observation frame, the first measurement data of the first measurement frame closest in time can be interpolated to the time of the observation frame, so as to obtain the first measurement frame time-aligned with the observation frame, that is, the interpolated first measurement frame has a time alignment relationship with the observation frame.

[0110] After obtaining an observation frame, a first measurement frame time-aligned with the observation frame can be determined, and based on the first measurement data of the first measurement frame time-aligned with the observation frame by the main measurement device, pose state information can be determined, which is the pose state information of the main measurement device at the observation frame.

[0111] In some embodiments, to further reduce the computational load, the generation frequency of the positioning information of the mobile carrier can be determined according to the acquisition frequency of the lower-frequency one of the main measurement device and the observation device, or the generation frequency of the positioning information of the mobile carrier can be the same as the acquisition frequency of the lower-frequency one of the main measurement device and the observation device. For example, if the observation acquisition frequency of the observation device is less than the measurement acquisition frequency of the main measurement device, when the observation device acquires the observation data of an observation frame, a first measurement frame time-aligned with the observation frame is obtained, and based on the first measurement data of the first measurement frame, the pose state information of the main measurement device at the first measurement frame is determined. Based on the observation data of the observation device at the observation frame, the pose state information of the main measurement device at the first measurement frame can be updated in an error state to generate the positioning information of the mobile carrier at the observation frame.

[0112] In some embodiments, the observation device may include at least one external observer with an observation function, observe the environment from at least one observation dimension to obtain at least one type of observation data, and in combination with the observation function of the external observer, adopt a corresponding update method in an error state for the pose state information of the main measurement device according to the type of the observation data, so as to make full use of the observation data of the external observer to enhance the update effect and improve the accuracy of the positioning information of the mobile carrier.

[0113] In an alternative example, the observation device may include at least one of a first external observer and a second external observer, where the first external observer is configured to observe the environment to obtain first observation data, and the second external observer is configured to observe the environment to obtain second observation data.

[0114] Figure 2 The flowchart of another exemplary data processing method provided by an embodiment of the present disclosure is shown. In this example, the data processing method may include:

[0115] S21, determining the pose state information of the main measurement device based on the first measurement data of the main measurement device;

[0116] S22. Update the pose state information of the main measurement device in an error state based on the observation data of the observation device. Updating the pose state information of the main measurement device in an error state may include at least one of the following steps S22-1 and S22-2.

[0117] S22-1. Update the pose state information of the main measurement device in a relative error state based on the first observation data.

[0118] In some examples, the first observation data may be two-dimensional data, and the first observation data may also provide grayscale information. It can be understood that the grayscale information can be directly determined from the first observation data or indirectly determined from the first observation data. The embodiments of the present disclosure do not make specific limitations in this regard.

[0119] For example, if the first observation data includes color information and the color information is grayscale information, the grayscale information can be directly obtained from the first observation data. For another example, if the first observation data includes color information and the color information includes at least one color sub-information (the color sub-information may be red sub-information, green sub-information, or blue sub-information, etc.), the grayscale information can be calculated from the color sub-information. For still another example, the first observation data may include intensity information, and the intensity information can be converted into grayscale information.

[0120] Coordinate transformation can be performed between the first external observer coordinate system and the main measurement device coordinate system through calibration parameters. According to the pose state information of the main measurement device in each first observation frame, the pose state information of the first external observer in each first observation frame can be determined, and based on the first observation data of multiple first observation frames, the error state information of the main measurement device in each first observation frame can be relatively corrected, thereby realizing the update of the pose state information of the main measurement device in each first observation frame.

[0121] The first observation data of multiple first observation frames in the first external observer coordinate system can form relative constraints on the main measurement device, so as to update the pose state information of the main measurement device in a relative error state.

[0122] S22-2. Update the pose state information of the main measurement device in an absolute error state based on the second observation data.

[0123] In some examples, the second observation data may include spatial information. Optionally, the spatial information may be depth information.

[0124] In some embodiments, the second observation data of each second observation frame can be transformed from the second external observer coordinate system to the global coordinate system. Based on the second observation data of each second observation frame in the global coordinate system, the error state information of the main measurement device in each second observation frame can be absolutely corrected to update the pose state information of the main measurement device in each second observation frame.

[0125] Thus, the second observation data in the global coordinate system can form an absolute constraint on the main measurement device, thereby updating the pose state information of the main measurement device under an absolute error state.

[0126] In summary, by making full use of the first observation data of the first external observer and / or the second observation data of the second external observer to form a relative constraint and / or an absolute constraint on the main measurement device, the update effect can be improved, and further the accuracy of the positioning information of the mobile carrier can be improved.

[0127] In some embodiments, the specific process of updating the pose state information of the main measurement device in a relative error state can be set according to actual situations and requirements. The embodiments of the present disclosure do not make specific limitations thereto.

[0128] In some alternative examples, when updating the pose state information of the main measurement device in a relative error state based on the first observation data, it may include at least one of the following:

[0129] 1) Based on the multiple first observation data respectively obtained by the first external observer in multiple first observation frames, determine the observation perturbation information of the same first observation feature among the multiple first observation data, and convert the observation perturbation information into the pose error information to update the pose state information of the main measurement device in a relative error state.

[0130] Among them, the first observation feature can be a feature that is easy to identify in the first observation data, which is beneficial to the association among multiple first observation data. For example, the first observation feature can be a corner feature in the first observation data. Among them, the corner feature can be obtained through a corner extraction algorithm. The corner extraction algorithm can be the FAST (features from accelerated segment test) algorithm, the SIFT (scale-invariant feature transform) algorithm, or the ORB (oriented FAST and rotated BRIEF) algorithm. Another example is that the first observation feature can be a line feature in the first observation data. Among them, the line feature can be obtained, for example, through a line feature extraction algorithm.

[0131] In addition, the observation perturbation information of the first observation feature may be in the first observation data coordinate system, or may be in a specified coordinate system (such as the global coordinate system) according to actual requirements.

[0132] Optionally, based on the multiple first observation data respectively obtained by the first external observer in multiple first observation frames, the first observation residual information of the same first observation feature among the multiple first observation data can be determined. By adding an observation perturbation amount to the position information of the first observation feature in the first observation data, the first observation residual information is made to converge, and the observation perturbation amount determined when the first observation residual information converges is used as the observation perturbation information. The observation perturbation information is converted into the pose error information to correct the error state information of the main measurement device and update the pose state information of the main measurement device.

[0133] Thus, through the observation perturbation information, the pose error information can be quickly determined, and the update efficiency of the pose state information of the main measurement device can be improved.

[0134] 2) Based on the multiple first observation data respectively obtained by the first external observer in multiple first observation frames, using the state augmentation information of the first external observer in the current first observation frame, determine the second observation residual information of the same first observation feature among the multiple first observation data, so as to update the pose state information of the main measurement device in the relative error state.

[0135] Among them, after obtaining the second observation residual information, a Kalman filter framework can be adopted to update the pose state information of the main measurement device in the relative error state, thereby improving the calculation efficiency. Optionally, the Kalman filter framework can be a multi-constraint error Kalman filter framework. At this time, the state augmentation information may include: the error state information of the main measurement device in the current first observation frame and the error state information of the first external observer in multiple first observation frames.

[0136] Through the state augmentation information, a relative constraint form of multiple states can be formed, so that when updating the pose state information of the main measurement device in the relative error state, the error situation of the main measurement device in the current first observation frame and the error situation of the first external observer in multiple first observation frames are considered, thereby improving the robustness of the error state update.

[0137] In some embodiments, in order to improve the accuracy of the state augmentation information, before updating the pose state information of the main measurement device in each observation frame in the relative error state based on the first observation data, determine the state augmentation information of the main measurement device in each observation frame.

[0138] In some alternative examples, refer to Figure 3 , which shows a schematic flowchart of an exemplary process for determining state augmentation information provided by an embodiment of the present disclosure. Before updating the pose state information of the main measurement device in the relative error state based on the first observation data, the following steps may further be included:

[0139] SA1. Based on a plurality of first measurement data of the main measurement device in each observation frame, determine the error state information of the main measurement device in each first observation frame.

[0140] SA2. Based on the error state information of the main measurement device in each observation frame and the calibration parameters between the main measurement device and the first external sensor, determine the error state information of the first external observer in each first observation frame.

[0141] SA3. Based on the error state information of the main measurement device in each first observation frame and the error state information of the first external observer in each first observation frame, perform state augmentation processing to determine the state augmentation information of the main measurement device in each observation frame.

[0142] Optionally, the state augmentation information may include the error state information of the main measurement device in historical first observation frames and the error state information of the first external observer in historical first observation frames. After obtaining the error state information of the main measurement device in the current first observation frame, the error state information of the main measurement device in the historical first observation frames in the state augmentation information may be replaced, and after obtaining the error state information of the first external observer in the current first observation frame, it may be added to the state augmentation information.

[0143] As can be seen from the above, the error state information of the first external observer in each first observation frame is provided by the main measurement device, which can enhance the coupling between the first external observer and the main measurement device. Moreover, by performing state augmentation processing, the state augmentation information can include historical information and current information, providing richer and more reliable information for updating the pose state of the main measurement device in the relative error state, improving the accuracy of the update, and thus improving the accuracy of the positioning information of the mobile carrier.

[0144] In some embodiments, the state augmentation information of the main measurement device in each observation frame may further include the calibration parameters between the main measurement device and the first external observer, enriching the content of the state augmentation information.

[0145] In some embodiments, by performing feature matching on the first observation data of multiple first observation frames, the observation perturbation information may be determined. The following is a schematic illustration through specific examples.

[0146] In some alternative examples, refer to Figure 4, which shows a schematic flowchart of an exemplary process for observing disturbance information provided by an embodiment of the present disclosure. Determining the observation disturbance information of the same first observation feature among the multiple first observation data obtained by the first external observer in multiple first observation frames may specifically include the following steps:

[0147] SB1. Among the multiple first observation data, determine the key observation data corresponding to the key frame and the reference observation data corresponding to the reference frame.

[0148] Among them, the selection rules for the key frame and the reference frame can be determined according to specific circumstances, and the embodiments of the present disclosure do not make specific limitations on this.

[0149] For example, the current first observation frame among the multiple first observation frames can be used as the key frame, and the first observation data corresponding to the current first observation frame can be used as the key observation data; the historical first observation frame among the multiple first observation frames can be used as the reference frame, and the first observation data corresponding to the historical first observation frame can be used as the reference observation data.

[0150] For another example, among the multiple first observation frames, the first observation frame whose pose change of the first external observer satisfies the key frame screening condition can be used as the key frame, and the corresponding first observation data can be used as the key observation data; the remaining first observation frames can be used as the reference frames, and the corresponding first observation data can be used as the reference observation data.

[0151] For still another example, among the multiple first observation frames, the first observation frame whose number of feature points in the first observation data satisfies the key frame screening condition can be used as the key frame, and the corresponding first observation data can be used as the key observation data, and the remaining first observation frames can be used as the reference frames, and the corresponding first observation data can be used as the reference observation data.

[0152] SB2. According to the pose state information of the first external observer in the key frame and the pose state information in the reference frame, perform feature matching on the key observation data and the reference observation data to determine the observation disturbance information of the same first observation feature in the reference observation data relative to the key observation data.

[0153] In some embodiments, when performing feature matching on the first observation data of multiple first observation frames, feature points can be used for matching, that is, the first observation feature is a feature point. Optionally, in order to improve the accuracy of feature matching, feature blocks can also be used for matching, that is, the first observation feature can be a feature block that includes feature points and has a certain area range.

[0154] In addition, when there are multiple reference frames, after feature matching, multiple observation perturbation information will be generated. In order to ensure the accuracy of the pose state update of the main measurement device while reducing the computational load, the multiple observation perturbation information can be fused, so as to convert the fused observation perturbation information into the pose error information to update the pose state information of the main measurement device in multiple first observation frames. Among them, the fusion process can specifically include at least one of the following: sliding window processing, downsampling processing, and illumination change processing.

[0155] In some embodiments, before determining the observation perturbation information of the same first observation feature among the multiple first observation data based on the multiple first observation data respectively obtained by the first external observer in multiple first observation frames, the following steps may further be included:

[0156] Based on the first measurement data of the main measurement device in the key frame and the first measurement data in the reference frame, as well as the calibration parameters between the main measurement device and the first external observer, determine the pose state information of the first external observer in the key frame and the pose state information in the reference frame.

[0157] Thus, the pose state information of the first external observer is related to the pose state information of the main measurement device, further enhancing the coupling between the first external observer and the main measurement device.

[0158] In some embodiments, feature matching can be performed through optical flow tracking. Optionally, Figure 5 FIG. shows a schematic flow diagram of an exemplary feature matching provided by an embodiment of the present disclosure. In the process of performing feature matching on the key observation data and the reference observation data according to the pose state information of the first external observer in the key frame and the pose state information in the reference frame, the following steps may specifically be included:

[0159] SC1. Based on the reference observation data, determine the first observation feature corresponding to the reference frame;

[0160] SC2. Based on the pose state information of the first external observer in the key frame and the pose state information in the reference frame, perform optical flow tracking on the first observation feature corresponding to the reference frame in the key observation data.

[0161] Among them, optical flow tracking can be implemented by an optical flow method. Optionally, the optical flow method can be a reverse optical flow method, so as to improve the calculation rate and matching accuracy.

[0162] In some embodiments, the determining of the first observation feature corresponding to the reference frame based on the reference observation data may include at least one of the following:

[0163] 1) Extract features from the reference observation data to determine the first observation feature.

[0164] 2) Obtain object point data in the environment and project the object point data into the reference observation data to determine the first observation feature.

[0165] In some embodiments, before obtaining the object point data in the environment, the following steps may further be included: Generate map data in the global coordinate system based on the second observation data, and determine the object point data based on the map data.

[0166] Thus, by establishing map data in the global coordinate system through the second observation data without establishing map data in the global coordinate system through the first observation data, the data volume can be reduced, and excessive consumption of computing resources and storage space can be avoided. Moreover, since the object point data is determined through the map data, tight coupling between the first external observer and the second external observer can be achieved. In addition, since the object point data is determined based on the map data, when projecting the object point data into the reference observation data to determine the first observation feature, the quantity of the first observation feature and the spatial stability and accuracy of the first observation feature can be improved.

[0167] In some embodiments, since the first observation data can provide color information, after updating the pose state information of the main measurement device in the relative error state based on the first observation data, the color information of the first observation feature may further be projected into the object point data based on the projection relationship between the first observation feature and the object point data for map rendering. Among them, map rendering may adopt an online rendering method or an offline rendering method. Thus, at least part of the data in the map can have color information, enriching the visual effect of the map.

[0168] In some embodiments, before performing optical flow tracking on the first observation feature corresponding to the reference frame in the key observation data based on the pose state information of the first external observer at the key frame and the pose state information at the reference frame, obtain the spatial information of the first observation feature corresponding to the reference frame.

[0169] Optionally, in the process of obtaining the spatial information of the first observation feature corresponding to the reference frame, at least one of the following may be included:

[0170] 1) Extract features from the key observation data, determine the first observation feature corresponding to the key frame, and perform triangulation based on the position information of the first observation feature in the key observation data and its position information in the reference observation data to determine the spatial information of the first observation feature corresponding to the reference frame.

[0171] 2) Obtain the spatial information of the object point data projected into the reference observation data, and determine the spatial information of the first observation feature with a projection relationship.

[0172] Thus, the spatial information of the first observation feature can be obtained in different ways.

[0173] In some embodiments, in the process of converting the observation perturbation information into the pose error information, the following steps may be included: convert the observation perturbation information into the pose error information according to the perturbation error conversion matrix.

[0174] In some embodiments, the perturbation error conversion matrix between the observation perturbation information and the pose error information can be derived by using the optical flow method and the pose state information of the first external observer in multiple first observation frames. Among them, the optical flow method can be the forward optical flow method or the backward optical flow method.

[0175] For the convenience of those skilled in the art to understand and implement, the following will be explained in conjunction with the backward optical flow method. It can be understood that the following explanatory content is only an example and does not limit the present disclosure. Those skilled in the art can obtain the perturbation error conversion matrix by combining the improved methods or transformation methods of the optical flow method.

[0176] Construct the inter-frame pose information p kr , which is used to represent the coordinate transformation of the first external observer between multiple first observation frames. Taking one first external observer as an example, the inter-frame pose information p kr may include the pose state information of the first external observer in the key frame and the pose state information of the first external observer in the reference frame That is Optionally, the inter-frame pose information p kr may further include: the spatial information d i of the i-th first observation feature in the reference observation data, that is

[0177] Among them, ρ Ck represents the key displacement information of the first external observer in the key frame, and φ Ck represents the key rotation information of the first external observer in the key frame; ρ Cr represents the key displacement information of the first external observer in the key frame, and φ CrIndicates the reference rotation information of the first external observer in the reference frame; the i-th first observation feature can be characterized by the feature point s i = d i × f i where f i is the direction vector of the feature point, and d i is the spatial information of the feature point.

[0178] Taking the i-th first observation feature s i as an example, its backlight flow residual observation equation under the backlight flow condition can be expressed as where I is used to represent the pixel gray value, π is used to represent the projection transformation, and T wk represents the conversion parameter between the first external observer coordinate system and the global coordinate system at the key frame, and T wr represents the conversion parameter between the first external observer coordinate system and the global coordinate system at the reference frame.

[0179] Define the affine transformation as The backlight flow residual observation equation can be expressed as res = I r (π(W(s i ; ∈p)))-I k (π(W(s i ; p))), where ∈p represents the observation perturbation information. Under the backlight flow condition, the update method of the affine transformation can be defined as: W(s; p) = W(s; p)W(s; ∈p) -1 . The optimal observation perturbation information ∈p can be determined through iteration.

[0180] Based on this, by repeatedly correcting the pose state information of the key frame and the pose state information of the reference frame, the relationship between the observation perturbation information ∈p and the pose error information δp under the backlight flow condition can be deduced as follows:

[0181]

[0182] where Exp represents the transformation from the rotation vector to the rotation matrix; φ Crk represents the key rotation information of the first external observer from the key frame to the reference frame; M represents the perturbation error conversion matrix.

[0183] Optionally, the backlight flow residual can also be adjusted according to at least one of the sliding window processing, downsampling processing, and illumination change processing.

[0184] In some embodiments, as the first external observer continuously acquires the first observation data of new first observation frames, the key frame and the reference frame will be dynamically adjusted, so the perturbation error conversion matrix also needs to be dynamically adjusted accordingly. Optionally, Figure 6The figure shows a schematic flowchart of an exemplary method for determining a perturbation error conversion matrix provided by an embodiment of the present disclosure. Before converting the observed perturbation information into the pose error information, the following steps may further be included:

[0185] SD1. Determine key rotation information and reference rotation information based on the pose state information of the first external observer at key frames and the pose state information at reference frames;

[0186] SD2. Determine a perturbation error conversion matrix based on the key rotation information and the reference rotation information.

[0187] In some embodiments, based on the pose error information, the state augmentation information may further be corrected, so that the content in the state augmentation information is continuously corrected according to the pose error information, thereby improving the robustness of the state augmentation information.

[0188] In some embodiments, when updating the pose state information of the main measurement device in an absolute error state based on the first observation data, the following steps may be included: Determine third observation residual information based on the second observation data obtained by the second external observer at each second observation frame and the map data, so as to update the pose state information of the main measurement device.

[0189] Optionally, feature extraction may be performed on the second observation data to obtain second observation features, where the second observation features may include at least one of point features, line features, and surface features. The third observation residual information may be determined according to the distance residual between the pose information of the same second observation feature in the second observation data and the pose information in the map data.

[0190] After obtaining the third observation residual information, a Kalman filter framework may be used to update the pose state information of the main measurement device in an absolute error state, thereby improving the calculation efficiency. Optionally, the Kalman filter framework may be an iterative error state Kalman filter framework, so as to perform multiple iterative updates on the error state information of the main measurement device to obtain the optimal error state information of the main measurement device.

[0191] Thus, through repeated iteration, multiple absolute constraints are imposed on the main measurement device, thereby improving the robustness of the error state update.

[0192] In some embodiments, the iterative error state Kalman filter framework may use the error covariance P. The iterative error state Kalman filter framework at time j and the iterative error state Kalman filter framework at time j + 1 may not be in the same manifold space. At this time, the error covariance P at time j j may be projected onto the manifold space at time j + 1. Optionally, the projection may be performed through the convergence state information, such as Among them, is the error covariance obtained after k + 1 iterations at time j, is the error covariance after projecting to the Riemannian space at time j + 1, and L k+1 is the iterative state information, which can be determined according to the error gain and the error noise covariance.

[0193] In practical applications, as the second external observer moves with the mobile carrier, it takes a certain amount of time for the second external observer to collect the second observation data of a second observation frame, which may cause distortion of the second observation data. In order to reduce or even eliminate the distortion, before updating the pose state information of the main measurement device in the absolute error state based on the second observation data of the second external observer, it may further include: performing motion compensation on the second observation data of each second observation frame based on the first measurement data collected by the main measurement device at the start time of each second observation frame of the second external observer and the first measurement data collected at the end time of each second observation frame.

[0194] Thus, through the first measurement data of the main measurement device, the consideration of speed can be increased during the motion compensation process, thereby improving the distortion correction effect.

[0195] In some embodiments, the motion compensation is performed by the non-uniform B-spline algorithm. Since the non-uniform B-spline algorithm can change the influence range of the motion compensation control points, select and remove the first measurement data of the main measurement device with larger noise, and increase the influence range of the motion compensation control points with smaller noise, thereby improving the accuracy of the motion compensation result.

[0196] In some embodiments, there are dynamic objects in the environment, such as pedestrians, vehicles, and animals, etc., which may have some impacts on positioning and mapping, and filter the dynamic objects from the data. Optionally, after the second external observer collects the second observation data, the second observation data can be filtered for dynamic objects.

[0197] In some embodiments, the number of the first external observers can be multiple; before updating the pose state information of the main measurement device in the relative error state based on the first observation data, it may further include: selecting one of the multiple first external observers as a reference first external observer, and converting the first observation data of the remaining first external observers to the coordinate system of the reference first external observer. The first observation data of the multiple first external observers can be uniformly converted to the same coordinate system for subsequent operations.

[0198] In some embodiments, the number of the second external observers may be multiple, and the second observation data of the multiple second external observers may be normalized to facilitate subsequent data processing. For example, Figure 7 The figure shows a schematic flowchart of an exemplary process for normalizing the second observation data provided by an embodiment of the present disclosure. Before updating the pose state information of the main measurement device in the absolute error state based on the second observation data, the process may further include:

[0199] SE1, performing motion compensation on the second observation data of each of the second external observers respectively based on the first measurement data collected by the main measurement device at the start moment of the second observation frame of each of the second external observers and the first measurement data collected at the end moment of the second observation frame;

[0200] SE2, selecting one of the multiple second external observers as a reference second external observer, and converting the second observation data of the remaining second external observers after motion compensation to the coordinate system of the reference second external observer.

[0201] Thereby, the second observation data of the multiple second external observers are uniformly converted to the same coordinate system for subsequent operations.

[0202] In some embodiments, when the formats of the first observation data from multiple first external observers are inconsistent, the format of the first observation data of one of the first external observers may be selected as the target format, and then the formats of the first observation data of the remaining first external observers are converted to the target format.

[0203] In some embodiments, the mobile carrier may further be provided with an auxiliary measurement device configured to measure its own motion state to obtain second measurement data. Based on this, before updating the pose state information of the main measurement device in the relative error state based on the first observation data, the process may further include:

[0204] Determining the relative state information of the auxiliary measurement device based on the second measurement data of the auxiliary measurement device for combining with the first observation data to update the pose state information of the main measurement device in the relative error state.

[0205] The second measurement data of the auxiliary measurement device may include second linear acceleration data and second angular velocity data. Optionally, the second measurement data may further include second bias data, where the second bias data may be pre-set data, data generated by an algorithm, or data transmitted through software and hardware, etc.

[0206] The relative state information of the auxiliary measurement device may include: relative displacement information and relative rotation information. Among them, the relative displacement information can be obtained by pre-integrating the second acceleration information in the second measurement data; the relative rotation information can be obtained by pre-integrating the second angular velocity in the second measurement data.

[0207] Optionally, the relative state information of the auxiliary measurement device may further include at least one of relative velocity information, relative bias information, and relative gravitational acceleration information. Among them, the relative velocity information can be obtained by pre-integrating the second linear acceleration information in the second measurement data; the relative bias information can be determined by the second bias data; the relative gravitational acceleration information can be information set in advance, generated by an algorithm, or transmitted by software and hardware, etc.

[0208] When the relative state information is combined with the first observation data to update the pose state information of the main measurement device in the relative error state, the relative state information of the auxiliary measurement device can be added to the state augmentation information to enrich the content of the state augmentation information. The Kalman filter framework can also be used to update the pose state information of the main measurement device in the relative error state to improve the calculation efficiency.

[0209] Thus, optionally, the accuracy of updating the pose state information of the main measurement device in the relative error state can be improved.

[0210] In some embodiments, the auxiliary measurement device can measure its own motion state according to the second measurement acquisition frequency. According to the time of each measurement of its own motion state and the obtained second measurement data, it can be recorded as the second measurement data of a second measurement frame.

[0211] To reduce the calculation amount, time alignment can be performed after obtaining a frame of data in the device with the slowest frequency among the main measurement device, the auxiliary measurement device, and the observation device. For example, if the observation acquisition frequency of the observation device is the slowest, when the observation device acquires the observation data of an observation frame, the first measurement frame and the second measurement frame that are closest to the time of this observation frame can be found as the first measurement frame and the second measurement frame for time alignment with this observation frame. The first measurement data corresponding to the first measurement frame is set to have a time alignment relationship with the observation data of this observation frame, and the second measurement data corresponding to the second measurement frame is set to have a time alignment relationship with the observation data of this observation frame.

[0212] In some embodiments, the main measurement device may be proprioceptive sensors. In some embodiments, proprioceptive sensors may include: inertial measurement unit (IMU), wheel speed sensors, encoders, or other similar sensors. In some embodiments, inertial sensors may include gyroscopes, accelerometers, odometers, and single, dual, and triaxial combined sensors of gyroscopes and accelerometers, attitude sensors, or other similar sensors. In some embodiments, wheel speed sensors may include electromagnetic induction type, excitation type, Hall effect type, eddy current type, magnetoresistive type, etc. wheel speed sensors.

[0213] In some embodiments, the auxiliary measurement device may be proprioceptive sensors. In some embodiments, proprioceptive sensors may include: inertial sensors, wheel speed sensors, encoders, or other similar sensors. In some embodiments, inertial sensors may include gyroscopes, accelerometers, odometers, and single, dual, and triaxial combined sensors of gyroscopes and accelerometers, attitude sensors, or other similar sensors. In some embodiments, wheel speed sensors may include electromagnetic induction type, excitation type, Hall effect type, eddy current type, magnetoresistive type, etc. wheel speed sensors. In some embodiments, encoders may include optical type, magnetic type, inductive type, capacitive type, etc. encoders.

[0214] In some embodiments, the observation device may include one or more external observers. For example, the observation device may include at least one of a first external observer and a second external observer.

[0215] In some embodiments, the first external observer may be an external sensor. In some embodiments, external sensors may include external sensors with image acquisition functions or external sensors with lidar detection functions. In some embodiments, the first external observer may be an image sensor or a lidar sensor. When the first external observer is an image sensor, observing the environment from the image acquisition dimension can obtain observation data of the image type (such as image data). When the first external observer is a lidar sensor, observing the environment from the lidar detection dimension can obtain observation data of the lidar detection type (such as lidar point cloud data).

[0216] In some embodiments, the image sensor may be a camera. In some embodiments, the camera may include various types such as a monocular camera, a multi - camera, and an RGBD camera. In some embodiments, the camera may include various types such as a pinhole camera and a fish - eye camera. In some embodiments, the lidar sensor may include various types such as a mechanically rotating lidar, a polygon - mirror lidar, and a solid - state lidar.

[0217] In some embodiments, when the number of the first external observers is multiple, the multiple first external observers may be selected as external sensors with partially the same functions or all the same functions. In some embodiments, the multiple first external observers may be image sensors or lidar sensors. In some embodiments, the multiple first external observers may include external sensors with at least partially different functions. In some embodiments, the multiple first external observers may include at least one image sensor and at least one lidar sensor. The present disclosure does not limit the specific forms of the multiple first external observers.

[0218] It can also be understood that data conversion can be performed between multiple first external observers with the same function but different types. For example, the image data of a fish - eye camera can be converted into the image data of a pinhole camera.

[0219] In some embodiments, the second external observer may be an external sensor with a detection function. In some embodiments, the second external observer may include a lidar sensor, a millimeter - wave radar sensor, an infrared radar sensor, and an ultrasonic radar sensor, or other similar sensors. Taking the lidar sensor as an example, when the second external observer is a lidar sensor, the environment can be observed from the laser detection dimension, and the observation data of the laser detection type (such as laser point cloud data) can be obtained.

[0220] In some embodiments, when the number of the second external observers is multiple, the multiple second external observers may be sensors of the same type. For example, all the multiple second external observers are lidar sensors. In some embodiments, the multiple second external observers may include external sensors with different types of detection functions. For example, the multiple second external observers include at least one lidar sensor and at least one millimeter - wave radar sensor. In some embodiments, the multiple second external observers may include multiple external sensors of the same type and multiple external sensors of different types. For example, the multiple second external observers include two lidar sensors, one millimeter - wave radar sensor, and one infrared radar sensor. The present disclosure does not limit the specific types of the multiple second external observers.

[0221] For the convenience of those skilled in the art to understand and implement, the following is a schematic description through an example.

[0222] In an optional example, the mobile carrier may be provided with a main measurement device, an observation device, and an auxiliary measurement device. The main measurement device may be an inertial sensor. The observation device may include a first external observer and a second external observer. The first external observer may be an image sensor, the second external observer may be a lidar sensor, and the auxiliary measurement device may be a wheel speed sensor.

[0223] Combined with reference Figure 8 , a schematic flowchart of an exemplary application scenario of the data processing method provided by an embodiment of the present disclosure is shown. Among them, the inertial sensor, the image sensor, the lidar sensor, and the wheel speed sensor work according to their respective frequencies.

[0224] Referring to step S80, the point cloud data of several second observation frames initially acquired by the lidar sensor can be used for map initialization. After being converted from the lidar coordinate system to the global coordinate system, it serves as the initial map data in the global coordinate system to construct the global initial map. In this example, the data processing method may include:

[0225] Referring to step S811, after the image sensor outputs the image data of a first observation frame, or after the lidar sensor outputs the point cloud data of a second observation frame, a state determination is performed. Then, the data processing flow is carried out according to the type of sensor.

[0226] First, the branch related to the image sensor will be described. After the image sensor outputs the image data of the first observation frame, the first measurement data of the first measurement frame with time alignment is obtained. Based on the first measurement data, the pose state information X of the inertial sensor in the first observation frame is determined imu1 =(R I p I v I b a b g g), and, based on the first measurement data of the inertial sensor, the error state information of the inertial sensor is determined

[0227] Among them, R I represents the first rotation information of the inertial sensor, P I represents the first displacement information of the inertial sensor, V I represents the first speed information of the inertial sensor, b a represents the first bias information of the inertial sensor with respect to the accelerometer, b g represents the first bias information of the inertial sensor with respect to the gyroscope, and g represents the first gravitational acceleration information of the inertial sensor. Indicates the rotational error information of the inertial sensor, Indicates the displacement error information of the inertial sensor, Indicates the velocity error information of the inertial sensor, Indicates the bias error information of the inertial sensor with respect to the accelerometer, Indicates the bias error information of the inertial sensor with respect to the gyroscope, Indicates the gravitational acceleration error information of the inertial sensor.

[0228] According to the calibration parameters between the inertial sensor and the image sensor, the pose state information X of the image sensor can be determined C =(R C p C ) and the error state information of the image sensor Among them, R c Indicates the rotational information of the image sensor, p c Indicates the displacement information of the image sensor; Indicates the rotational error information of the image sensor, Indicates the displacement error information of the image sensor.

[0229] Obtain the second measurement data of the wheel speed sensor with time alignment. Through pre-integration, the relative state information X of the wheel speed sensor can be determined wheel .

[0230] Referring to step S812, based on the pose state information X of the inertial sensor imu1 , the relative state information X of the wheel speed sensor wheel and the pose state information X of the image sensor c , perform state augmentation to obtain the state augmentation information X SYS =(X imu1 X wheel X history X c ). Among them, X history Indicates the pose state information of the image sensor in the historical first observation frame.

[0231] Referring to step S813, based on the state augmentation information X SYS , adopt a multi-constraint error Kalman filter framework to correct the error state information of the inertial sensor , and update the pose state information X by superimposing the corrected error state information imu1 on the pose state information X to perform state update on the pose state information X imu1 to obtain the updated pose state information X of the inertial sensor imu1 ', for generating the positioning information of the mobile carrier.

[0232] Take the first observation frame of the image sensor in history as the reference frame, and the image data of the image sensor in the first observation frame of history as the reference image data. Take the first observation frame of the image sensor in the current as the key frame, and the image data of the image sensor in the first observation frame of the current as the key image data. Among them, when using the sliding window algorithm for data processing, the reference frame can be the first observation frame in history in the sliding window. Further, it can be the first observation frames with a co-visibility relationship among multiple first observation frames in history.

[0233] Referring to step S821, according to the pose state information X of the image sensor in the key frame c and the pose state information X of the image sensor in the reference frame history , perform image feature extraction on the key image data and the reference image data to determine the first observation feature, and / or determine object point data from the map data, and project the object point data onto the reference image data to determine the first observation feature. When determining object point data from the map data and projecting the object point data onto the reference image data, determining the first observation feature through the object point data can ensure that the first observation feature has relatively accurate depth information.

[0234] When using the sliding window algorithm for data processing, a certain first observation frame in the history within the sliding window can be selected as the reference frame. For example, the first observation frame with the most triangulated features can be selected as the reference frame. Among them, the triangulated feature can represent the first observation feature obtained through image feature extraction and co-visible by at least three first observation frames. Optionally, the first observation frame in history with the highest co-visibility with the current first observation frame can be selected as the reference frame to be projected.

[0235] After meshing the reference image data, if there are triangulated features in the image grid, the object point data can be projected onto the image grid, and all the object point data projected onto the image grid within a certain range (such as a range with a radius of 5 pixels) around each (for example, each) triangulated feature can be recorded.

[0236] The object point data projected onto the image grid can be optimized, for example, by using algorithms such as robust principal component analysis (PCA) and outlier filtering for optimization. Optionally, the singular value decomposition (SVD) algorithm can also be used to decompose and calculate the eigenvector corresponding to the minimum eigenvalue of the covariance matrix of each object point data projected onto the image grid to remove the non-planar point data in the object point data. For the planar point data in the decomposed object point data, a plane equation can also be obtained by fitting, and optionally, for example, the least squares fitting algorithm can be used for plane fitting. Optionally, after the plane equation is identified, the depth information of the projected object point data can also be calculated based on the pose of the image sensor, the direction vector of the object point data, and the plane parameters. Then, the reprojection error of the first observed frame of the sliding window co-visibility is calculated using the depth information of the projected object point data. If the reprojection error is less than the triangulated depth information, the depth information of the projected object point data can be adopted, otherwise, the depth information of the projected object point data may not be adopted.

[0237] Referring to step S822, based on the pose state information of the image sensor at the key frame and the pose state information at the reference frame, the first observed feature corresponding to the reference frame in the key image data is tracked by back optical flow to determine the observation perturbation information of the same first observed feature in the reference observed data relative to the key observed data. The color information of the first observed feature can also be projected into the object point data for map rendering.

[0238] Continuing to refer to step S813, according to the perturbation error transformation matrix, the observation perturbation information is converted into the pose error information. According to the pose error information, the error state information of the inertial sensor is corrected. By superimposing the corrected error state information on the pose state information X imu1 the pose state information X is updated to obtain the updated pose state information X of the inertial sensor imu1 for generating the positioning information of the mobile carrier. imu1 ”

[0239] The following describes the relevant branches of the lidar sensor. Continuing to refer to step S811, after the initial map is constructed, the lidar sensor outputs the point cloud data of the second observation frame, and the first measurement data of the inertial sensor with time alignment is obtained. Based on the point cloud data of the lidar sensor and the first measurement data of the inertial sensor, the pose state information and error state information of the inertial sensor in the second observation frame are determined. For specific reference, please refer to the relevant description of the image sensor part, which will not be elaborated here.

[0240] Referring to step S831, based on the pose state information and error state information of the inertial sensor in the second observation frame, non-uniform B-spline motion compensation and dynamic object filtering can be performed on the point cloud data.

[0241] Referring to step S832, dynamic object filtering can be performed on the point cloud data. And referring to step S833, feature extraction can be performed on the point cloud data to determine the second observation feature.

[0242] Continuing to refer to step S813, based on the second observation feature, using the iterative error state Kalman filter framework, the error state information of the inertial sensor is iteratively updated multiple times to obtain the optimal error state information of the main measurement device. And by adding the corrected error state information to the pose state information X imu2 the updated pose state information X ' of the main measurement device is obtained for generating the positioning information of the mobile carrier. imu2

[0243] It can be understood that after obtaining the positioning information, the positioning information can be converted from the inertial sensor coordinate system to the global coordinate system to update the map data.

[0244] In some alternative embodiments, the method described in the present disclosure can be executed by devices provided on sensors, mobile carriers, and servers. In some alternative embodiments, the method described in the present disclosure can be executed by processors, controllers, cloud platforms, or similar devices. In some alternative embodiments, the method described in the present disclosure can be implemented in a software, hardware, or a combination of software and hardware manner.

[0245] The present disclosure also provides a data processing device corresponding to the above data processing method. The following will be introduced in detail with reference to the accompanying drawings through some embodiments. It should be noted that the data processing device described below can be considered as a functional module required to implement the data processing method provided by the present disclosure. The content of the data processing device described below can be mutually corresponded and referred to the content of the data processing method described above.

[0246] In some embodiments, such as Figure 9 ​As shown, it is a structural block diagram of an exemplary data processing device provided by an embodiment of the present disclosure. In this example, the data processing device D1 can be respectively connected to the main measurement device DA and the observation device DB. The main measurement device DA is configured to measure its own motion state to obtain first measurement data, and the observation device DB is configured to observe the environment to obtain observation data; the data processing device D1, the main measurement device DA, and the observation device DB are all assembled on a mobile carrier (not shown in the figure); the data processing device D1 may include:

[0247] An information acquisition module D11, configured to determine the pose state information of the main measurement device DA based on the first measurement data of the main measurement device DA;

[0248] An information update module D12, configured to update the pose state information of the main measurement device DA in an error state based on the observation data of the observation device DB, so as to generate the positioning information of the mobile carrier.

[0249] Thus, by combining the first measurement data and the observation data, it is possible to reduce the interference caused by the measurement noise of the main measurement device, improve the data processing efficiency, and improve the accuracy of the positioning information of the mobile carrier.

[0250] In some embodiments, the data processing device can be respectively connected to the first external observer and / or the second external observer in the observation device, where the first external observer is configured to observe the environment to obtain first observation data, and the second external observer is configured to observe the environment to obtain second observation data; the information update module includes at least one of the following:

[0251] A first information update unit, configured to update the pose state information of the main measurement device in a relative error state based on the first observation data;

[0252] A second information update unit, configured to update the pose state information of the main measurement device in an absolute error state based on the second observation data.

[0253] It should be understood that the division of each module and unit in the above system is only a division of logical functions. In actual implementation, there may be other division methods. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. In addition, the modules and units in the device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or the functions of each module and unit of the device. The processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory inside or outside the system. Alternatively, the modules and units in the device can be implemented in the form of a hardware circuit, and the functions of some or all of the modules can be implemented through the design of the hardware circuit. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application specific integrated circuit (ASIC), and the functions of some or all of the above modules are implemented through the design of the logical relationship of the components in the circuit. Again, for example, in another implementation, the hardware circuit can be implemented by a programmable logic device (PLD), which can include a large number of logic gate circuits, and the logical relationship between the logic gate circuits is configured through a configuration file to implement the functions of some or all of the above modules. All the modules of the above system can be fully implemented in the form of a processor calling a program, or fully implemented in the form of a hardware circuit, or partially implemented in the form of a processor calling a program, and the remaining part is implemented in the form of a hardware circuit.

[0254] The present disclosure also provides an autonomous exploration system, which is assembled on a mobile carrier. The autonomous exploration system may include:

[0255] A main measurement device configured to measure its own motion state to obtain first measurement data;

[0256] An observation device configured to observe the environment to obtain observation data;

[0257] A data processing device configured to determine the pose state information of the main measurement device based on the first measurement data of the main measurement device, and update the pose state information of the main measurement device under an error state based on the observation data of the observation device, so as to generate the positioning information of the mobile carrier.

[0258] In some embodiments, the mobile carrier may include devices such as vehicles, aircraft, unmanned aerial vehicles, robots, ships, etc. In some embodiments, the vehicle may include automobiles, trucks, motorcycles, golf carts, agricultural vehicles, or any other vehicle (e.g., buses, lawn mowers, amusement park equipment or vehicles, construction equipment or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, trams, trains, trolleys, logistics vehicles, etc.).

[0259] The present disclosure also provides a data processing device, including: a memory and a processor. A computer instruction capable of running on the processor is stored on the memory. When the processor runs the computer instruction, it executes the steps of the method described in any of the above embodiments.

[0260] In some embodiments, the processor may be implemented by a processing chip such as a CPU, FPGA (field programmable gate array), or may be implemented by an ASIC or one or more integrated circuits configured to implement the embodiments of the present disclosure.

[0261] In some embodiments, the memory may include a random access memory (RAM), or may also include a non-volatile memory. Further, the memory may include at least one of: phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), read-only memory (ROM), and electrically erasable programmable read-only memory (EEPROM).

[0262] In some embodiments, the data processing device may further include an expansion interface, adapted to be connected to other devices to achieve data communication. For example, the data processing device may be connected to a display device through the expansion interface to display the movement trajectory of the mobile carrier on a map.

[0263] The present disclosure also provides a readable storage medium, on which a computer instruction is stored. When the computer instruction runs, it executes the steps of the method described in any of the above embodiments.

[0264] The present disclosure may take the form of a computer program product implemented on one or more storage media that contain program code. Computer-usable storage media include both permanent and non-permanent, removable and non-removable media, and may implement information storage by any method or technology. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: PRAM, SRAM, DRAM, other types of RAM, ROM, EEPROM, flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device.

[0265] It should be noted that in the description of the present disclosure, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present disclosure.

[0266] It should also be noted that the so-called "one embodiment" or "embodiment" in the present disclosure refers to a specific feature, structure or characteristic that may be included in at least one implementation manner of the present disclosure. And in the description of the present disclosure, terms such as "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with terms such as "first" and "second" may explicitly or implicitly include one or more of such features. Moreover, terms such as "first" and "second" are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or indicate importance. It can be understood that such terms can be interchanged under appropriate circumstances, so that the embodiments described in the present disclosure can be implemented in an order other than the manner shown or described in the present disclosure.

[0267] Although the embodiments of the present invention are disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention should be determined by the scope defined by the claims.

Claims

1. A data processing method, characterized in that, it is applied to a mobile carrier, the mobile carrier is provided with a main measurement device and an observation device, the main measurement device is configured to measure its own motion state to obtain first measurement data, and the observation device is configured to observe the environment to obtain observation data; the method includes: Based on the first measurement data of the main measurement device, determine the pose state information of the main measurement device; Based on the observation data of the observation device, update the pose state information of the main measurement device in an error state to generate the positioning information of the mobile carrier.

2. The data processing method according to claim 1, characterized in that, the observation device includes a first external observer and / or a second external observer, wherein the first external observer is configured to observe the environment to obtain first observation data, and the second external observer is configured to observe the environment to obtain second observation data; The updating the pose state information of the main measurement device in an error state based on the observation data of the observation device includes at least one of the following: Based on the first observation data, update the pose state information of the main measurement device in a relative error state; Based on the second observation data, update the pose state information of the main measurement device in an absolute error state.

3. The data processing method according to claim 2, characterized in that, The updating the pose state information of the main measurement device in a relative error state based on the first observation data includes at least one of the following: Based on the multiple first observation data respectively obtained by the first external observer in multiple first observation frames, determine the observation perturbation information between the same first observation features in the multiple first observation data, and convert the observation perturbation information into the pose error information to update the pose state information of the main measurement device in multiple first observation frames in a relative error state; Based on the multiple first observation data respectively obtained by the first external observer in multiple first observation frames, use the state augmentation information of the first external observer in the current first observation frame to determine the second observation residual information between the same first observation features in the multiple first observation data, so as to update the pose state information of the main measurement device in multiple first observation frames in a relative error state, wherein the state augmentation information includes: the error state information of the main measurement device in the current first observation frame and the error state information of the first external observer in multiple first observation frames.

4. The data processing method according to claim 3, characterized in that, Before the updating the pose state information of the main measurement device in a relative error state based on the first observation data, it further includes: Based on the first measurement data of the main measurement device in each observation frame, determine the error state information of the main measurement device in each first observation frame; Determine the error state information of the first external observer in each first observation frame based on the error state information of the master measurement device in each observation frame and the calibration parameters between the master measurement device and the first external sensor; Perform state augmentation processing based on the error state information of the master measurement device in each first observation frame and the error state information of the first external observer in each first observation frame to determine the state augmentation information of the master measurement device in each observation frame.

5. The data processing method according to claim 3, wherein, The determining the observation perturbation information of the same first observation feature among the multiple first observation data respectively obtained by the first external observer in multiple first observation frames includes: In the multiple first observation data, determine the key observation data corresponding to the key frame and the reference observation data corresponding to the reference frame; According to the pose state information of the first external observer in the key frame and the pose state information in the reference frame, perform feature matching on the key observation data and the reference observation data to determine the observation perturbation information of the same first observation feature in the reference observation data relative to the key observation data.

6. The data processing method according to claim 5, wherein, Before the determining the observation perturbation information of the same first observation feature among the multiple first observation data respectively obtained by the first external observer in multiple first observation frames, further includes: Determine the pose state information of the first external observer in the key frame and the pose state information in the reference frame based on the first measurement data of the master measurement device in the key frame and the first measurement data in the reference frame, and the calibration parameters between the master measurement device and the first external sensor.

7. The data processing method according to claim 5, wherein, The performing feature matching on the key observation data and the reference observation data according to the pose state information of the first external observer in the key frame and the pose state information in the reference frame includes: Based on the reference observation data, determine the first observation feature corresponding to the reference frame; Based on the pose state information of the first external observer in the key frame and the pose state information in the reference frame, perform optical flow tracking on the first observation feature corresponding to the reference frame in the key observation data.

8. The data processing method according to claim 7, wherein, The determining the first observation feature corresponding to the reference frame based on the reference observation data includes at least one of the following: Perform feature extraction on the reference observation data to determine the first observation feature; Obtain object point data in the environment and project the object point data into the reference observation data to determine the first observation feature.

9. The data processing method according to claim 8, wherein, Before the obtaining the object point data in the environment, further includes: Generate map data in the global coordinate system based on the second observation data, and determine object point data based on the map data.

10. The data processing method according to claim 9, wherein, the first observation data is suitable for providing color information, and the method further includes: Project the color information of the first observation feature into the object point data based on the projection relationship between the first observation feature and the object point data for map rendering.

11. The data processing method according to claim 8, wherein, Before performing optical flow tracking on the first observation feature corresponding to the reference frame in the key observation data based on the pose state information of the first external observer in the key frame and the pose state information in the reference frame, it further includes: Obtain the spatial information of the first observation feature corresponding to the reference frame.

12. The data processing method according to claim 11, wherein, The obtaining the spatial information of the first observation feature corresponding to the reference frame includes at least one of the following: Perform feature extraction on the key observation data to determine the first observation feature corresponding to the key frame, and based on the position information of the first observation feature in the key observation data and the position information in the reference observation data, perform triangulation processing to determine the spatial information of the first observation feature corresponding to the reference frame; Obtain the spatial information of the object point data projected into the reference observation data, and determine the spatial information of the first observation feature with a projection relationship.

13. The data processing method according to claim 3, wherein, The converting the observation perturbation information into the pose error information includes: Convert the observation perturbation information into the pose error information according to the perturbation error conversion matrix.

14. The data processing method according to claim 13, wherein, Before the converting the observation perturbation information into the pose error information, it further includes: Determine the key rotation information and the reference rotation information based on the pose state information of the first external observer in the key frame and the pose state information in the reference frame; Determine the perturbation error conversion matrix based on the key rotation information and the reference rotation information.

15. The data processing method according to claim 3, wherein, The method further includes: correcting the state augmentation information based on the pose error information.

16. The data processing method according to claim 2, wherein, Before updating the pose state information of the main measurement device in the absolute error state based on the second observation data of the second external observer, it further includes: Perform motion compensation on the second observation data of each second observation frame based on the first measurement data collected by the main measurement device at the start time of each second observation frame of the second external observer and the first measurement data collected at the end time of each second observation frame.

17. The data processing method according to claim 16, wherein, The motion compensation is performed by a non-uniform B-spline algorithm.

18. The data processing method according to claim 2, It is characterized in that the number of the first external observers is multiple; before updating the pose state information of the main measurement device in the relative error state based on the first observation data, it further includes: selecting one of the multiple first external observers as a reference first external observer, and converting the first observation data of the remaining first external observers to the coordinate system of the reference first external observer.

19. The data processing method according to claim 2, It is characterized in that the number of the second external observers is multiple; before updating the pose state information of the main measurement device in the absolute error state based on the second observation data, it further includes: performing motion compensation on the second observation data of each of the second external observers respectively based on the first measurement data collected by the main measurement device at the start moment of the second observation frame of each of the second external observers and the first measurement data collected at the end moment of the second observation frame; selecting one of the multiple second external observers as a reference second external observer, and converting the second observation data after motion compensation of the remaining second external observers to the coordinate system of the reference second external observer.

20. The data processing method according to any one of claims 2 to 19, It is characterized in that the mobile carrier is further provided with an auxiliary measurement device adapted to measure its own motion state to obtain second measurement data; before updating the pose state information of the main measurement device in the relative error state based on the first observation data, it further includes: determining the relative state information of the auxiliary measurement device based on the second measurement data of the auxiliary measurement device, for combining with the first observation data to update the pose state information of the main measurement device in the relative error state.

21. The data processing method according to claim 20, It is characterized in that the auxiliary measurement device includes at least one of an inertial sensor, a wheel speed sensor, and an encoder.

22. The data processing method according to any one of claims 2 to 19, It is characterized in that the first external observer is an image sensor or a lidar sensor; the second external observer is one of a lidar sensor, a millimeter wave radar sensor, an infrared radar sensor, and a ultrasonic radar sensor.

23. The data processing method according to any one of claims 1 to 19, It is characterized in that the main measurement device includes at least one of an inertial sensor and a wheel speed sensor.

24. A data processing device, It is characterized in that being respectively connected to a main measurement device and an observation device, the main measurement device is configured to measure its own motion state to obtain first measurement data, and the observation device is configured to observe the environment to obtain observation data; the data processing device, the main measurement device, and the observation device are all assembled on a mobile carrier; the data processing device includes: an information acquisition module configured to determine the pose state information of the main measurement device based on the first measurement data of the main measurement device; An information update module, configured to update the pose state information of the main measurement device in an error state based on the observation data of the observation device, so as to generate the positioning information of the mobile carrier.

25. The data processing device according to claim 24, wherein, the data processing device is respectively connected to the first external observer and / or the second external observer in the observation device, wherein the first external observer is configured to observe the environment to obtain first observation data, and the second external observer is configured to observe the environment to obtain second observation data; the information update module includes at least one of the following: a first information update unit, configured to update the pose state information of the main measurement device in a relative error state based on the first observation data; a second information update unit, configured to update the pose state information of the main measurement device in an absolute error state based on the second observation data.

26. An autonomous exploration system, wherein, assembled on a mobile carrier, the autonomous exploration system includes: a main measurement device, configured to measure its own motion state to obtain first measurement data; an observation device, configured to observe the environment to obtain observation data; a data processing device, configured to determine the pose state information of the main measurement device based on the first measurement data of the main measurement device, and update the pose state information of the main measurement device in an error state based on the observation data of the observation device, so as to generate the positioning information of the mobile carrier.

27. A data processing device, comprising: a memory and a processor, wherein computer instructions capable of running on the processor are stored on the memory, and characterized in that when the processor runs the computer instructions, it executes the steps of the method according to any one of claims 1 to 23.

28. A readable storage medium, on which computer instructions are stored, wherein, when the computer instructions run, they execute the steps of the method according to any one of claims 1 to 23.

Images