A method and apparatus for calibrating visual navigation data

Abstract

The specification discloses a visual navigation data calibration method and device. The method determines the conversion relationship between the first coordinate system of the visual navigation data and the second coordinate system of the satellite navigation data according to the calibration data of each historical moment during the driving of the unmanned device, and then converts the visual navigation data into the second coordinate system according to the conversion relationship to calibrate the visual navigation data. The method does not need to fit the trajectory, but determines the conversion relationship according to the calibration data of at least two specified moments. When the cumulative error is small, the accurate conversion relationship can be determined, and the timeliness and efficiency of the visual navigation data calibration are improved.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a visual navigation data calibration method and apparatus. Background Technology

[0002] Currently, autonomous vehicles determine their position and navigate based on data collected by sensors. Common sensors include inertial measurement units (IMUs), cameras, Global Navigation Satellite Systems (GNSS), and gyroscopes.

[0003] Because visual navigation data determined from IMU and imagery contains cumulative errors, it requires frequent calibration. Typically, during autonomous driving, a visual trajectory is determined based on visual navigation data derived from image and IMU data over a specific period. Then, GNSS data from the same time period is acquired to determine the GNSS trajectory. Next, the transformation relationship between the coordinate systems of the visual navigation data and the GNSS data is determined based on the visual and GNSS trajectories. Subsequent calibration of the visual navigation data can then be performed based on this transformation relationship, minimizing the difference between the GNSS and visual navigation data at any given moment. Summary of the Invention

[0004] This specification provides a visual navigation data calibration method and apparatus to partially solve the aforementioned problems existing in the prior art.

[0005] The following technical solution is adopted in this specification:

[0006] This manual provides methods for calibrating visual navigation data, including:

[0007] Acquire calibration data at various historical moments during the operation of the unmanned vehicle, wherein the calibration data includes at least: visual navigation data and satellite navigation data;

[0008] Based on at least two specified times from each historical time, and with the constraint that the positions of the visual navigation data and satellite navigation data of the calibration data at each specified time are the same, the transformation relationship between the first coordinate system of the visual navigation data and the second coordinate system of the satellite navigation data is determined.

[0009] Based on the transformation relationship, the visual navigation data is transformed into the second coordinate system, and the visual navigation data is calibrated.

[0010] Optionally, according to the transformation relationship, the visual navigation data is transformed into the second coordinate system, and the visual navigation data is calibrated, specifically including:

[0011] For each moment after the transformation relationship is determined, the calibration matrix of the visual navigation filter updated according to the previous moment is determined.

[0012] The visual navigation data is calibrated based on the calibration matrix.

[0013] Optionally, the visual navigation data is calibrated according to the calibration matrix, specifically including:

[0014] Based on the calibration matrix, determine the calibration matrix of the visual navigation filter updated in the previous moment, the calibrated visual navigation data, and based on the transformation relationship, determine the difference between the calibrated visual navigation data and the satellite navigation data at this moment;

[0015] Update the calibration matrix of the visual navigation filter based on the gap at that moment;

[0016] The visual navigation data for the next time step is calibrated based on the updated calibration matrix.

[0017] Optionally, the calibration matrix of the visual navigation filter is updated based on the gap at that moment, specifically including:

[0018] Obtain the calibration matrix of the visual navigation filter updated in the previous time step;

[0019] Based on the calibration matrix and the gap, determine the amount of increase in the calibration matrix;

[0020] The calibration matrix of the visual navigation filter is updated according to the increment of the calibration matrix, and the updated result is used as the calibration matrix at that moment.

[0021] Optionally, the visual navigation data is calibrated according to the calibration matrix, specifically including:

[0022] When the satellite navigation data is lost, acquire the visual navigation data at that moment and the calibration matrix of the visual navigation filter at the most recent moment;

[0023] Based on the calibration matrix, determine the increase in the visual navigation data and the increase in the calibration matrix, respectively.

[0024] The visual navigation data is updated according to the increase in the visual navigation data, and the update result is used as the final calibration result of the visual navigation data at that moment. The calibration matrix of the visual navigation filter is updated according to the increase in the calibration matrix, and the update result is used as the calibration matrix at that moment.

[0025] Optionally, based on at least two specified times from each historical time period, and constrained by the fact that the positions of the visual navigation data and satellite navigation data of the calibration data at each specified time period are the same, the transformation relationship between the first coordinate system of the visual navigation data and the second coordinate system of the satellite navigation data is determined, specifically including:

[0026] From various historical moments, determine the calibration data at two specified moments, and use them as the first calibration data and the second calibration data, respectively. The timestamp of the first calibration data is earlier than that of the second calibration data.

[0027] Based on the satellite navigation data and visual navigation data contained in the first calibration data, the initial satellite data and initial visual data contained in the second calibration data, the rotation matrix to be solved and the translation matrix to be solved, the position constraint relationships corresponding to the first calibration data and the second calibration data are determined respectively.

[0028] Based on the positional constraints, the rotation matrix and the translation matrix are solved;

[0029] Based on the determined rotation matrix and translation matrix, the transformation relationship between the first coordinate system and the second coordinate system is determined.

[0030] Optionally, based on the determined rotation matrix and translation matrix, the transformation relationship between the first coordinate system and the second coordinate system is determined, specifically including:

[0031] The determined rotation and translation matrices are used as the initial state variables for calibrating the filter;

[0032] With the goal of minimizing the difference between satellite navigation data and visual navigation data in the calibration data at each historical moment, the rotation matrix and the translation matrix are updated until the stability of the calibration filter is higher than a preset stability threshold.

[0033] Based on the updated rotation and translation matrices, determine the transformation relationship between the first coordinate system and the second coordinate system.

[0034] Optionally, before updating the visual navigation data, the method further includes:

[0035] When the difference between the calibrated visual navigation data and the satellite navigation data acquired at that moment is greater than a preset error threshold, record the number of times the difference between the visual navigation data and the satellite navigation data is greater than the error threshold, and determine whether the number of times is greater than a preset number threshold.

[0036] If so, then it is determined that the transformation relationship between the first coordinate system and the second coordinate system needs to be redefined;

[0037] If not, then the transformation relationship does not need to be redefined.

[0038] This specification provides a visual navigation data calibration device, the device comprising:

[0039] The acquisition module is used to acquire calibration data at various historical moments during the operation of the unmanned vehicle. The calibration data includes at least visual navigation data and satellite navigation data.

[0040] The determination module is used to determine the transformation relationship between the first coordinate system of the visual navigation data and the second coordinate system of the satellite navigation data based on at least two specified times in each historical time, with the constraint that the positions of the visual navigation data and the satellite navigation data of the calibration data at each specified time are the same;

[0041] The calibration module is used to convert the visual navigation data to the second coordinate system according to the transformation relationship and to calibrate the visual navigation data.

[0042] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described visual navigation data calibration method.

[0043] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described visual navigation data calibration method.

[0044] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0045] In the visual navigation data calibration method provided in this specification, the transformation relationship between the first coordinate system of visual navigation data and the second coordinate system of satellite navigation data is determined based on the calibration data at each historical moment during the operation of the unmanned vehicle. Then, based on the transformation relationship, the visual navigation data is transformed into the second coordinate system and the visual navigation data is calibrated.

[0046] As can be seen from the above method, this method does not require trajectory fitting when determining the first coordinate system and the second coordinate system. Instead, it determines them based on calibration data at at least two specified times. This allows for the determination of an accurate transformation relationship when the cumulative error is small. Based on this transformation relationship, the difference between visual navigation data and satellite navigation data is determined. The calibration matrix of the visual navigation filter is then updated based on this difference. This ensures that even when satellite navigation data is lost, the accurate position of the unmanned vehicle can still be determined, thus improving the timeliness and efficiency of visual navigation data calibration. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart illustrating the visual navigation data calibration method provided in this manual.

[0049] Figure 2 The visual navigation data calibration device provided in this manual;

[0050] Figure 3 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0052] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0053] Generally, to ensure the safety of autonomous vehicles, they determine their position and navigate using data collected by various sensors during movement. Among these sensors, satellite positioning data is generally considered the most accurate and is therefore primarily used for navigation control. However, satellite positioning relies on satellite signals obtained from the Global Navigation Satellite System (GNSS). Therefore, when the autonomous vehicle is in an area with blocked satellite signals, it may be unable to acquire satellite positioning data, or the acquired data may be inaccurate.

[0054] At this point, navigation can be assisted by other navigation devices within the autonomous vehicle until it leaves the area where satellite signals are blocked and resumes receiving satellite positioning data. Cameras and IMUs are two such navigation devices. However, since visual navigation data is determined based on image data and IMU data, and positioning based on visual navigation data involves comparing historical visual navigation data with current visual navigation data and performing feature point matching, errors are inevitable. Without the constraint of real position information, these errors accumulate, causing the autonomous vehicle to deviate from its intended position when using visual navigation data to assist navigation. Therefore, calibration of the visual navigation data is necessary.

[0055] Common methods for calibrating visual navigation data include adding loop closure detection and calibration based on satellite navigation data.

[0056] A common method for calibrating visual navigation data by adding loop closure detection is based on the bag-of-words model. Specifically, it typically requires acquiring image data from the current moment and from each historical moment of the autonomous vehicle. Then, the bag-of-words model is invoked to calculate the similarity between the current image data and the image data from each historical moment. Based on preset target conditions, the image data from historical moments whose similarity to the current image data meets the target conditions is used as the loop closure image data. Finally, based on the relative pose between the current image data and the loop closure image data, the pose determined from the visual navigation data of the autonomous vehicle at the current moment is corrected.

[0057] However, the method of adding loop closure detection requires a large amount of computational resources due to the need to build a bag-of-words model and perform similarity matching between multiple image data, making it inconvenient to deploy in autonomous driving equipment.

[0058] The calibration method based on satellite navigation data can be implemented in the following two ways:

[0059] The first method involves the unmanned vehicle acquiring visual navigation data and satellite navigation data determined at historical moments.

[0060] Secondly, assuming that the transformation relationship between the first coordinate system containing visual navigation data and the second coordinate system containing satellite navigation data is an identity matrix at the initial time, a constraint is constructed based on this transformation relationship: "the difference between visual navigation data and satellite navigation data at each time step is minimized, and the difference between the change in visual navigation data and the change in satellite navigation data from any time step to that time step is minimized." The variable corresponding to this constraint is the projection of the visual navigation data into the second coordinate system.

[0061] Then, the variables are solved according to the constraints to determine the projection of the visual navigation data in the second coordinate system at that moment. Based on the projection and the visual navigation data at that moment, the transformation relationship at the next moment is determined until the transformation relationship converges.

[0062] Finally, based on this transformation relationship, the autonomous driving device can calibrate the acquired visual navigation data from various historical moments. One common method for calibrating visual navigation data in this manner is based on an optimized visual-inertial system fusion (VINS-Fusion) method.

[0063] However, the above method uses the identity matrix as the initial value when constructing constraints, meaning it uses inaccurate transformation relationships to construct constraints. This means that an accurate transformation relationship can only be determined when the transformation relationship converges. Therefore, existing technologies require a large amount of data to determine an accurate transformation relationship, and the convergence time of the transformation relationship may vary in different scenarios, resulting in poor calibration performance when using the above method.

[0064] The second approach involves fitting visual navigation trajectories and GNSS trajectories to the visual navigation data and satellite navigation data determined from historical moments of the autonomous vehicle. A transformation relationship is then established based on the fitted visual navigation and GNSS trajectories, and this relationship is used to calibrate the visual navigation data. A common method for calibrating visual navigation data using this approach is the Open-Source Visual-Inertial System Fusion (Open-VINS) method, which is based on filtering.

[0065] However, existing technologies do not require entirely the same track length for different motion trajectories when determining the conversion relationship between visual and GNSS trajectories. For example, when the motion trajectory is a straight line, the required data is 3 seconds of visual and GNSS data, while when the motion trajectory is an ellipse, the required data is 10 seconds of visual and GNSS data. Clearly, when data is collected for 3-second intervals, it is impossible to determine an accurate conversion relationship based on the visual trajectory and satellite navigation data when the motion trajectory is elliptical, resulting in poor calibration performance in existing technologies.

[0066] Furthermore, the trajectory of autonomous vehicles is unknown in different scenarios, therefore, the data length required to fit the corresponding trajectory is unknown. If the acquisition time is too short, it is impossible to fit and determine an accurate visual trajectory and GNSS trajectory. If the acquisition time is too long, due to the influence of accumulated errors, it is impossible to determine an accurate visual trajectory, and thus impossible to obtain an accurate conversion relationship. Consequently, when calibration is performed based on this conversion relationship, the calibration results will not be satisfactory.

[0067] Therefore, in order to avoid the problems existing in the visual navigation data calibration of the prior art, this specification provides a visual navigation data calibration method.

[0068] Figure 1 This is a flowchart illustrating the visual navigation data calibration method provided in this manual, which specifically includes the following steps:

[0069] S100: Acquire calibration data at various historical moments during the operation of the unmanned vehicle. The calibration data includes at least visual navigation data and satellite navigation data.

[0070] Unlike existing technologies that require fitting the visual trajectory and GNSS trajectory of an autonomous vehicle to determine the transformation relationship between the coordinate systems of visual navigation data and satellite navigation data, this specification provides a novel visual navigation data calibration method. This method eliminates the need to fit visual and GNSS trajectories. Instead, it determines the transformation relationship between the coordinate systems of visual navigation data and satellite navigation data by analyzing the mapping relationship between the two at a specified time. Based on this transformation relationship, and using the satellite navigation data as a reference, the visual navigation data is then calibrated.

[0071] In one or more embodiments provided in this specification, during the operation of the autonomous driving device, the device can acquire calibration data at various times according to a preset frequency. This calibration data is required for calibrating visual navigation data and includes at least: visual navigation data determined by images and IMU data, that is, uncalibrated visual navigation data determined by the Visual-Inertial System (VINS) based on image data and IMU data. Alternatively, the autonomous driving device can send the acquired calibration data to a server, which will then perform subsequent steps to determine the calibration matrix of the visual filter. For ease of description, the following explanation will use the autonomous driving device performing this visual navigation data calibration process as an example.

[0072] Specifically, the autonomous vehicle can first acquire its own collected images, IMU data, and received satellite navigation data. Since the IMU data acquisition frequency is higher than the image data acquisition frequency, the autonomous vehicle can acquire IMU data from the acquisition of the previous image frame to the acquisition of the current image frame. Alternatively, the autonomous vehicle can also acquire IMU data from a second timestamp greater than the acquisition time of the previous image frame, to the IMU data from a first timestamp greater than the timestamp of the current image frame.

[0073] Secondly, the autonomous driving device can determine the visual navigation data corresponding to a given moment based on the image, the IMU data, and the timestamp corresponding to the image.

[0074] Then, since the differences between the data can only be determined under the same coordinate system, the unmanned vehicle can perform coordinate system transformation on the received satellite navigation data, transforming the received satellite navigation data in latitude, longitude and altitude coordinates into the North East Down (NED) coordinate system.

[0075] Finally, the autonomous vehicle can store the determined visual navigation data and the satellite navigation data corresponding to the NED coordinate system based on the timestamps corresponding to the images. Of course, since the image data and satellite navigation data are acquired almost simultaneously, the calibration data can also be stored based on the timestamps corresponding to the satellite navigation data.

[0076] The aforementioned method of determining visual navigation data based on images and IMU information can involve extracting features from the image, identifying corresponding feature points, tracking these feature points, filtering out noise, pre-integrating the IMU data, fusing the image's feature points and the pre-integrated IMU data, and then obtaining the current state of the autonomous vehicle based on the fused data. This can be represented as follows: Among them, X V This represents the state variables of the autonomous driving equipment in the first coordinate system, where V represents the first coordinate system where the visual navigation data is located. Ii represents the position of the IMU in the first coordinate system in the i-th frame, and Ii represents the IMU data in the i-th frame. This represents the attitude of the IMU in the first coordinate system at frame i. This represents the velocity of the IMU in the first coordinate system in the i-th frame. This represents the bias of the IMU accelerometer in the first coordinate system. This represents the bias of the gyroscope in the first coordinate system.

[0077] Of course, the autonomous driving device can also perform triangulation and pose calculation based on the extracted feature points of the image and IMU data to determine the state variables of the autonomous driving device in the first coordinate system.

[0078] Furthermore, the aforementioned step of performing coordinate transformation on the received satellite navigation data in the second coordinate system to determine the NED coordinate system can be as follows: the unmanned vehicle uses the pre-stored conversion relationship between the latitude, longitude, and altitude coordinate system and the North-Eastern coordinate system to perform coordinate transformation on the received satellite navigation data and determine the position of the satellite navigation data in the NED coordinate system; or it can be that the satellite navigation data is first transformed to the geocentric coordinate system and then transformed from the geocentric coordinate system to the NED coordinate system to determine its corresponding position information in the NED coordinate system.

[0079] Furthermore, the autonomous driving device can also determine the visual navigation data corresponding to each historical moment based on the images and IMU data corresponding to each historical moment, and store the visual navigation data determined at each historical moment and the satellite navigation data received at each historical moment, so that when the visual navigation calibration method needs to be executed, subsequent steps can be performed based on the stored visual navigation data and satellite navigation data.

[0080] S102: Based on the calibration data of at least two specified times in each historical time, and with the constraint that the positions of the visual navigation data and the satellite navigation data of the calibration data at each specified time are the same, determine the transformation relationship between the first coordinate system of the visual navigation data and the second coordinate system of the satellite navigation data.

[0081] In one or more embodiments provided in this specification, the difference between the visual navigation data and satellite navigation data of the autonomous vehicle can only be compared within the same coordinate system. For the autonomous vehicle, since its pitch and roll angles can be measured, the transformation relationship between the first coordinate system containing the visual navigation data and the second coordinate system containing the satellite navigation data can be determined once the relationship between the yaw angle and the origin coordinates is known. Furthermore, for two historical moments, the difference between the corresponding yaw angle and the origin coordinates can be determined using the positions corresponding to those two historical moments as constraints.

[0082] Specifically, based on the above, the unmanned driving device can first determine the transformation relationship between the first coordinate system and the second coordinate system as follows: Where G represents GNSS data, i.e., the second coordinate system in which satellite navigation data resides; V represents the first coordinate system in which visual navigation data resides; and Gi represents the i-th frame of satellite navigation data. This is a rotation matrix, related to the yaw angle. Let be the difference between the origins of the first and second coordinate systems, i.e., the translation matrix. Let i be the position of the i-th frame of satellite navigation data in the second coordinate system. Let be the position of the i-th frame of satellite navigation data in the first coordinate system. For the first frame and the i-th frame of satellite navigation data, we have: Right now, and Let represent the distance traveled by the i-th frame GNSS relative to the first frame GNSS in the second coordinate system and the first coordinate system, respectively. Then, the yaw angle can be determined, and consequently, the rotation matrix can be determined. Based on the above transformation relationship, it can be determined that Then the translation matrix can be determined. The position of GNSS data in the first coordinate system can be determined based on the visual navigation data at the same moment. The translation matrix is ​​the difference between the origins of the first and second coordinate systems, denoted as (x, y, z).

[0083] Then based on the determined and This allows us to determine the transformation relationship between the first and second coordinate systems.

[0084] Furthermore, since the observation matrix determined based on the two calibration data may not be accurate enough, the unmanned vehicle can also update the rotation and translation matrices based on the calibration filter.

[0085] Specifically, the unmanned vehicle can use the determined yaw angle and translation matrix used to determine the rotation matrix as the initial state variable X for calibrating the filter. e =(yaw e ,x e ,y e ,z e ), among which, yaw e For the yaw angle determined above, x e ,y e ,z e These are (x, y, z) in the translation matrix determined above.

[0086] Secondly, the unmanned vehicle can determine the observation matrix corresponding to each time point based on the satellite navigation data and visual navigation data in the calibration data of each historical time point. Among them, X m =(yaw m ,x m ,y m ,z m The observation matrix H1 represents the observation values ​​corresponding to each historical moment, that is, the yaw angle and position determined by the satellite navigation data corresponding to each historical moment. The observation matrix H1 represents the derivative of the observation values ​​with respect to the state variables.

[0087] Then, the autonomous driving device can determine the update amount d of the state variables at each time step based on the observation matrix. x =C*H1 T *(H1*C*H1 T +R) -1 *(X m -X e ), and the update amount f of the covariance at each time point. x =-C*H1 T *(H1*C*H1 T +R) -1 *H1*C T Among them, d x H1 is the update quantity of the state variable, C is the observation matrix, C is the covariance matrix corresponding to the previous time step of the filter, which can be determined by the transformation relationship between the first coordinate system and the second coordinate system determined in step S102, T represents transpose, R is the covariance matrix of the observation noise, which can be determined according to the horizontal component precision factor (HDOP), the vertical component precision factor (VDOP), and the yaw angle error Δyaw of the unmanned vehicle. HDOP and VDOP can be obtained by GNSS signal measurement or determined by GNSS calculation, and Δyaw can be determined by preset by experience.

[0088] Finally, the autonomous driving device can update the state variables and covariance matrix at each time step based on the determined increase in the state variables and covariance matrix, until the stability of the calibration filter is higher than a preset stability threshold. The state variables include rotation and translation matrices; that is, for each time step, the autonomous driving device can redetermine the corresponding state variable X based on the increase in the state variables and covariance matrix. e +d x And the covariance matrix C+f at that moment. x Then, based on the redefined covariance matrix, it is determined whether the calibration filter is stable. When the calibration filter is stable, the determined state variables are used as the yaw angle and translation matrix to determine the transformation relationship between the first and second coordinate systems. Based on the updated yaw angle and translation matrix, the unmanned vehicle can determine the transformation relationship between the first and second coordinate systems.

[0089] S104: Based on the transformation relationship, the visual navigation data is transformed into the second coordinate system, and the visual navigation data is calibrated.

[0090] In one or more embodiments provided in this specification, since the coordinate systems of the visual navigation data and the satellite navigation data are different, and the visual navigation data can only be calibrated within the same coordinate system, the autonomous driving device can, after determining the transformation relationship between the first and second coordinate systems, transform the visual navigation data into the second coordinate system based on this transformation relationship to calibrate the visual navigation data.

[0091] Specifically, the unmanned vehicle can, for each moment after the conversion relationship is determined, convert the visual navigation data at that moment to the second coordinate system where the satellite navigation data is located, and determine the difference between the visual navigation data and the satellite navigation data based on the received satellite navigation data.

[0092] Based on this gap, the server can determine the variation pattern of the gap between the visual navigation data and the satellite navigation data at each time point, and continue to calibrate the subsequently collected calibration data based on this variation pattern.

[0093] Furthermore, as mentioned above, the visual data navigation calibration method in this specification is applied to scenarios where filters are used to calibrate visual navigation data. For a filter, its corresponding calibration matrix typically includes the filter's covariance matrix, the observation noise covariance matrix, etc. Moreover, during the operation of an autonomous vehicle, since the calibration parameters at each moment affect the parameters of the visual navigation filter, for each moment, the visual navigation data located in the second coordinate system at that moment will be affected by the calibration matrix of the filter from the previous moment.

[0094] Therefore, when determining the difference between visual navigation data and satellite navigation data at a given moment, the autonomous vehicle can first determine the calibration matrix of the visual navigation filter from the previous moment. Based on the covariance matrix and noise covariance matrix in this calibration matrix, it transforms the visual navigation data in the first coordinate system corresponding to the current moment to the second coordinate system, thus determining the difference between the visual navigation data and satellite navigation data at that moment. In other words, the visual navigation data is calibrated according to the calibration matrix of the visual navigation filter updated in the previous moment.

[0095] Furthermore, as mentioned earlier, since the coordinate systems of visual navigation data and satellite navigation data are different, and the differences between data can only be compared within the same coordinate system, the autonomous driving device can determine the differences between each visual navigation data and satellite navigation data based on the predetermined transformation relationship between the first and second coordinate systems, and then calibrate the calibration matrix of the visual navigation filter based on the differences.

[0096] Specifically, for each moment after the transformation relationship is determined, the autonomous driving device can determine the visual navigation data at that moment, and transform the visual navigation data from the first coordinate system to the second coordinate system according to the predetermined transformation relationship.

[0097] Taking the conversion of visual navigation data to a second coordinate system as an example, the autonomous driving device can first obtain the conversion relationship between the coordinate system where the visual navigation data is located and the coordinate system where the satellite navigation data is located. Then, for each calibration data, according to the conversion relationship, the visual navigation data in the calibration data is converted to the second coordinate system, thus determining the visual navigation data in the second coordinate system. in, G represents the second coordinate system, Ii is the IMU data of the i-th frame, and X... G This represents the state variables of the unmanned driving equipment in the second coordinate system. This represents the position of the IMU in the second coordinate system in the i-th frame. This represents the attitude of the IMU in the second coordinate system in the i-th frame. This represents the velocity of the IMU in the second coordinate system in the i-th frame. This represents the offset of the IMU accelerometer in the second coordinate system. This represents the bias of the gyroscope in the second coordinate system. Let be the rotation matrix from the first coordinate system to the second coordinate system. This is the translation matrix between the first and second coordinate systems.

[0098] Then, after determining the visual navigation data in the second coordinate system, the autonomous vehicle can determine the difference between the visual navigation data and the satellite navigation data. This difference can be the positional difference between the two, etc. It means that t m =(x m y m , z m ) represents the observation values ​​corresponding to the calibration data at each time point, that is, the positions of the satellite navigation data at each time point. The observation matrix H represents the derivative of the observation value with respect to the state variables.

[0099] Finally, based on this gap, the autonomous vehicle can update the visual navigation filter to calibrate the visual navigation data for the next moment based on the updated calibration matrix.

[0100] Furthermore, as mentioned earlier, the visual data navigation calibration method in this specification is applied to scenarios where visual navigation data is calibrated using filters. For a filter, its corresponding calibration matrix typically includes the filter's covariance matrix, the observation noise covariance matrix, etc. Moreover, during the operation of the autonomous vehicle, the calibration matrices corresponding to the filters for each calibration data point are often different. Therefore, the autonomous vehicle can determine the calibration matrix corresponding to the filter for each calibration data point based on the difference between the visual navigation data and satellite navigation data at each time point.

[0101] Specifically, for each moment, the autonomous vehicle can obtain the calibration matrix of the visual navigation filter updated in the previous moment.

[0102] Secondly, the unmanned vehicle can determine the visual navigation data in the second coordinate system based on the calibration matrix, and determine the observation matrix H2 at that moment based on satellite navigation data. This observation matrix can be determined using the formula... Confirmed, t m =(x m y m , z m X represents the position of the unmanned equipment determined by satellite navigation data in the second coordinate system. G The visual navigation data located in the second coordinate system as determined in step S104. The observation matrix represents the derivative of the observation with respect to the state variable, which is the difference between the observation and the state variable. The difference can be used to characterize the calibration matrix.

[0103] Then, the autonomous driving device can also determine the increase in the covariance matrix contained in the calibration matrix based on the calibration matrix of the visual navigation filter obtained in the previous moment and the determined observation matrix.

[0104] Finally, the autonomous driving device can determine the increase f of the covariance matrix based on the calculated increase f. y For the covariance matrix C included in the calibration matrix G Perform the update and confirm the update result C. G +f y The updated result is then used as the covariance matrix at that moment.

[0105] The calibration matrix of the visual navigation filter may include at least the covariance matrix of the filter, and may also include the observation matrix determined based on the visual navigation data and satellite navigation data, as well as the covariance matrix of the observation noise.

[0106] The increase in the covariance matrix mentioned above can be calculated using formula f. y =-C G *H2T *(H2*C G *H2 T +R) -1 *H2*C G T Determined. Where f y The update amount is the covariance matrix, where T represents the matrix transpose, and C... G R is the covariance matrix corresponding to the previous moment of the visual navigation filter, where T represents the transpose and R is the covariance matrix of the observation noise. It can be determined based on the error Δyaw of HDOP, VDOP, and the yaw angle of the unmanned vehicle. HDOP and VDOP can be obtained by GNSS signal measurement or determined by GNSS calculation. Δyaw can be preset by experience.

[0107] based on Figure 1 The provided visual navigation data calibration method determines the transformation relationship between a first coordinate system for visual navigation data and a second coordinate system for satellite navigation data based on calibration data from historical moments during the operation of the autonomous vehicle. Then, according to this transformation relationship, the visual navigation data is transformed into the second coordinate system for calibration. This method does not require trajectory fitting when determining the first and second coordinate systems; instead, it uses calibration data from at least two specified moments. This allows for accurate determination of the transformation relationship when the cumulative error is small. Based on this transformation relationship, the difference between the visual navigation data and the satellite navigation data is determined, and the calibration matrix of the visual navigation filter is updated according to this difference. This enables the accurate determination of the autonomous vehicle's position even when satellite navigation data is lost, improving the timeliness and efficiency of visual navigation data calibration.

[0108] It should be noted that, compared to methods that add loop closure detection, this scheme does not require pre-building a bag-of-words model or performing similarity matching between multiple image datasets. This results in significantly fewer computational resources required compared to methods that add loop closure detection, making it more suitable for deployment in autonomous vehicles. Compared to schemes that update the transformation relationship based on the initial assumption of an identity matrix, this scheme requires less calibration data to determine the transformation relationship, and the required data volume remains consistent across various environments, making it more flexible and ensuring the timeliness and efficiency of visual navigation data calibration.

[0109] Furthermore, since the VINS system typically uses filters to process the visual navigation data located in the first coordinate system based on IMU data and images, in step S104, the autonomous vehicle can also adjust the parameters in the VINS system so that after determining the visual navigation data located in the first coordinate system, it can also transform the visual navigation data to the second coordinate system. The difference can then be determined based on the second coordinate system after transformation. In other words, the calibration matrix of the filter used in the VINS system can be adjusted to make it a visual navigation filter.

[0110] The initial state variable of the filter can be the visual navigation data X in the second coordinate system as described above. G The initial covariance matrix of the filter can be obtained by formula Determined, among which, C v This is the covariance matrix of the filters used in the VINS system. Among them, I 3*3 It is a 3x3 identity matrix, 0 3*3 It is a 3x3 matrix of zeros. This is the rotation matrix between the first and second coordinate systems.

[0111] It should be noted that when this moment is the first moment after the transformation relationship is determined, the unmanned vehicle can determine the initial calibration matrix of the visual navigation filter based on the transformation relationship, and then determine the difference between the satellite navigation data and the visual navigation data at this moment based on the initial calibration matrix. Furthermore, the step of determining the difference between the visual navigation data and the satellite navigation data can also involve transforming the satellite navigation data to a first coordinate system to determine the difference between the visual navigation data and the satellite navigation data at each moment. The specific method for determining the difference can be set as needed, and this specification does not limit it.

[0112] In addition, while updating the calibration matrix of the visual navigation filter, the autonomous driving device can also update the state variable X of the visual navigation filter. G The update involves calibrating the visual navigation data at that moment based on the satellite navigation data at that moment.

[0113] Specifically, this unmanned driving device can be based on the formula Determine the update amount of the state variables of the visual navigation filter, where d y H2 is the update amount of the visual navigation filter, H2 is the observation matrix, T represents the matrix transpose, and C is the update amount of the visual navigation filter. G R is the covariance matrix of the visual navigation filter at the previous time step, and R is the covariance matrix of the observation noise.

[0114] Then, the autonomous driving device can update the state quantity corresponding to the current moment based on the update amount, which serves as the final calibration result of the visual navigation data at that moment.

[0115] After updating the calibration matrix of the visual navigation filter at that moment, the server can determine the gap between the calibrated visual navigation filter and the satellite navigation filter at the next moment based on the calibration matrix, and continue to update the calibration matrix of the visual navigation filter according to the determined gap.

[0116] Furthermore, since VINS systems typically use an Extended Kalman Filter (EKF) to process visual navigation data, after determining the visual navigation data located in the second coordinate system, the autonomous vehicle can use the EKF to predict the visual navigation data and determine the predicted visual navigation data X. G The visual navigation data can be determined by the EKF filter based on the IMU data between the previous and current times.

[0117] Then, the autonomous vehicle can use the predicted visual navigation data X G Based on the satellite navigation data at that time, the observation matrix H3 is determined. This observation matrix can be obtained using the formula... Confirmed, t m =(x m y m , z m The coordinate system represents the location of the unmanned vehicle determined by satellite navigation data in the second coordinate system. This represents the derivative of the observed value with respect to the state variable.

[0118] Finally, based on the determined observation matrix and the determined gap, the autonomous driving device can determine the increase in the state variables and the increase in the covariance matrix. Then, based on the determined increase in the covariance matrix, the covariance matrix is ​​updated, and the updated result is used as the calibration matrix for the updated visual navigation filter at that moment. Furthermore, based on the update of the predicted variables, the predicted variables are updated, and the updated result is used as the calibration result for the visual navigation data at the current moment.

[0119] Furthermore, this autonomous driving device can also use a symmetric formula to transform the redefined covariance matrix. Symmetrical values ​​are used as one of the calibration matrices for the visual navigation filter at the current moment. The autonomous vehicle can then calibrate the calibration data for the next moment based on the updated calibration matrix. Wherein, C... G ′ represents the covariance matrix after symmetry, C GThis is the covariance matrix in the calibration matrix corresponding to the visual navigation filter at the current moment. That is, the updated covariance matrix is ​​determined based on the increase in the covariance matrix and the covariance matrix in the calibration matrix updated at the previous moment.

[0120] The visual navigation filter with the determined calibration matrix can predict the position of the autonomous vehicle at the next moment based on the current visual navigation data of the autonomous vehicle, and redetermine the calibration matrix of the filter based on the satellite navigation data received at the next moment and the predicted position at the next moment.

[0121] Furthermore, when satellite navigation data is lost, the autonomous vehicle can acquire the current visual navigation data and the calibration matrix of the visual navigation filter from the previous moment. Next, the autonomous vehicle can calibrate the visual navigation data based on each calibration matrix. Then, the autonomous vehicle can determine the increment of the visual navigation data based on the covariance matrix in the calibration matrix. Finally, the autonomous vehicle can update the visual navigation data according to the increment and use the updated result as the calibration result for navigation based on the calibrated visual navigation data.

[0122] Of course, as mentioned above, the autonomous driving device can determine the weights corresponding to the current time and each historical time based on the EKF filter of the VINS system, based on the visual navigation data input to the VINS system at the current time and in the past, and perform a weighted sum based on the weights corresponding to each time and the visual navigation data to determine the increase in the visual navigation data at the current time, and update the visual navigation data based on the increase.

[0123] Furthermore, the process of calibrating visual navigation data using the visual navigation filter can include two processes: prediction and updating the prediction results based on satellite navigation data. Therefore, in the event of loss of satellite navigation data, the autonomous vehicle can perform only the prediction process and update the filter parameters based on the visual navigation data obtained during the prediction process.

[0124] It should be noted that the steps described above for updating the calibration matrix of the visual navigation filter, when GNSS data is not lost, typically include using the visual navigation data in the second coordinate system at that moment as the initial correction value, then determining the correction increment based on the difference between the visual navigation data and the satellite navigation data, and finally determining the final correction value based on this state value and the state update value. Then, based on the satellite navigation data at that moment and the final correction value, the calibration matrix corresponding to the visual navigation filter at that moment is determined. Therefore, when GNSS data is lost, the unmanned vehicle can calibrate each visual navigation data point based on the calibration matrix of the most recent moment to determine the accurate position of the unmanned vehicle.

[0125] In addition, due to factors such as cumulative errors, there may be situations where errors cannot be eliminated. Therefore, during the operation of the unmanned driving equipment, it is possible to determine whether the conversion relationship needs to be redefined.

[0126] Specifically, the autonomous driving device can calibrate the determined visual navigation data based on the most recently updated calibration matrix. Then, the autonomous driving device can determine the difference between the calibrated visual navigation data and the acquired satellite navigation data. If the difference is greater than a preset error threshold, the autonomous driving device can consider the error of the determined conversion relationship to be large, and the autonomous driving device can then redetermine the conversion relationship.

[0127] Furthermore, during the operation of the autonomous vehicle, issues such as vehicle vibration may occur, leading to significant discrepancies between the determined visual navigation data and satellite navigation data. These vibrations typically subside within a short period. Therefore, to improve the accuracy of the judgment, the autonomous vehicle can record the number of times the difference between the visual navigation data and satellite navigation data exceeds an error threshold. When the difference is less than the error threshold, the conversion relationship is considered correct. Conversely, when the number of times the difference exceeds the error threshold reaches a preset threshold, the determined conversion relationship is considered unreliable.

[0128] Furthermore, during the operation of the autonomous driving device, there may be instances where the transformation relationship between the first and second coordinate systems is accurate, but due to the prolonged uncertainty in determining this transformation relationship, the cumulative number of times the difference between the visual navigation data and the satellite navigation data exceeds an error threshold reaches a preset threshold. To avoid this situation, the autonomous driving device can clear the recorded number of times the difference between the visual navigation data and the satellite navigation data exceeds the error threshold when the difference is less than the error threshold, i.e., when the transformation relationship is considered correct. In other words, the autonomous driving device can only record the number of times the difference between the visual navigation data and the satellite navigation data exceeds the error threshold at consecutive moments.

[0129] Of course, the autonomous driving device can also preset a duration threshold to determine whether the conversion relationship is reliable. That is, the autonomous driving device can determine, based on the duration threshold, the number of times within the duration threshold when the difference between the calibrated visual navigation data and the satellite navigation data at each moment exceeds the preset duration threshold, and record this number. It can also determine whether this number exceeds a preset number threshold. If it does, the autonomous driving device considers the conversion relationship unreliable. If it does not exceed the threshold, the autonomous driving device considers the conversion relationship still reliable. Specific number thresholds, error thresholds, and duration thresholds can be set as needed, and this manual does not impose any restrictions on them.

[0130] The above describes one or more embodiments of the visual navigation data calibration method provided in this specification. Based on the same idea, this specification also provides a corresponding visual navigation data calibration device, such as... Figure 2 As shown.

[0131] Figure 2 The visual navigation data calibration device provided in this specification includes:

[0132] The acquisition module 200 is used to acquire calibration data at various historical moments during the operation of the unmanned vehicle. The calibration data includes at least visual navigation data and satellite navigation data.

[0133] The determination module 202 is used to determine the transformation relationship between the first coordinate system of the visual navigation data and the second coordinate system of the satellite navigation data based on at least two specified times in each historical time, with the constraint that the positions of the visual navigation data and the satellite navigation data of the calibration data at each specified time are the same.

[0134] The calibration module 204 is used to convert the visual navigation data to the second coordinate system according to the conversion relationship and to calibrate the visual navigation data.

[0135] Optionally, the calibration module 204 is used to sequentially determine, for each time after the transformation relationship is determined, a calibration matrix of the visual navigation filter updated according to the previous time of that time, and to calibrate the visual navigation data according to the calibration matrix.

[0136] Optionally, the calibration module 204 is configured to determine, based on the calibration matrix, the calibration matrix of the visual navigation filter updated at the previous moment, the calibrated visual navigation data, and based on the conversion relationship, determine the difference between the calibrated visual navigation data and the satellite navigation data at the current moment, update the calibration matrix of the visual navigation filter based on the difference at the current moment, and calibrate the visual navigation data at the next moment based on the updated calibration matrix.

[0137] Optionally, the calibration module 204 is configured to obtain the calibration matrix of the visual navigation filter updated at the previous moment, determine the increase amount of the calibration matrix based on the calibration matrix and the difference, update the calibration matrix of the visual navigation filter based on the increase amount of the calibration matrix, and use the update result as the calibration matrix at the current moment.

[0138] Optionally, the calibration module 204 is configured to, when the satellite navigation data is lost, acquire the visual navigation data at that moment and the calibration matrix of the visual navigation filter at the most recent moment, determine the increase in the visual navigation data and the increase in the calibration matrix based on the calibration matrix, update the visual navigation data based on the increase in the visual navigation data, and use the update result as the final calibration result of the visual navigation data at that moment, and update the calibration matrix of the visual navigation filter based on the increase in the calibration matrix, and use the update result as the calibration matrix at that moment.

[0139] Optionally, the determining module 202 is configured to determine calibration data at two specified times from various historical times, respectively serving as first calibration data and second calibration data, wherein the timestamp of the first calibration data is earlier than that of the second calibration data; determine the positional constraint relationships corresponding to the first calibration data and the second calibration data based on the satellite navigation data and visual navigation data contained in the first calibration data, the initial satellite data and initial visual data contained in the second calibration data, the rotation matrix to be solved, and the translation matrix to be solved; solve the rotation matrix and the translation matrix based on each positional constraint relationship; and determine the transformation relationship between the first coordinate system and the second coordinate system based on the determined rotation matrix and the translation matrix.

[0140] Optionally, the calibration module 204 is configured to use the calibrated visual navigation data at that moment as the initial correction amount, determine the correction increment corresponding to the visual navigation data at that moment based on the gap, determine the calibration matrix corresponding to the visual navigation filter at that moment based on the final correction amount determined by adding the initial correction amount and the correction increment, and the satellite navigation data at that moment, and update the calibration matrix of the visual navigation filter based on the determined calibration matrix.

[0141] Optionally, the determining module 202 is used to use the determined rotation matrix and translation matrix as the initial state variables of the calibration filter, and to update the rotation matrix and translation matrix with the goal of minimizing the difference between satellite navigation data and visual navigation data in the calibration data at each historical moment, until the stability of the calibration filter is higher than a preset stability threshold. Based on the updated rotation matrix and translation matrix, the transformation relationship between the first coordinate system and the second coordinate system is determined.

[0142] Optionally, the determining module 202 is used to record the number of times the difference between the calibrated visual navigation data and the acquired satellite navigation data is greater than the error threshold, and to determine whether the number of times is greater than a preset number threshold. If yes, it is determined that the transformation relationship between the first coordinate system and the second coordinate system needs to be re-determined; otherwise, it is determined that the transformation relationship does not need to be re-determined.

[0143] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided visual navigation data calibration method.

[0144] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided visual navigation data calibration method.

[0145] This instruction manual also provides Figure 3 The diagram shows a schematic structural representation of the electronic device. Figure 3 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The visual navigation data calibration method described above. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0146] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0147] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0148] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0149] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0150] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0155] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0156] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0157] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0160] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0161] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A visual navigation data calibration method, characterized in that, The method includes: Acquire calibration data at various historical moments during the operation of the unmanned vehicle, wherein the calibration data includes at least: visual navigation data and satellite navigation data; Based on at least two specified times from each historical time, and with the constraint that the positions of the visual navigation data and satellite navigation data of the calibration data at each specified time are the same, the transformation relationship between the first coordinate system of the visual navigation data and the second coordinate system of the satellite navigation data is determined. Based on the transformation relationship, the visual navigation data is transformed into the second coordinate system, and the visual navigation data is calibrated. Specifically, based on at least two specified times from each historical time period, and constrained by the fact that the positions of the visual navigation data and satellite navigation data of the calibration data at each specified time period are the same, the transformation relationship between the first coordinate system of the visual navigation data and the second coordinate system of the satellite navigation data is determined, including: From various historical moments, determine the calibration data at two specified moments, and use them as the first calibration data and the second calibration data, respectively. The timestamp of the first calibration data is earlier than that of the second calibration data. Based on the satellite navigation data and visual navigation data contained in the first calibration data, the initial satellite data and initial visual data contained in the second calibration data, the rotation matrix to be solved and the translation matrix to be solved, the position constraint relationships corresponding to the first calibration data and the second calibration data are determined respectively. Based on the positional constraints, the rotation matrix and the translation matrix are solved; Based on the determined rotation matrix and translation matrix, the transformation relationship between the first coordinate system and the second coordinate system is determined.

2. The method as described in claim 1, characterized in that, Based on the transformation relationship, the visual navigation data is transformed to the second coordinate system, and the visual navigation data is calibrated, specifically including: For each moment after the transformation relationship is determined, the calibration matrix of the visual navigation filter updated according to the previous moment is determined. The visual navigation data is calibrated based on the calibration matrix.

3. The method as described in claim 2, characterized in that, The visual navigation data is calibrated according to the calibration matrix, specifically including: Based on the calibration matrix, determine the calibration matrix of the visual navigation filter updated in the previous moment, the calibrated visual navigation data, and based on the transformation relationship, determine the difference between the calibrated visual navigation data and the satellite navigation data at this moment; Update the calibration matrix of the visual navigation filter based on the gap at that moment; The visual navigation data for the next time step is calibrated based on the updated calibration matrix.

4. The method as described in claim 3, characterized in that, Based on the difference at that moment, the calibration matrix of the visual navigation filter is updated, specifically including: Obtain the calibration matrix of the visual navigation filter updated in the previous time step; Based on the calibration matrix and the gap, determine the amount of increase in the calibration matrix; The calibration matrix of the visual navigation filter is updated according to the increment of the calibration matrix, and the updated result is used as the calibration matrix at that moment.

5. The method as described in claim 2, characterized in that, The visual navigation data is calibrated according to the calibration matrix, specifically including: When the satellite navigation data is lost, acquire the visual navigation data at that moment and the calibration matrix of the visual navigation filter at the most recent moment; Based on the calibration matrix, determine the increase in the visual navigation data and the increase in the calibration matrix, respectively. The visual navigation data is updated according to the increase in the visual navigation data, and the update result is used as the final calibration result of the visual navigation data at that moment. The calibration matrix of the visual navigation filter is updated according to the increase in the calibration matrix, and the update result is used as the calibration matrix at that moment.

6. The method as described in claim 1, characterized in that, Based on the determined rotation matrix and translation matrix, the transformation relationship between the first coordinate system and the second coordinate system is determined, specifically including: The determined rotation and translation matrices are used as the initial state variables for calibrating the filter; With the goal of minimizing the difference between satellite navigation data and visual navigation data in the calibration data at each historical moment, the rotation matrix and the translation matrix are updated until the stability of the calibration filter is higher than a preset stability threshold. Based on the updated rotation and translation matrices, determine the transformation relationship between the first coordinate system and the second coordinate system.

7. The method as described in claim 1, characterized in that, Before updating the visual navigation data, the method further includes: When the difference between the calibrated visual navigation data and the satellite navigation data acquired at that moment is greater than a preset error threshold, record the number of times the difference between the visual navigation data and the satellite navigation data is greater than the error threshold, and determine whether the number of times is greater than a preset number threshold. If so, then it is determined that the transformation relationship between the first coordinate system and the second coordinate system needs to be redefined; If not, then the transformation relationship does not need to be redefined.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

9. An unmanned device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 7.

Images