Multi-source data fusion positioning and attitude determination method and device and inspection system

Through multi-source data fusion, the synchronous processing and correction of GNSS, IMU and visual data is achieved, and the problem of insufficient positioning accuracy in power inspection is solved.

CN118936478APending Publication Date: 2024-11-12GUANGDONG POWER GRID CO LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411243997.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, a single information source is difficult to meet the high-precision positioning requirements during power inspection, especially in complex electromagnetic environments and vast space scenarios.

Method used

Using multi-source data fusion method, GNSS data, IMU data and visible light images are obtained, target input data is generated through time synchronization and data processing, anchor optimization model is constructed for preliminary positioning, and GNSS positioning information is corrected using IMU and visual data to obtain targets Positioning information.

Benefits of technology

It realizes high-precision positioning in complex electromagnetic environments and vast space scenarios, solving the problem that a single information source is difficult to meet high-precision needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118936478A_ABST
    Figure CN118936478A_ABST
Patent Text Reader

Abstract

The invention provides a multi-source data fusion positioning and attitude determination method and device and an inspection system, and the method comprises the steps: obtaining multi-source input data, carrying out the preprocessing of the multi-source input data, obtaining target input data, the multi-source input data at least comprising GNSS data, IMU data and a visible light image, and the preprocessing at least comprising time synchronization; an anchor point optimization model is constructed, the anchor point optimization model is used for performing preliminary positioning according to GNSS data in the target input data and correcting the preliminary positioning according to IMU data and the visible light image in the target input data to obtain target positioning information, and the target positioning information at least comprises position coordinates and pose orientation of the target object; and inputting the target input data into the anchor point optimization model to obtain target positioning information. The method solves the problem that a single information source is difficult to meet the high-precision positioning requirement in the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data communication technology, and in particular to a positioning and attitude determination method and device for multi-source data fusion, a computer-readable storage medium and an inspection system. Background Art

[0002] At present, the positioning and attitude determination technologies for power inspection are mainly divided into the following categories:

[0003] 1. GPS-based positioning and attitude determination: This is one of the most common positioning methods, using the global positioning system (GPS) to obtain the location information of the inspection equipment. However, since there is usually electromagnetic interference in power inspection scenarios, and in some scenarios such as near high-voltage lines or indoor inspections, GPS signals may be seriously affected, resulting in reduced positioning accuracy. In addition, the GPS solution has limited capabilities in attitude estimation and cannot meet the needs of high-precision attitude estimation in complex environments.

[0004] 2. Positioning and attitude determination based on inertial navigation: The inertial navigation system (INS) measures acceleration and angular velocity to infer the attitude and position of the device, and has the characteristics of high frequency response and short-term high precision. However, since inertial sensors are susceptible to noise and drift, positioning and attitude errors will gradually accumulate over time. Therefore, inertial navigation systems usually need to rely on other information sources for error correction.

[0005] 3. Vision-based positioning and attitude determination: Visual positioning technology uses cameras to obtain images of the surrounding environment and estimates the position and attitude of the device through image processing and feature matching. This solution performs well indoors and in environments without GPS signals, but its positioning accuracy depends on the richness of environmental features and the complexity of the algorithm. The robustness and accuracy of visual positioning may be affected in the case of lighting changes, dynamic scenes, and sparse environmental features.

[0006] In summary, the prior art lacks a high-precision solution that integrates multiple sensors for comprehensive positioning and attitude determination. That is, the prior art has the problem that a single information source cannot meet the high-precision requirements, and the positioning and attitude determination accuracy is insufficient during power inspections. Summary of the invention

[0007] The main purpose of the present application is to provide a positioning and attitude determination method, device, computer-readable storage medium and inspection system for multi-source data fusion, so as to at least solve the problem in the prior art that a single information source is difficult to meet the high-precision positioning requirements.

[0008] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a positioning and attitude determination method of multi-source data fusion is provided, including: acquiring multi-source input data and preprocessing the multi-source input data to obtain target input data, the multi-source input data at least including GNSS data, IMU data and visible light images, and the preprocessing at least includes time synchronization; constructing an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least includes the position coordinates and posture orientation of the target object; the target input data is input into the anchor point optimization model to obtain the target positioning information.

[0009] Optionally, multi-source input data is obtained, including: obtaining GNSS raw data after differential RTK through an RTK base station to obtain the GNSS data; obtaining IMU raw data through an inertial navigation software driver to obtain the IMU data, wherein the IMU data at least includes the acceleration and angular velocity of the target object; and photographing the target object through a visible light camera to obtain the visible light image.

[0010] Optionally, the multi-source input data is preprocessed to obtain target input data, including: obtaining first clock data based on the clock data in the GNSS data; correcting the local clock data based on the first clock data to obtain second clock data; and performing time synchronization on the IMU data and the visible light image based on the second clock data.

[0011] Optionally, preprocessing the multi-source input data to obtain target input data also includes: determining the elevation angle of the satellite corresponding to each GNSS data relative to the target object based on the elevation data in the multiple GNSS data to obtain first angle information, deleting the GNSS data corresponding to the first angle information less than a first threshold, to obtain the GNSS data in the target input data; pre-integrating the IMU data based on the first preset frame and the second preset frame to determine the deviation between each IMU data to obtain the IMU data in the target input data, the first preset frame being the previous frame of the second preset frame; based on each image data in the visible light image through The KLT sparse optical flow algorithm is used for tracking to obtain a first trajectory feature, and feature extraction is performed on each of the image data based on the GFTT corner points to obtain a second trajectory feature, until the sum of the number of the first trajectory feature and the second trajectory feature is within a preset range, the first trajectory feature and the second trajectory feature are determined as target trajectory features, outlier rejection processing is performed on the target trajectory features based on the RANSAC algorithm and the basic matrix model and the target trajectory features are projected onto a unit sphere, a target key frame is determined based on the average parallax between adjacent frames, and the visible light image corresponding to the target key frame is determined as the visible light image in the target input data, and the pixel distance between each pair of the target trajectory features is greater than a second threshold.

[0012] Optionally, pre-integrating the IMU data of the first preset frame and the second preset frame to determine the deviation between the IMU data to obtain a first target deviation includes: substituting the IMU data of the first preset frame and the second preset frame into a target formula to obtain the first target deviation:

[0013] Among them, a t is the acceleration measured by the accelerometer at time t, is the acceleration deviation at time t, is the transformation matrix between the world coordinate system and the drone coordinate system, n a is the additional noise of the accelerometer, is the corrected acceleration, w t is the angular velocity measured by the gyroscope at time t, is the gyroscope bias at time t, n w is the additional noise of the gyroscope, Ω(w) is a preset function, and w z and w y are the z-axis and y-axis angular velocities of the drone, t k The rotation matrix of the coordinate system between time t and any subsequent time t, t k Time and t k+1 the relative position deviation in the first target deviation between the moments, t k Time and t k+1 The speed deviation in the first target deviation between the times, t k Time and t k+1 The rotational deviation in said first target deviation between time instants.

[0014] Optionally, before constructing the anchor point optimization model, the method further includes: constructing an initial positioning model, the initial positioning model is used to generate ECEF coordinates according to the GNSS data in the target input data to obtain initial positioning information; constructing a yaw angle correction model, the yaw angle correction model is used to calculate a direction vector and a rotation vector according to the initial positioning information to obtain an initial direction vector and an initial rotation vector, determining an initial yaw angle and a receiving clock drift rate based on the initial direction vector and the initial rotation vector through Doppler measurement, and correcting the deviation between the ENU coordinate system and the world coordinate system based on the initial yaw angle, wherein determining the initial yaw angle and the initial receiving clock drift rate based on the initial direction vector and the initial rotation vector through Doppler measurement includes: In the formula, is the receiving clock drift rate of the receiver, ψ is the initial yaw angle, is the connection system status χ and satellite s j The Doppler measurement residual is n, the size of the sliding window, and p k is the number of satellites observed at the kth epoch in the window, represents a code pseudorange residual, the system state x includes the initial direction vector, the initial rotation vector and the target input data; constructing an anchor point refinement model, the anchor point refinement model is used to correct the initial positioning information according to the system state and the initial receiving clock drift rate to obtain a target clock offset and a target anchor point, wherein the initial receiving clock drift is corrected according to the system state and the initial receiving clock drift to obtain the target clock offset and the target anchor point, including: in, represents the code pseudorange residual, represents the residual of the code pseudorange noise, represents the residual error of the receiver clock, D t,k is the relevant covariance matrix, δt is the target clock offset, The target anchor point.

[0015] Optionally, constructing the anchor point optimization model includes: constructing the anchor point optimization model: Among them, {r p ,H p} encapsulates the prior information of the system state, H p is the prior coefficient, z i is the target input data, r(·) is the residual function, r(z i ,χ) is z i The corresponding residual factor, χ ※ is the target location information, ‖·‖ P Mahalanobis norm.

[0016] According to another aspect of the present application, a positioning and attitude determination device for multi-source data fusion is provided, characterized in that the device includes: a first acquisition unit, used to acquire multi-source input data and pre-process the multi-source input data to obtain target input data, the multi-source input data at least includes GNSS data, IMU data and visible light images, and the pre-processing at least includes time synchronization; a first construction unit, used to construct an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least includes the position coordinates and posture orientation of the target object; a positioning unit, used to input the target input data into the anchor point optimization model to obtain the target positioning information.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0018] According to another aspect of the present application, an inspection system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0019] Applying the technical solution of the present application, in the above-mentioned positioning and attitude determination method of multi-source data fusion, first, obtain multi-source input data and pre-process the multi-source input data to obtain target input data, the multi-source input data at least includes GNSS data, IMU data and visible light image, and the pre-processing at least includes time synchronization; then, construct an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the above-mentioned GNSS data in the above-mentioned target input data, and correct the above-mentioned preliminary positioning according to the above-mentioned IMU data and visible light image in the above-mentioned target input data to obtain target positioning information, the above-mentioned target positioning information at least includes the position coordinates and posture orientation of the above-mentioned target object; finally, input the above-mentioned target input data into the above-mentioned anchor point optimization model to obtain the above-mentioned target positioning information. This application is based on the efficient fusion of multiple sensor data such as original GNSS measurement data, inertial navigation data, visual information, etc., to correct the positioning information of GNSS measurement data, so as to realize the positioning of inspection equipment in complex electromagnetic environment and wide space inspection scenes, and solves the problem that a single information source in the prior art is difficult to meet the high-precision positioning requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A hardware structure block diagram of a mobile terminal according to a positioning and attitude determination method for multi-source data fusion provided in an embodiment of the present application is shown;

[0021] Figure 2 A schematic flow chart of a positioning and attitude determination method for multi-source data fusion provided according to an embodiment of the present application is shown;

[0022] Figure 3 A structural block diagram of a multi-source data fusion positioning and attitude determination device provided according to an embodiment of the present application is shown.

[0023] The above drawings include the following reference numerals:

[0024] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] As introduced in the background technology, a single information source in the prior art cannot adapt to all inspection environments and has the problem of insufficient accuracy in complex environments. In order to solve the problem that a single information source in the prior art is difficult to meet the high-precision positioning requirements, the embodiments of the present application provide a multi-source data fusion positioning and attitude determination method, device, computer-readable storage medium and inspection system.

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a multi-source data fusion positioning and attitude determination method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] In this embodiment, a positioning and attitude determination method for multi-source data fusion running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 2 FIG. 1 is a flow chart of a method for positioning and determining a posture by fusion of multi-source data according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0034] Step S201, acquiring multi-source input data and preprocessing the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the preprocessing at least includes time synchronization;

[0035] Specifically, during the power inspection process, the accuracy of positioning and attitude determination is highly dependent on the synchronization and fusion of multi-source information. In order to ensure the effective fusion of sensor data, the present application provides for time synchronization and data processing of the data of each sensor. The above data processing includes data screening, data correction and data calculation, and then the above target input data that is easy to fuse is obtained through time synchronization and data processing.

[0036] Step S202, constructing an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least including the position coordinates and posture orientation of the target object;

[0037] Specifically, the present application sets up a method to correct the GNSS preliminary positioning based on nonlinear state estimation to obtain the above target positioning information, which specifically includes taking the state that maximizes the posterior as the optimal state of the system according to the acquired sensor measurement data, i.e., the above target input data, to obtain the above target positioning information.

[0038] Step S203: input the target input data into the anchor point optimization model to obtain the target positioning information.

[0039] Specifically, when constructing the above-mentioned anchor point optimization model, the above-mentioned target input data is input into the above-mentioned anchor point optimization model to obtain the above-mentioned target positioning information.

[0040] Through this embodiment, first, multi-source input data is obtained and the multi-source input data is preprocessed to obtain target input data, the multi-source input data includes at least GNSS data, IMU data and visible light images, and the preprocessing includes at least time synchronization; then, an anchor point optimization model is constructed, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and the preliminary positioning is corrected according to the IMU data and visible light images in the target input data to obtain target positioning information, the target positioning information includes at least the position coordinates and posture orientation of the target object; finally, the target input data is input into the anchor point optimization model to obtain the target positioning information. This application is based on the efficient fusion of multiple sensor data such as original GNSS measurement data, inertial navigation data, visual information, etc., to correct the positioning information of GNSS measurement data, so as to realize the positioning of inspection equipment in complex electromagnetic environments and inspection scenes with vast space, and solves the problem that a single information source in the prior art is difficult to meet the high-precision positioning requirements.

[0041] In order to obtain the above multi-source input data, in an optional implementation, the above step S201 includes:

[0042] Step S2011, obtaining the GNSS raw data after differential RTK through the RTK base station to obtain the above GNSS data;

[0043] Specifically, a high-precision differential RTK base station is used to acquire GNSS raw data after high-precision differential RTK to obtain the above GNSS data.

[0044] Step S2012, obtaining IMU raw data through inertial navigation software driving to obtain the above-mentioned IMU data, wherein the above-mentioned IMU data at least includes the acceleration and angular velocity of the above-mentioned target object;

[0045] Specifically, the IMU raw data is acquired through the inertial navigation software driver to obtain the above-mentioned IMU data.

[0046] Step S2013, photographing the target object with a visible light camera to obtain the visible light image.

[0047] Specifically, the visible light camera is controlled by a visible light camera driver, and then the target object is photographed to obtain the visible light image.

[0048] In order to achieve time synchronization of multi-source input data, in an optional implementation, the above step S201 includes:

[0049] Step S2014, obtaining first clock data based on the clock data in the above GNSS data;

[0050] It is understandable that since the GNSS raw data has a time parameter, and the timestamp of the local data only depends on the local time, if the two times are not synchronized, the time base of the multi-source data will become inconsistent, thus affecting the recognition results of the model.

[0051] Specifically, a GNSS clock is acquired from the acquired high-precision original GNSS data to obtain first clock data.

[0052] Step S2015, using the first clock data as a reference, correcting the local clock data to obtain second clock data;

[0053] Specifically, the first clock data is used as a reference, and the local time is modified to make the two the same, thereby obtaining the second clock data.

[0054] Step S2016, time synchronization of the IMU data and the visible light image based on the second clock data.

[0055] Specifically, the data acquisition time of the IMU data and the visible light image is marked based on the second clock data to achieve time synchronization.

[0056] In order to pre-process the multi-source input data to obtain the target input data, in an optional implementation, the step S201 further includes:

[0057] Step S2017, determining the elevation angle of the satellite corresponding to each of the GNSS data relative to the target object based on the elevation data in the plurality of GNSS data, obtaining first angle information, deleting the GNSS data corresponding to the first angle information less than a first threshold, and obtaining the GNSS data in the target input data;

[0058] Specifically, the present application uses code pseudo-range measurement based on the single point positioning algorithm (SPP) to determine the global position of the GNSS receiver through trilateral measurement. In theory, the coordinates of the receiver are obtained with the help of three different satellites to complete the positioning, that is, the above-mentioned GNSS data is obtained. Since the signal propagation time is affected by factors such as the atmosphere, the accuracy of the SPP single-point positioning algorithm is low, and the noise of the SPP single-point positioning algorithm includes measurement noise and noise generated by the satellite set distribution. Therefore, the present application is set to screen the corresponding satellites based on the elevation data in the above-mentioned GNSS data, and filter out satellites with low elevation angles.

[0059] Furthermore, to ensure data availability, the present application sets that only satellites that have been locked for a certain period of time are allowed to enter the system.

[0060] Understandably, since ephemeris data is acquired via a slow satellite receiver wireless link (50 bits / second on GPS L1C), a GNSS measurement cannot be used until its corresponding ephemeris has been fully transmitted.

[0061] Step S2018, performing pre-integration based on the IMU data of the first preset frame and the second preset frame, determining the deviation between the IMU data, and obtaining the IMU data in the target input data, wherein the first preset frame is a previous frame of the second preset frame;

[0062] Specifically, the present application is set up to follow the IMU pre-integration derivation based on continuous-time quaternions, and to process the IMU deviations based on pairs.

[0063] Step S2019, based on each image data in the above-mentioned visible light image, tracking is performed through the KLT sparse optical flow algorithm to obtain a first trajectory feature, and based on the GFTT corner point, feature extraction is performed on each of the above-mentioned image data to obtain a second trajectory feature, until the sum of the number of the above-mentioned first trajectory features and the second trajectory features is within a preset range, the above-mentioned first trajectory features and the above-mentioned second trajectory features are determined as target trajectory features, outlier rejection processing is performed on the above-mentioned target trajectory features based on the RANSAC algorithm and the basic matrix model and they are projected onto a unit sphere, the target key frame is determined based on the average parallax between adjacent frames, and the above-mentioned visible light image corresponding to the above-mentioned target key frame is determined as the above-mentioned visible light image in the above-mentioned target input data, and the pixel distance between the above-mentioned target trajectory features is greater than the second threshold.

[0064] Specifically, for each image data, the KLT sparse optical flow algorithm is used for tracking to obtain the above-mentioned first trajectory features. At the same time, new GFTT corner point features are detected to obtain the above-mentioned second trajectory features to meet the minimum number (100-300) requirements in each image.

[0065] Furthermore, the detector enforces uniform feature distribution by setting a minimum pixel spacing between two adjacent features, and projects them onto the unit sphere by rejecting outliers while ensuring that the 2D features are not distorted.

[0066] In the specific implementation, outlier rejection is performed using RANSAC and the basic matrix model. Keyframe screening is performed at the same time as outlier rejection, and there are two criteria for selecting keyframes. The first is the average disparity with the previous keyframe. If the average disparity of the tracked features between the current frame and the latest keyframe exceeds a certain threshold, the frame is considered a new keyframe. The other criterion is the tracking quality. If the number of tracked features is below a certain threshold, this frame will be considered a new keyframe. This criterion is to avoid complete loss of feature tracks.

[0067] It is important to note that not only translations but also rotations can cause parallax. However, it is not possible to triangulate features in rotation-only motion. To avoid this, short-term integration of gyroscope measurements is used to compensate for rotation when computing parallax. For this reason, even if the gyroscope contains large noise or is biased, it only leads to suboptimal keyframe selection results without directly affecting the estimation quality.

[0068] In order to pre-integrate the above IMU data, in an optional implementation, the above step S2018 includes:

[0069] Step S20181, substituting the IMU data of the first preset frame and the second preset frame into the target formula to obtain the first target deviation:

[0070]

[0071] Among them, a t is the acceleration measured by the accelerometer at time t, is the acceleration deviation at time t, is the transformation matrix between the world coordinate system and the drone coordinate system, n a is the additional noise of the accelerometer, is the corrected acceleration, w t is the angular velocity measured by the gyroscope at time t, is the gyroscope bias at time t, n w is the additional noise of the gyroscope, Ω(w) is the preset function, w z and w y are the z-axis and y-axis angular velocities of the drone, t k The rotation matrix of the coordinate system between time t and any subsequent time t, t k Time and t k+1 The relative position deviation in the first target deviation between moments, t k Time and t k+1 The speed deviation in the first target deviation between times, t k Time and t k+1 The rotational deviation in the first target deviation between the times.

[0072] Specifically, IMU measurements are measured in the system coordinate system, incorporating the forces of anti-gravity and platform dynamics, so their errors are affected by acceleration bias and gyroscope bias.

[0073] It is understandable that the measurements of the gyroscope and accelerometer in the raw IMU data are as follows:

[0074]

[0075] Assume that the additional noise in the measurement values ​​of the accelerometer and gyroscope in the above formula satisfies the Gaussian distribution, that is, The acceleration bias and gyroscope bias are then modeled as random walks, whose derivatives are Gaussian white noise: Then there is

[0076] Furthermore, for two temporally consecutive frames b k and b k+1 , in the time interval [t k , t k+1 ]There are multiple inertial measurements in the local frame b. Given the bias estimates, integrate them into the local frame b k In the following formula:

[0077]

[0078] Covariance of α, β, and γ The above formula can be solved based on the original IMU network data, with b k As a reference frame for a given bias.

[0079] In order to initialize the operation of multi-source data in the above anchor point optimization model, before constructing the anchor point optimization model, the above method further includes:

[0080] Step S301, constructing an initial positioning model, wherein the initial positioning model is used to generate ECEF coordinates according to the GNSS data in the target input data to obtain initial positioning information;

[0081] Specifically, the coarse ECEF coordinates are generated based on the GNSS data in the target input data. The SPP algorithm takes all code pseudorange measurements of the most recent epoch as input.

[0082] Step S302, constructing a yaw angle correction model, the yaw angle correction model is used to calculate the direction vector and the rotation vector according to the initial positioning information, obtain the initial direction vector and the initial rotation vector, determine the initial yaw angle and the receiving clock drift rate based on the initial direction vector and the initial rotation vector through Doppler measurement, and correct the deviation between the ENU coordinate system and the world coordinate system based on the initial yaw angle, wherein the initial yaw angle and the initial receiving clock drift rate are determined based on the initial direction vector and the initial rotation vector through Doppler measurement, including:

[0083]

[0084] In the formula, is the receiving clock drift rate of the receiver, ψ is the initial yaw angle, is the connection system status χ and satellite s j The Doppler measurement residual is n, the size of the sliding window, and p k is the number of satellites observed at the kth epoch in the window, represents the code pseudorange residual, the system state χ includes the initial direction vector, the initial rotation vector and the target input data;

[0085] Specifically, the Doppler measurement with less noise is used to calibrate the yaw offset between the ENU coordinate system and the local world coordinate system. As shown in the above formula, and It is not sensitive to the position of the GNSS receiver and is used to calculate the direction vector based on the above coarse anchor point coordinates. and rotation Get the system state, and estimate the parameters as the yaw angle offset ψ and the average clock bias drift rate in the entire window measurement The transformation between the ENU coordinate system and the local world coordinate system is then calibrated based on the estimated parameters.

[0086] Step S303, constructing an anchor point refinement model, wherein the anchor point refinement model is used to correct the initial positioning information according to the system state and the initial receiving clock drift rate to obtain a target clock offset and a target anchor point, wherein the initial receiving clock drift is corrected according to the system state and the initial receiving clock drift to obtain the target clock offset and the target anchor point, including:

[0087]

[0088] in, represents the code pseudorange residual, represents the residual of the code pseudorange noise, represents the residual error of the receiver clock, D t,k is the relevant covariance matrix, δt is the above target clock offset, The target anchor point mentioned above.

[0089] Specifically, the previous coarse anchor points are refined and the local world trajectory is aligned with the trajectory in the ECEF frame. The position results of the visual-inertial alignment are used as prior information. The sliding window is further optimized based on the above formula. By optimizing the above problem, the anchor point coordinates and receiver clock bias associated with each GNSS epoch can be refined. After this step, the anchor point (the origin of the ENU coordinate system) will be set to the origin of the local world coordinate system. Finally, the initialization phase of the entire estimator is completed, and all necessary quantities have been assigned initial values.

[0090] In order to construct the above anchor point optimization model, in an optional implementation, the above step S202 includes:

[0091] Step S2021, construct the above anchor point optimization model:

[0092]

[0093] Among them, {r p ,H p} encapsulates the prior information of the system state, H p is the prior coefficient, z i is the input data of the above target, r(·) is the residual function, r(z i ,χ) is z i The corresponding residual factor, χ ※ To locate the above target information, ‖·‖ PMahalanobis norm.

[0094] Specifically, state estimation based on nonlinear optimization; the state that maximizes the a posteriori (MAP) given all measurements is the optimal state of the system. Assuming that all measurements are independent of each other and the noise of each measurement is a zero-mean Gaussian distribution, the MAP problem can be further transformed into a problem of minimizing the sum of a series of costs, each cost corresponding to a specific measurement.

[0095] In the above embodiment, a multi-stage GNSS-VI initialization procedure is used to calibrate the anchor point and yaw offset online. Before GNSS-VI initialization, it is assumed that VIO has been successfully initialized, that is, the gravity vector, initial velocity, initial IMU bias and scale have obtained initial values. After that, a smooth trajectory is formed in the local world coordinate system and prepared for the GNSS-VI initialization phase. The GNSS-VI initialization process requires tracking at least 4 satellites (if all satellites belong to a single system, or (N+3) if N satellite systems are involved).

[0096] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0097] The embodiment of the present application also provides a positioning and attitude determination device for multi-source data fusion. It should be noted that the positioning and attitude determination device for multi-source data fusion in the embodiment of the present application can be used to execute the positioning and attitude determination method for multi-source data fusion provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0098] The following is an introduction to the multi-source data fusion positioning and attitude determination device provided in an embodiment of the present application.

[0099] Figure 3 is a structural block diagram of a positioning and attitude determination device for multi-source data fusion according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0100] A first acquisition unit 10 is used to acquire multi-source input data and pre-process the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the pre-processing at least includes time synchronization;

[0101] Specifically, during the power inspection process, the accuracy of positioning and attitude determination is highly dependent on the synchronization and fusion of multi-source information. In order to ensure the effective fusion of sensor data, the present application provides for time synchronization and data processing of the data of each sensor. The above data processing includes data screening, data correction and data calculation, and then the above target input data that is easy to fuse is obtained through time synchronization and data processing.

[0102] A first construction unit 20 is used to construct an anchor point optimization model, wherein the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and the visible light image in the target input data to obtain target positioning information, wherein the target positioning information at least includes the position coordinates and the posture orientation of the target object;

[0103] Specifically, the present application sets up a method to correct the GNSS preliminary positioning based on nonlinear state estimation to obtain the above target positioning information, which specifically includes taking the state that maximizes the posterior as the optimal state of the system according to the acquired sensor measurement data, i.e., the above target input data, to obtain the above target positioning information.

[0104] The positioning unit 303 is used to input the target input data into the anchor point optimization model to obtain the target positioning information.

[0105] Specifically, when constructing the above-mentioned anchor point optimization model, the above-mentioned target input data is input into the above-mentioned anchor point optimization model to obtain the above-mentioned target positioning information.

[0106] Through this embodiment, the first acquisition unit acquires multi-source input data and pre-processes the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the pre-processing at least includes time synchronization; the first construction unit constructs an anchor point optimization model, wherein the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and visible light images in the target input data to obtain target positioning information, wherein the target positioning information at least includes the position coordinates and posture orientation of the target object; the positioning unit inputs the target input data into the anchor point optimization model to obtain the target positioning information. This application is based on the correction of the positioning information of the GNSS measurement data by efficiently fusing multiple sensor data such as original GNSS measurement data, inertial navigation data, and visual information, so as to realize the positioning of the inspection equipment in complex electromagnetic environments and inspection scenes with vast space, and solves the problem in the prior art that a single information source is difficult to meet the high-precision positioning requirements.

[0107] In order to obtain the multi-source input data, in an optional implementation manner, the first obtaining unit includes:

[0108] The first acquisition module is used to acquire the GNSS raw data after differential RTK through the RTK base station to obtain the above GNSS data;

[0109] Specifically, a high-precision differential RTK base station is used to acquire GNSS raw data after high-precision differential RTK to obtain the above GNSS data.

[0110] A second acquisition module is used to acquire IMU raw data through an inertial navigation software driver to obtain the above-mentioned IMU data, wherein the above-mentioned IMU data at least includes the acceleration and angular velocity of the above-mentioned target object;

[0111] Specifically, the IMU raw data is acquired through the inertial navigation software driver to obtain the above-mentioned IMU data.

[0112] The third acquisition module is used to photograph the target object through a visible light camera to obtain the visible light image.

[0113] Specifically, the visible light camera is controlled by a visible light camera driver, and then the target object is photographed to obtain the visible light image.

[0114] In order to achieve time synchronization of multi-source input data, in an optional implementation manner, the first acquisition unit includes:

[0115] A fourth acquisition module, configured to obtain first clock data based on the clock data in the GNSS data;

[0116] It is understandable that since the GNSS raw data has a time parameter, and the timestamp of the local data only depends on the local time, if the two times are not synchronized, the time base of the multi-source data will become inconsistent, thus affecting the recognition results of the model.

[0117] Specifically, a GNSS clock is acquired from the acquired high-precision original GNSS data to obtain first clock data.

[0118] A first correction module, used to correct the local clock data based on the first clock data to obtain second clock data;

[0119] Specifically, the first clock data is used as a reference, and the local time is modified to make the two the same, thereby obtaining the second clock data.

[0120] The second correction module is used to synchronize the time of the IMU data and the visible light image based on the second clock data.

[0121] Specifically, the data acquisition time of the IMU data and the visible light image is marked based on the second clock data to achieve time synchronization.

[0122] In order to preprocess the multi-source input data to obtain the target input data, in an optional implementation manner, the first acquisition unit further includes:

[0123] A first processing module is used to determine the elevation angle of the satellite corresponding to each of the GNSS data relative to the target object based on the elevation data in the plurality of GNSS data, obtain first angle information, delete the GNSS data corresponding to the first angle information less than a first threshold, and obtain the GNSS data in the target input data;

[0124] Specifically, the present application uses code pseudo-range measurement based on the single point positioning algorithm (SPP) to determine the global position of the GNSS receiver through trilateral measurement. In theory, the coordinates of the receiver are obtained with the help of three different satellites to complete the positioning, that is, the above-mentioned GNSS data is obtained. Since the signal propagation time is affected by factors such as the atmosphere, the accuracy of the SPP single-point positioning algorithm is low, and the noise of the SPP single-point positioning algorithm includes measurement noise and noise generated by the satellite set distribution. Therefore, the present application is set to screen the corresponding satellites based on the elevation data in the above-mentioned GNSS data, and filter out satellites with low elevation angles.

[0125] Furthermore, to ensure data availability, the present application sets that only satellites that have been locked for a certain period of time are allowed to enter the system.

[0126] Understandably, since ephemeris data is acquired via a slow satellite receiver wireless link (50 bits / second on GPS L1C), a GNSS measurement cannot be used until its corresponding ephemeris has been fully transmitted.

[0127] A second processing module is used to perform pre-integration based on the IMU data of the first preset frame and the second preset frame, determine the deviation between the IMU data, and obtain the IMU data in the target input data, wherein the first preset frame is a previous frame of the second preset frame;

[0128] Specifically, the present application is set up to follow the IMU pre-integration derivation based on continuous-time quaternions, and to process the IMU deviations based on pairs.

[0129] The third processing module is used to track each image data in the above-mentioned visible light image through the KLT sparse optical flow algorithm to obtain the first trajectory feature, and extract the features of each of the above-mentioned image data based on the GFTT corner point to obtain the second trajectory feature, until the sum of the number of the above-mentioned first trajectory features and the second trajectory features is within a preset range, and the above-mentioned first trajectory features and the above-mentioned second trajectory features are determined as target trajectory features, and the above-mentioned target trajectory features are rejected based on the RANSAC algorithm and the basic matrix model. Outlier rejection processing is performed on the above-mentioned target trajectory features and projected onto a unit sphere, and the target key frame is determined based on the average parallax between adjacent frames, and the above-mentioned visible light image corresponding to the above-mentioned target key frame is determined as the above-mentioned visible light image in the above-mentioned target input data, and the pixel distance between the above-mentioned target trajectory features is greater than the second threshold.

[0130] Specifically, for each image data, the KLT sparse optical flow algorithm is used for tracking to obtain the above-mentioned first trajectory features. At the same time, new GFTT corner point features are detected to obtain the above-mentioned second trajectory features to meet the minimum number (100-300) requirements in each image.

[0131] Furthermore, the detector enforces uniform feature distribution by setting a minimum pixel spacing between two adjacent features, and projects them onto the unit sphere by rejecting outliers while ensuring that the 2D features are not distorted.

[0132] In the specific implementation, outlier rejection is performed using RANSAC and the basic matrix model. Keyframe screening is performed at the same time as outlier rejection, and there are two criteria for selecting keyframes. The first is the average disparity with the previous keyframe. If the average disparity of the tracked features between the current frame and the latest keyframe exceeds a certain threshold, the frame is considered a new keyframe. The other criterion is the tracking quality. If the number of tracked features is below a certain threshold, this frame will be considered a new keyframe. This criterion is to avoid complete loss of feature tracks.

[0133] It is important to note that not only translations but also rotations can cause parallax. However, it is not possible to triangulate features in rotation-only motion. To avoid this, short-term integration of gyroscope measurements is used to compensate for rotation when computing parallax. For this reason, even if the gyroscope contains large noise or is biased, it only leads to suboptimal keyframe selection results without directly affecting the estimation quality.

[0134] In order to pre-integrate the above-mentioned IMU data, in an optional implementation, the above-mentioned second processing module includes:

[0135] The calculation submodule is used to substitute the IMU data of the first preset frame and the second preset frame into the target formula to obtain the first target deviation:

[0136]

[0137]

[0138] Among them, a t is the acceleration measured by the accelerometer at time t, is the acceleration deviation at time t, is the transformation matrix between the world coordinate system and the drone coordinate system, n a is the additional noise of the accelerometer, is the corrected acceleration, w t is the angular velocity measured by the gyroscope at time t, is the gyroscope bias at time t, n w is the additional noise of the gyroscope, Ω(w) is the preset function, w z and w y are the z-axis and y-axis angular velocities of the drone, t k The rotation matrix of the coordinate system between time t and any subsequent time t, t k Time and t k+1 The relative position deviation in the first target deviation between moments, t k Time and t k+1 The speed deviation in the first target deviation between times, t k Time and t k+1 The rotational deviation in the first target deviation between the times.

[0139] Specifically, IMU measurements are measured in the system coordinate system, incorporating the forces of anti-gravity and platform dynamics, so their errors are affected by acceleration bias and gyroscope bias.

[0140] It is understandable that the measurements of the gyroscope and accelerometer in the raw IMU data are as follows:

[0141]

[0142] Assume that the additional noise in the measurement values ​​of the accelerometer and gyroscope in the above formula satisfies the Gaussian distribution, that is, The acceleration bias and gyroscope bias are then modeled as random walks, whose derivatives are Gaussian white noise: Then there is

[0143] Furthermore, for two temporally consecutive frames b k and b k+1 , in the time interval [t k , t k+1]There are multiple inertial measurements in the local frame b. Given the bias estimates, integrate them into the local frame b k In the following formula:

[0144]

[0145]

[0146] Covariance of α, β, and γ The above formula can be solved based on the original IMU network data, with b k As a reference frame for a given bias.

[0147] In order to initialize the operation of multi-source data in the anchor point optimization model, the device further includes:

[0148] A second construction unit is used to construct an initial positioning model before constructing the anchor point optimization model, wherein the initial positioning model is used to generate ECEF coordinates according to the GNSS data in the target input data to obtain initial positioning information;

[0149] Specifically, the coarse ECEF coordinates are generated based on the GNSS data in the target input data. The SPP algorithm takes all code pseudorange measurements of the most recent epoch as input.

[0150] The second construction unit is used to construct a yaw angle correction model, wherein the yaw angle correction model is used to calculate a direction vector and a rotation vector according to the initial positioning information to obtain an initial direction vector and an initial rotation vector, determine an initial yaw angle and a receiving clock drift rate based on the initial direction vector and the initial rotation vector through Doppler measurement, and correct the deviation between the ENU coordinate system and the world coordinate system based on the initial yaw angle, wherein determining the initial yaw angle and the initial receiving clock drift rate based on the initial direction vector and the initial rotation vector through Doppler measurement includes:

[0151]

[0152] In the formula, is the receiving clock drift rate of the receiver, ψ is the initial yaw angle, is the connection system status χ and satellite s j The Doppler measurement residual is n, the size of the sliding window, and p k is the number of satellites observed at the kth epoch in the window, represents the code pseudorange residual, the system state χ includes the initial direction vector, the initial rotation vector and the target input data;

[0153] Specifically, the Doppler measurement with less noise is used to calibrate the yaw offset between the ENU coordinate system and the local world coordinate system. As shown in the above formula, and It is not sensitive to the position of the GNSS receiver and is used to calculate the direction vector based on the above coarse anchor point coordinates. and rotation Get the system state, and estimate the parameters as the yaw angle offset ψ and the average clock bias drift rate in the entire window measurement The transformation between the ENU coordinate system and the local world coordinate system is then calibrated based on the estimated parameters.

[0154] The second construction unit is used to construct an anchor point refinement model, wherein the anchor point refinement model is used to correct the initial positioning information according to the system state and the initial receiving clock drift rate to obtain a target clock offset and a target anchor point, wherein the initial receiving clock drift is corrected according to the system state and the initial receiving clock drift to obtain the target clock offset and the target anchor point, including:

[0155]

[0156] in, represents the code pseudorange residual, represents the residual of the code pseudorange noise, represents the residual error of the receiver clock, D t,k is the relevant covariance matrix, δt is the above target clock offset, The target anchor point mentioned above.

[0157] Specifically, the previous coarse anchor points are refined and the local world trajectory is aligned with the trajectory in the ECEF frame. The position results of the visual-inertial alignment are used as prior information. The sliding window is further optimized based on the above formula. By optimizing the above problem, the anchor point coordinates and receiver clock bias associated with each GNSS epoch can be refined. After this step, the anchor point (the origin of the ENU coordinate system) will be set to the origin of the local world coordinate system. Finally, the initialization phase of the entire estimator is completed, and all necessary quantities have been assigned initial values.

[0158] In order to construct the above anchor point optimization model, in an optional implementation, the above first construction unit includes:

[0159] Building blocks for constructing the above anchor optimization model:

[0160]

[0161] Among them, {r p ,H p} encapsulates the prior information of the system state, H pis the prior coefficient, z i is the input data of the above target, r(·) is the residual function, r(z i ,χ) is z i The corresponding residual factor, χ ※ To locate the above target information, ‖·‖ P Mahalanobis norm.

[0162] Specifically, state estimation based on nonlinear optimization; the state that maximizes the a posteriori (MAP) given all measurements is the optimal state of the system. Assuming that all measurements are independent of each other and the noise of each measurement is a zero-mean Gaussian distribution, the MAP problem can be further transformed into a problem of minimizing the sum of a series of costs, each cost corresponding to a specific measurement.

[0163] In the above embodiment, a multi-stage GNSS-VI initialization procedure is used to calibrate the anchor point and yaw offset online. Before GNSS-VI initialization, it is assumed that VIO has been successfully initialized, that is, the gravity vector, initial velocity, initial IMU bias and scale have obtained initial values. After that, a smooth trajectory is formed in the local world coordinate system and prepared for the GNSS-VI initialization phase. The GNSS-VI initialization process requires tracking at least 4 satellites (if all satellites belong to a single system, or (N+3) if N satellite systems are involved).

[0164] The above-mentioned multi-source data fusion positioning and attitude determination device includes a processor and a memory. The above-mentioned first acquisition unit, the first construction unit and the positioning unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in the form of any combination.

[0165] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the positioning accuracy can be improved by adjusting the kernel parameters.

[0166] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0167] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the positioning and attitude determination method of multi-source data fusion.

[0168] Specifically, the positioning and attitude determination method of multi-source data fusion includes:

[0169] Step S201, acquiring multi-source input data and preprocessing the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the preprocessing at least includes time synchronization;

[0170] Specifically, during the power inspection process, the accuracy of positioning and attitude determination is highly dependent on the synchronization and fusion of multi-source information. In order to ensure the effective fusion of sensor data, the present application provides for time synchronization and data processing of the data of each sensor. The above data processing includes data screening, data correction and data calculation, and then the above target input data that is easy to fuse is obtained through time synchronization and data processing.

[0171] Step S202, constructing an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least including the position coordinates and posture orientation of the target object;

[0172] Specifically, the present application sets up a method to correct the GNSS preliminary positioning based on nonlinear state estimation to obtain the above target positioning information, which specifically includes taking the state that maximizes the posterior as the optimal state of the system according to the acquired sensor measurement data, i.e., the above target input data, to obtain the above target positioning information.

[0173] Step S203: input the target input data into the anchor point optimization model to obtain the target positioning information.

[0174] Specifically, when constructing the above-mentioned anchor point optimization model, the above-mentioned target input data is input into the above-mentioned anchor point optimization model to obtain the above-mentioned target positioning information.

[0175] An embodiment of the present invention provides a processor, which is used to run a program, wherein the positioning and attitude determination method of multi-source data fusion is executed when the program is run.

[0176] Specifically, the positioning and attitude determination method of multi-source data fusion includes:

[0177] Step S201, acquiring multi-source input data and preprocessing the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the preprocessing at least includes time synchronization;

[0178] Specifically, during the power inspection process, the accuracy of positioning and attitude determination is highly dependent on the synchronization and fusion of multi-source information. In order to ensure the effective fusion of sensor data, the present application provides for time synchronization and data processing of the data of each sensor. The above data processing includes data screening, data correction and data calculation, and then the above target input data that is easy to fuse is obtained through time synchronization and data processing.

[0179] Step S202, constructing an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least including the position coordinates and posture orientation of the target object;

[0180] Specifically, the present application sets up a method to correct the GNSS preliminary positioning based on nonlinear state estimation to obtain the above target positioning information, which specifically includes taking the state that maximizes the posterior as the optimal state of the system according to the acquired sensor measurement data, i.e., the above target input data, to obtain the above target positioning information.

[0181] Step S203: input the target input data into the anchor point optimization model to obtain the target positioning information.

[0182] Specifically, when constructing the above-mentioned anchor point optimization model, the above-mentioned target input data is input into the above-mentioned anchor point optimization model to obtain the above-mentioned target positioning information.

[0183] An embodiment of the present invention provides an inspection system, the inspection system comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0184] Step S201, acquiring multi-source input data and preprocessing the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the preprocessing at least includes time synchronization;

[0185] Specifically, during the power inspection process, the accuracy of positioning and attitude determination is highly dependent on the synchronization and fusion of multi-source information. In order to ensure the effective fusion of sensor data, the present application provides for time synchronization and data processing of the data of each sensor. The above data processing includes data screening, data correction and data calculation, and then the above target input data that is easy to fuse is obtained through time synchronization and data processing.

[0186] Step S202, constructing an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least including the position coordinates and posture orientation of the target object;

[0187] Specifically, the present application sets up a method to correct the GNSS preliminary positioning based on nonlinear state estimation to obtain the above target positioning information, which specifically includes taking the state that maximizes the posterior as the optimal state of the system according to the acquired sensor measurement data, i.e., the above target input data, to obtain the above target positioning information.

[0188] Step S203: input the target input data into the anchor point optimization model to obtain the target positioning information.

[0189] Specifically, when constructing the above-mentioned anchor point optimization model, the above-mentioned target input data is input into the above-mentioned anchor point optimization model to obtain the above-mentioned target positioning information.

[0190] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0191] Step S201, acquiring multi-source input data and preprocessing the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the preprocessing at least includes time synchronization;

[0192] Specifically, during the power inspection process, the accuracy of positioning and attitude determination is highly dependent on the synchronization and fusion of multi-source information. In order to ensure the effective fusion of sensor data, the present application provides for time synchronization and data processing of the data of each sensor. The above data processing includes data screening, data correction and data calculation, and then the above target input data that is easy to fuse is obtained through time synchronization and data processing.

[0193] Step S202, constructing an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least including the position coordinates and posture orientation of the target object;

[0194] Specifically, the present application sets up a method to correct the GNSS preliminary positioning based on nonlinear state estimation to obtain the above target positioning information, which specifically includes taking the state that maximizes the posterior as the optimal state of the system according to the acquired sensor measurement data, i.e., the above target input data, to obtain the above target positioning information.

[0195] Step S203: input the target input data into the anchor point optimization model to obtain the target positioning information.

[0196] Specifically, when constructing the above-mentioned anchor point optimization model, the above-mentioned target input data is input into the above-mentioned anchor point optimization model to obtain the above-mentioned target positioning information.

[0197] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0198] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0199] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0200] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0202] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0203] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0204] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0205] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0206] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0207] 1) The positioning and attitude determination method of multi-source data fusion of the present application, first, obtain multi-source input data and pre-process the multi-source input data to obtain target input data, the multi-source input data at least includes GNSS data, IMU data and visible light image, the pre-processing at least includes time synchronization; then, construct an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least includes the position coordinates and posture orientation of the target object; finally, the target input data is input into the anchor point optimization model to obtain the target positioning information. The present application is based on the efficient fusion of multiple sensor data such as original GNSS measurement data, inertial navigation data, visual information, etc., to correct the positioning information of GNSS measurement data, so as to realize the positioning of inspection equipment in complex electromagnetic environment and wide space inspection scenes, and solves the problem that a single information source in the prior art is difficult to meet the high-precision positioning requirements.

[0208] 2) The multi-source data fusion positioning and attitude determination device of the present application, the first acquisition unit acquires multi-source input data and pre-processes the multi-source input data to obtain target input data, the multi-source input data at least includes GNSS data, IMU data and visible light image, and the pre-processing at least includes time synchronization; the first construction unit constructs an anchor point optimization model, the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and correct the preliminary positioning according to the IMU data and visible light image in the target input data to obtain target positioning information, the target positioning information at least includes the position coordinates and posture orientation of the target object; the positioning unit inputs the target input data into the anchor point optimization model to obtain the target positioning information. The present application is based on the efficient fusion of multiple sensor data such as original GNSS measurement data, inertial navigation data, visual information, etc., to correct the positioning information of GNSS measurement data, so as to realize the positioning of inspection equipment in complex electromagnetic environments and inspection scenes with vast space, and solves the problem that a single information source in the prior art is difficult to meet the high-precision positioning requirements.

[0209] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A positioning and attitude determination method based on multi-source data fusion, characterized in that: include: Acquire multi-source input data and pre-process the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the pre-processing at least includes time synchronization; Constructing an anchor point optimization model, wherein the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and the visible light image in the target input data to obtain target positioning information, wherein the target positioning information at least includes the position coordinates and the posture orientation of the target object; The target input data is input into the anchor point optimization model to obtain the target positioning information.

2. The method according to claim 1, characterized in that Obtain input data from multiple sources, including: Acquire the GNSS raw data after differential RTK through the RTK base station to obtain the GNSS data; Acquire IMU raw data through inertial navigation software driving to obtain the IMU data, wherein the IMU data at least includes the acceleration and angular velocity of the target object; The target object is photographed by a visible light camera to obtain the visible light image.

3. The method according to claim 2, characterized in that Preprocessing the multi-source input data to obtain target input data includes: Obtaining first clock data based on the clock data in the GNSS data; Based on the first clock data, the local clock data is corrected to obtain the second clock data; based on the second clock data, the IMU data and the visible light image are time synchronized.

4. The method according to claim 2, characterized in that: Preprocessing the multi-source input data to obtain target input data also includes: Determine the elevation angle of the satellite corresponding to each GNSS data relative to the target object based on the elevation data in the plurality of GNSS data to obtain first angle information, delete the GNSS data corresponding to the first angle information less than a first threshold, and obtain the GNSS data in the target input data; Pre-integrate the IMU data of the first preset frame and the second preset frame, determine the deviation between the IMU data, and obtain the IMU data in the target input data, wherein the first preset frame is a previous frame of the second preset frame; Based on each image data in the visible light image, a first trajectory feature is obtained by tracking through a KLT sparse optical flow algorithm, and a feature of each image data is extracted based on a GFTT corner point to obtain a second trajectory feature, until the sum of the number of the first trajectory feature and the second trajectory feature is within a preset range, the first trajectory feature and the second trajectory feature are determined as target trajectory features, outlier rejection processing is performed on the target trajectory features based on a RANSAC algorithm and a basic matrix model and the target trajectory features are projected onto a unit sphere, a target key frame is determined based on an average parallax between adjacent frames, and the visible light image corresponding to the target key frame is determined as the visible light image in the target input data, and the pixel distance between each pair of the target trajectory features is greater than a second threshold.

5. The method according to claim 4, characterized in that Pre-integrating the IMU data of the first preset frame and the second preset frame, determining the deviation between the IMU data, and obtaining a first target deviation, includes: Substitute the IMU data of the first preset frame and the second preset frame into the target formula to obtain the first target deviation: Among them, a t is the acceleration measured by the accelerometer at time t, is the acceleration deviation at time t, is the transformation matrix between the world coordinate system and the drone coordinate system, n a is the additional noise of the accelerometer, is the corrected acceleration, w t is the angular velocity measured by the gyroscope at time t, is the gyroscope bias at time t, n w is the additional noise of the gyroscope, Ω(w) is a preset function, and w z and w y are the z-axis and y-axis angular velocities of the drone, t k The rotation matrix of the coordinate system between time t and any subsequent time t, t k Time and t k+1 the relative position deviation in the first target deviation between the moments, t k Time and t k+1 The speed deviation in the first target deviation between the times, t k Time and t k+1 The rotational deviation in said first target deviation between time instants.

6. The method according to claim 1, characterized in that Before constructing the anchor point optimization model, the method further includes: Constructing an initial positioning model, wherein the initial positioning model is used to generate ECEF coordinates according to the GNSS data in the target input data to obtain initial positioning information; Constructing a yaw angle correction model, wherein the yaw angle correction model is used to calculate a direction vector and a rotation vector according to the initial positioning information, obtain an initial direction vector and an initial rotation vector, determine an initial yaw angle and a receiving clock drift rate based on the initial direction vector and the initial rotation vector through Doppler measurement, and correct the deviation between the ENU coordinate system and the world coordinate system based on the initial yaw angle, wherein determining the initial yaw angle and the initial receiving clock drift rate based on the initial direction vector and the initial rotation vector through Doppler measurement includes: In the formula, is the receiving clock drift rate of the receiver, ψ is the initial yaw angle, is the connection system status χ and satellite s j The Doppler measurement residual is n, the size of the sliding window, and p k is the number of satellites observed at the kth epoch in the window, represents a code pseudorange residual, wherein the system state x comprises the initial direction vector, the initial rotation vector and the target input data; Constructing an anchor point refinement model, wherein the anchor point refinement model is used to correct the initial positioning information according to the system state and the initial receiving clock drift rate to obtain a target clock offset and a target anchor point, wherein the initial receiving clock drift is corrected according to the system state and the initial receiving clock drift to obtain the target clock offset and the target anchor point, including: in, represents the code pseudorange residual, represents the residual of the code pseudorange noise, represents the residual error of the receiver clock, D t,k is the relevant covariance matrix, δt is the target clock offset, The target anchor point.

7. The method according to claim 6, characterized in that Construct an anchor optimization model, including: Construct the anchor point optimization model: Among them, {r p ,H p } encapsulates the prior information of the system state, H p is the prior coefficient, z i is the target input data, r(·) is the residual function, r(z i ,χ) is z i The corresponding residual factor, χ ※ is the target location information, ‖·‖ P Mahalanobis norm.

8. A positioning and attitude determination device for multi-source data fusion, characterized in that: The device comprises: A first acquisition unit is used to acquire multi-source input data and pre-process the multi-source input data to obtain target input data, wherein the multi-source input data at least includes GNSS data, IMU data and visible light images, and the pre-processing at least includes time synchronization; A first construction unit is used to construct an anchor point optimization model, wherein the anchor point optimization model is used to perform preliminary positioning according to the GNSS data in the target input data, and to correct the preliminary positioning according to the IMU data and the visible light image in the target input data to obtain target positioning information, wherein the target positioning information at least includes a position coordinate and a posture orientation of a target object; A positioning unit is used to input the target input data into the anchor point optimization model to obtain the target positioning information.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A patrol inspection system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Citation Information

Cited By

  • Unmanned aerial vehicle route planning method and system based on electromagnetic environment data and three-dimensional map

    CN120333455A

  • Server, spread spectrum setting method, program product, apparatus, and medium

    CN120848686A