A fully automatic pod calibration method without known ground information

Abstract

The present invention proposes a fully automatic pod calibration method that does not require known ground information, belonging to the field of photoelectric pod error calibration. The pod is installed on an aircraft, and the timing alignment between the differential GPS, aircraft IMU, pod encoder and camera is completed. When the aircraft is in flight, images, aircraft position and attitude, pod frame angle, and camera focal length value data are collected. A feature point matching algorithm is used to select the feature points of the first image, eliminate the feature points of moving objects, calculate the pixel offset, input the relevant data into an iterative optimization function, calculate the error, and continuously estimate the true value of the attitude during the overlapping period of the area through a positioning and servo control program. When the field of view is out of view, the iteration stops and the error estimate is stored; subsequent detections all use the previous error estimate for iteration; the present invention does not require ground control points, specific flight routes and high-precision satellite map databases, and does not require calibration and additional operations, and can automatically complete pod calibration.

Description

Technical Field

[0001] The present invention belongs to the technical field of photoelectric pod error calibration, and in particular relates to a fully automatic pod calibration method without the need for known ground information. Background Art

[0002] When using an airborne electro-optical pod for target positioning, geo-steering, or locking onto a region of interest (ROI), systematic errors (generally referring to mounting errors between the aircraft and the pod, zero-position errors in the pod's frame angles, and focal length errors in the pod's camera) significantly impact accuracy and the final rendering. Therefore, each pod requires calibration. Previous methods, such as using GPS differential equipment or satellite maps of the corresponding area (such as the method described in patent CN 113415433 B, "Pod Attitude Correction Method, Apparatus, and Unmanned Aerial Vehicle Based on a Three-Dimensional Scene Model"), pre-acquire the precise geographic location of ground targets to be observed during flight. This method then locates the calibrated ground targets while the aircraft is flying along a planned, complex, and circumventing route. Systematic errors are then compared with the ground truth to determine the system error. Both methods require the accurate geographic location of known targets to be acquired in advance, supplemented by a corresponding circumventing route.

[0003] This type of method requires auxiliary work to be completed before the pod is installed on the aircraft and begins its mission. This includes carrying equipment specifically for the flight or collecting satellite maps to calibrate target points and plan flight routes. Therefore, it is less convenient and less time-effective. Furthermore, most previous methods require the use of laser ranging, which can easily expose the aircraft's position during flight and fail to meet the requirements for stealth flight. Patent CN 118603141 B (A method for calibrating pod system errors without requiring known ground position) can complete pod calibration without relying on ground information, but requires the operator to complete the operation according to a specific process during flight. This obviously also requires time dedicated to pod calibration. This reduces the operability of this method and also interferes with mission execution during flight to a certain extent.

[0004] Considering the above problems, there is an urgent need for a fully automatic optoelectronic pod calibration method that does not require known ground target position information, does not require laser ranging information, and does not require human intervention and special calibration time during flight. Summary of the Invention

[0005] To address the problems of previous methods requiring complex route planning and long periods of observation and data collection, the present invention proposes a fully automatic pod calibration method that does not require known ground information. This allows pod calibration to be completed during the detection process before or during flight without any additional instructions or advance settings, allowing subsequent flights to more accurately complete functions such as target positioning, geographic guidance, and locking of regions of interest (ROIs).

[0006] The present invention is achieved through the following technical solutions:

[0007] A fully automatic pod calibration method that does not require known ground information:

[0008] Step 1: Install the pod on the aircraft and align the timing between the aircraft and the pod's sensors.

[0009] Step 2: During the flight, when the pod is used to detect any area at any time, the image and the aircraft data and pod data at the same time are continuously collected;

[0010] Step 3: Use the feature point matching algorithm to perform feature point matching on the images detected within the period selected in step 2;

[0011] Step 4: Based on the data from steps 2 and 3, the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error are calculated through an iterative optimization function.

[0012] Step 5: Substitute the error calculated in step 4 into the positioning solution program and the servo control program. During the period when the next detected area still overlaps with the previous area, continue to estimate the true value of the attitude and transmit it to the positioning solution program and the servo control program at the same time for automatic calibration.

[0013] Furthermore, in step 1,

[0014] The pod is mounted on the aircraft's mounting base and the timing alignment is completed before installation on the aircraft;

[0015] The timestamp information of the differential GPS and IMU on the aircraft and the encoder and camera inside the pod are exported through the communication protocol to align the timing between the various sensors.

[0016] Furthermore, in step 2

[0017] The aircraft data includes: the latitude output by the aircraft differential GPS ,longitude ,high , the heading output by the aircraft IMU , pitch and roll , roll ;

[0018] The pod data includes: frame angle read by the pod encoder 、 , the camera's current focal length value data .

[0019] Furthermore, in step 3,

[0020] Select the feature points in the first image as the points to be tested, and use the motion detection algorithm to delete the feature points representing the moving objects, and mark the remaining feature points as ,

[0021] Then calculate the pixel offset of the feature points of these marks relative to the center of the visual axis at the current moment during the subsequent detection of this area .

[0022] Furthermore, in step 3,

[0023] The specific process of calculating the pixel offset is as follows: when the detection pod begins to detect a certain area, it uses the feature point algorithm to obtain all the feature points in the first image, matches them with the feature points of the subsequent image, and uses the common feature points in the two images to solve the transmission matrix of the subsequent image relative to the first image. From this, the pixel offset of the feature points of the first image relative to the center of the visual axis of the subsequent image is calculated;

[0024] If a subsequent image does not have the same feature points as the first image, the pixel offset of the feature points in the first image relative to its own visual axis center is indirectly calculated by comparing it with the previous image that has the same feature points as the current image and has solved the pixel offset of the feature points in the first image relative to its own visual axis center;

[0025] If an image does not have the same feature points as any previous image, the image and its attached data will be discarded;

[0026] During this process, a motion detection algorithm is used to filter out the parts of the feature points in the first image that represent moving objects.

[0027] Furthermore, in step 4,

[0028] The current position and attitude data of the aircraft at each moment, the current frame angle data of the pod, the current focal length value of the camera and the pixel offset of the marked feature point relative to the current image center are input into the iterative optimization function to calculate the installation error between the aircraft and the pod and the zero position error of the pod frame angle and , and camera focal length error ;

[0029] The installation error includes the heading component , pitch component , roll component .

[0030] Furthermore, the calibration method further includes:

[0031] Step 6: When the pod's field of view leaves the current area, the iterative optimization process stops, and the error estimate calculated during this period is stored in the program;

[0032] When the pod detects the next area, the previous error estimate is used as the initial value and substituted into the iterative optimization program to iterate out a more accurate error estimate. The updated error estimate is then passed to the positioning solution program and the servo control program.

[0033] A fully automatic pod calibration system that does not require known ground information:

[0034] The calibration system includes a timing calibration module, an acquisition module, a feature point matching module, an error calculation module and an automatic calibration module:

[0035] The timing calibration module is used to align the timing between the aircraft and the pod sensors;

[0036] The acquisition module continuously acquires images and aircraft data and pod data at the same time when the pod is used to detect any area during the flight of the aircraft at any time period;

[0037] The feature point matching module uses a feature point matching algorithm to perform feature point matching on images detected within a period of time selected by the acquisition module;

[0038] The error calculation module calculates the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error through an iterative optimization function based on the data from the acquisition module and the feature point matching module;

[0039] The automatic calibration module substitutes the error calculated by the error calculation module into the positioning solution program and the servo control program, and continuously estimates the true value of the attitude during the period when the next detected area still overlaps with the previous area, and transmits it to the positioning solution program and the servo control program at the same time to perform automatic calibration.

[0040] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0041] A computer-readable storage medium is used to store computer instructions, which implement the steps of the above method when executed by a processor.

[0042] Beneficial effects of the present invention

[0043] This invention overcomes the limitations of previous methods, which required extracting the precise geographic location of ground targets through various means. It also eliminates the need for complex flyby routes and laser ranging, and completely eliminates the need for manual intervention, specialized calibration instructions, or dedicated calibration time periods. This significantly improves the convenience, timeliness, and confidentiality of pod error calibration during flight, supporting subsequent high-precision target positioning, geographic guidance, and tracking functions during the same flight. This not only significantly reduces the difficulty and complexity of calibration, but also greatly facilitates practical applications.

[0044] The present invention can be applied in the fields of error calibration of optoelectronic pods and target positioning and geographic guidance of optoelectronic pods, and can more accurately lock the region of interest (ROI) during line inspection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] The experimental methods used in the following examples are conventional methods unless otherwise specified. The materials, reagents, methods, and instruments used are conventional in the art and can be obtained commercially by those skilled in the art unless otherwise specified.

[0048] like Figure 1 As shown, the present invention proposes a fully automatic pod calibration method without knowing ground information:

[0049] The method specifically comprises the following steps:

[0050] Step 1: Install the pod on the aircraft and align the timing between the aircraft and the pod's sensors.

[0051] The pod is mounted on the aircraft's mounting base and the timing alignment is completed before installation on the aircraft;

[0052] The timestamp information of the differential GPS and IMU on the aircraft and the encoder and camera inside the pod are exported through the communication protocol to align the timing between the various sensors.

[0053] The pod encoder's output frequency can reach up to 1000Hz, allowing for timing alignment before installation. The aircraft's differential GPS and IMU output frequencies are relatively low and have a delay, requiring time-stamp information to be used for delay estimation. This delay is then used to determine the aircraft's position and attitude data that precisely corresponds to the current image.

[0054] Step 2: During the flight, when the pod is used to detect any area at any time, the image and the aircraft data and pod data at the same time are continuously collected;

[0055] The aircraft data includes: the latitude output by the aircraft differential GPS ,longitude ,high , the heading output by the aircraft IMU , pitch and roll , roll ;

[0056] The pod data includes: frame angle read by the pod encoder 、 , the camera's current focal length value data .

[0057] During the entire flight, there are no external input pod calibration commands, nor are there any set pod calibration operations at any time. The operator can freely control the pod at any time to achieve the desired function.

[0058] At any time period, when the pod detects any area, the calibration function inside the pod is automatically activated without human intervention.

[0059] Step 3: Use a feature point matching algorithm to perform feature point matching on the image of a certain area detected within the period of time selected in step 2;

[0060] Select the feature points in the first image as the points to be tested, and use the motion detection algorithm to delete the feature points representing the moving objects, and mark the remaining feature points as ,It should be noted that the number of marked feature points must be greater than 4;

[0061] Then calculate the pixel offset of the feature points of these marks relative to the center of the visual axis at the current moment during the subsequent detection of this area .

[0062] The specific process of calculating the pixel offset is as follows: when the detection pod begins to detect a certain area, it uses the feature point algorithm to obtain all the feature points in the first image, matches them with the feature points of the subsequent image, and uses the common feature points in the two images to solve the transmission matrix of the subsequent image relative to the first image. From this, the pixel offset of the feature points of the first image relative to the center of the visual axis of the subsequent image is calculated;

[0063] If a subsequent image does not have the same feature points as the first image, the pixel offset of the feature points in the first image relative to its own visual axis center is indirectly calculated by comparing it with the previous image that has the same feature points as the current image and has solved the pixel offset of the feature points in the first image relative to its own visual axis center;

[0064] If an image does not have the same feature points as any previous image, the image and its attached data will be discarded;

[0065] During this process, a motion detection algorithm is used to filter out the parts of the feature points in the first image that represent moving objects.

[0066] There are no specific restrictions on the methods used for extracting feature points from images and detecting motion. For example, feature point matching algorithms can utilize traditional SIFT or SURF, or the deep learning-based SuperGlue. When image quality is average, after improving image clarity using methods such as image defogging algorithms (such as the dark channel method) or image enhancement algorithms (such as CLAHE), feature point matching algorithms can be used to detect feature points. For example, a Gaussian mixture model can be used for motion detection. Users can flexibly select an algorithm based on hardware requirements, as long as it meets the requirements for feature point matching and motion detection.

[0067] Step 4: Based on the data from steps 2 and 3, the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error are calculated through an iterative optimization function.

[0068] The current position and attitude data of the aircraft at each moment, the current frame angle data of the pod, the current focal length value of the camera and the pixel offset of the marked feature point relative to the current image center are input into the iterative optimization function to calculate the installation error between the aircraft and the pod and the zero position error of the pod frame angle and , and camera focal length error ;

[0069] The installation error includes the heading component , pitch component , roll component .

[0070] Step 5: Substitute the error calculated in step 4 into the positioning solution program and the servo control program. During the period when the next detected area still overlaps with the previous area, continue to estimate the true value of the attitude and transmit it to the positioning solution program and the servo control program at the same time for automatic calibration.

[0071] Step 6: When the pod's field of view leaves the current area (i.e., the pod's field of view switches significantly to another area, and the current image has no common feature points with the previous frame), the iterative optimization process stops, and the error estimate calculated during this period is stored in the program;

[0072] When the pod detects the next area, the previous error estimate is used as the initial value and substituted into the iterative optimization program to iterate out a more accurate error estimate. The updated error estimate is then passed to the positioning solution program and the servo control program.

[0073] A fully automatic pod calibration system that does not require known ground information:

[0074] The calibration system includes a timing calibration module, an acquisition module, a feature point matching module, an error calculation module and an automatic calibration module:

[0075] The timing calibration module is used to align the timing between the aircraft and the pod sensors;

[0076] The acquisition module continuously acquires images and aircraft data and pod data at the same time when the pod is used to detect any area during the flight of the aircraft at any time period;

[0077] The feature point matching module uses a feature point matching algorithm to perform feature point matching on images detected within a period of time selected by the acquisition module;

[0078] The error calculation module calculates the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error through an iterative optimization function based on the data from the acquisition module and the feature point matching module;

[0079] The automatic calibration module substitutes the error calculated by the error calculation module into the positioning solution program and the servo control program, and continuously estimates the true value of the attitude during the period when the next detected area still overlaps with the previous area, and transmits it to the positioning solution program and the servo control program at the same time to perform automatic calibration.

[0080] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0081] A computer-readable storage medium is used to store computer instructions, which implement the steps of the above method when executed by a processor.

[0082] The memory in the embodiments of the present application can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to comprise, but not be limited to, these and any other suitable types of memory.

[0083] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired connection such as a coaxial cable, optical fiber, digital subscriber line (DSL), or wireless connection such as infrared, wireless, or microwave. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium such as a floppy disk, hard disk, magnetic tape, an optical medium such as a high-density digital video disc (DVD), or a semiconductor medium such as a solid-state disc (SSD).

[0084] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0085] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-described method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-described processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-described method.

[0086] The above is a detailed introduction to the fully automatic pod calibration method proposed in the present invention that does not require known ground information, and the principles and implementation methods of the present invention are explained. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A fully automatic pod calibration method that does not require known ground information, characterized by: The method specifically comprises the following steps: Step 1: Install the pod on the aircraft and align the timing between the aircraft and the pod's sensors. Step 2: During the flight, when the pod is used to detect any area at any time, the image and the aircraft data and pod data at the same time are continuously collected; Step 3: Use the feature point matching algorithm to perform feature point matching on the images detected within the period selected in step 2; Select the feature points in the first image as the points to be tested, and use the motion detection algorithm to delete the feature points representing the moving objects, and mark the remaining feature points as [T1, T2, ..., T k ], Then calculate the pixel offset of the feature points of these marks relative to the center of the visual axis at the current moment during the subsequent detection of this area Step 4: Based on the data from steps 2 and 3, the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error are calculated through an iterative optimization function. Step 5: Substitute the error calculated in step 4 into the positioning solution program and the servo control program. During the period when the next detected area still overlaps with the previous area, continue to estimate the true value of the attitude and transmit it to the positioning solution program and the servo control program at the same time for automatic calibration.

2. The calibration method according to claim 1, wherein: In step 1, The pod is mounted on the aircraft's mounting base and the timing alignment is completed before installation on the aircraft; The timestamp information of the differential GPS and IMU on the aircraft and the encoder and camera inside the pod are exported through the communication protocol to align the timing between the various sensors.

3. The calibration method according to claim 2, wherein: In step 2 The aircraft data includes: the latitude output by the aircraft differential GPS longitude high The heading output by the aircraft IMU Pitch roll The pod data includes: frame angle read by the pod encoder The camera's current focal length data f c .

4. The calibration method according to claim 3, wherein: In step 3, The specific process of calculating the pixel offset is as follows: when the detection pod begins to detect a certain area, it uses the feature point algorithm to obtain all the feature points in the first image, matches them with the feature points of the subsequent image, and uses the common feature points in the two images to solve the transmission matrix of the subsequent image relative to the first image. From this, the pixel offset of the feature points of the first image relative to the center of the visual axis of the subsequent image is calculated; If a subsequent image does not have the same feature points as the first image, the pixel offset of the feature points in the first image relative to its own visual axis center is indirectly calculated by comparing it with the previous image that has the same feature points as the current image and has solved the pixel offset of the feature points in the first image relative to its own visual axis center; If an image does not have the same feature points as any previous image, the image and its attached data will be discarded; During this process, a motion detection algorithm is used to filter out the parts of the feature points in the first image that represent moving objects.

5. The calibration method according to claim 4, characterized in that: In step 4, The current position and attitude data of the aircraft at each moment, the current frame angle data of the pod, the current focal length value of the camera, and the pixel offset of the marked feature point relative to the current image center are input into the iterative optimization function to calculate the installation error between the aircraft and the pod, and the pod frame angle zero position error Δθ en1 and Δθ en2 , and the camera focal length error Δf c ; The installation error includes the heading component Pitch component Roll component Δγ a .

6. The calibration method according to claim 5, characterized in that: The calibration method further comprises: Step 6: When the pod's field of view leaves the current area, the iterative optimization process stops, and the error estimate calculated during this period is stored in the program; When the pod detects the next area, the previous error estimate is used as the initial value and substituted into the iterative optimization program to iterate out a more accurate error estimate. The updated error estimate is then passed to the positioning solution program and the servo control program.

7. A calibration system for executing the fully automatic pod calibration method without requiring known ground information according to any one of claims 1 to 6, characterized in that: The calibration system includes a timing calibration module, an acquisition module, a feature point matching module, an error calculation module and an automatic calibration module: The timing calibration module is used to align the timing between the aircraft and the pod sensors; The acquisition module continuously acquires images and aircraft data and pod data at the same time when the pod is used to detect any area during the flight of the aircraft at any time period; The feature point matching module uses a feature point matching algorithm to perform feature point matching on images detected within a period of time selected by the acquisition module; The error calculation module calculates the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error through an iterative optimization function based on the data from the acquisition module and the feature point matching module; The automatic calibration module substitutes the error calculated by the error calculation module into the positioning solution program and the servo control program, and continuously estimates the true value of the attitude during the period when the next detected area still overlaps with the previous area, and transmits it to the positioning solution program and the servo control program at the same time to perform automatic calibration.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images