Full-automatic pod calibration method without known ground information

Abstract

The invention provides a full-automatic pod calibration method without known ground information, and belongs to the field of photoelectric pod error calibration, a pod is installed on an aircraft, time sequence alignment among a differential GPS, an aircraft IMU, a pod encoder and a camera is completed, when the aircraft flies, image, aircraft position attitude, pod frame angle and camera focal length value data are collected, and the pod is calibrated according to the collected data. Utilizing a feature point matching algorithm, selecting feature points of a first image, removing feature points of a moving object, calculating pixel offset, inputting related data into an iterative optimization function, calculating an error, continuously estimating an attitude truth value in a region overlapping time period through a positioning and servo control program, and stopping iteration and storing an error estimation value when a field of view is separated; iteration is carried out on subsequent detection by using the last error estimation value; according to the method, a ground control point, a specific flight route and a high-precision satellite map database are not needed, calibration and additional operation are not needed, and pod calibration can be automatically completed.

Description

Technical Field

[0001] The invention belongs to the technical field of photoelectric pod error calibration, and in particular, relates to a fully automatic pod calibration method without the need for known ground information. Background Art

[0002] When using an airborne optoelectronic pod for target positioning, geographic guidance or locking a region of interest (ROI), the system error (generally speaking, the installation error between the aircraft and the pod, the zero position error of the pod frame angle, and the focal length error of the camera in the pod) has a great impact on the accuracy and the final presentation effect, so each pod needs to be calibrated. The methods used in the past, such as using GPS differential equipment or satellite maps of the corresponding area (such as the method described in patent CN 113415433 B "Pod attitude correction method, device and drone based on three-dimensional scene model") to obtain the precise geographic location information of the ground target points to be observed during flight in advance, locating the calibrated ground target points when the aircraft flies on a planned complex detour route, and obtaining the system error by comparing with the true value, all require the collection of accurate geographic location information of known target points in advance, supplemented by the corresponding detour route.

[0003] This type of method requires auxiliary work to be completed before the pod is installed on the aircraft and begins to perform the mission, including carrying equipment to the aircraft or collecting satellite maps to calibrate the target point, planning the detour route, etc., so the convenience and timeliness are poor. In addition, most of the solutions in the previous methods require the participation of laser ranging, which can easily expose the position of the aircraft during flight and does not meet the needs of covert flight. Patent CN 118603141 B (a pod system error calibration method without knowing the ground position) can complete the pod calibration without the help of ground information, but it requires the operator to complete the operation according to a specific process during the flight. Obviously, this also requires time specifically for pod calibration. This reduces the operability of this method and also interferes with the execution of tasks during flight to a certain extent.

[0004] Considering the above problems, there is an urgent need for a fully automatic optoelectronic pod calibration method that does not require known ground target position information, does not require laser ranging information, and does not require human intervention and dedicated calibration time during flight. Summary of the invention

[0005] In order to solve the problem that previous methods require complex route planning and long-term observation and data collection, the present invention proposes a fully automatic pod calibration method that does not require known ground information, so that before or during the flight, the pod calibration can be completed during the detection process without any additional instructions or advance settings, so that in the subsequent flights of the same flight, functions such as target positioning, geographic guidance, and locking of the region of interest (ROI) can be completed more accurately.

[0006] The present invention is achieved through the following technical solutions: A fully automatic pod calibration method without known ground information: Step 1: Install the pod on the aircraft and align the timing between the aircraft and the pod’s sensors; Step 2: When the aircraft is in flight, the pod is used to detect any area at any time period, and the image and the aircraft data and pod data at the same time are continuously collected; Step 3: Use a feature point matching algorithm to perform feature point matching on the images detected within a period of time selected in step 2; Step 4: Based on the data of steps 2 and 3, the installation error between the aircraft and the pod, the zero position error of the pod frame angle, and the camera focal length error are calculated by iterative optimization function; Step 5: Substitute the error calculated in step 4 into the positioning solution program and the servo control program, and continue to estimate the true value of the attitude while the next detected area still overlaps with the previous area, and transmit it to the positioning solution program and the servo control program at the same time for automatic calibration.

[0007] Furthermore, in step 1, The pod is mounted on the mounting base of the aircraft and the timing alignment is completed before installation on the aircraft; The differential GPS and IMU on the aircraft and the timestamp information of the encoder and camera inside the pod are exported through the communication protocol to align the timing between the sensors.

[0008] Furthermore, in step 2 The aircraft data includes: the latitude output by the aircraft differential GPS ,longitude ,high , the heading output by the aircraft IMU , Pitch , roll ; The pod data includes: frame angle read by the pod encoder , , the camera's current focal length value data .

[0009] Furthermore, in step 3, Select the feature points in the first image as the points to be tested, and use the motion detection algorithm to delete the feature points representing the moving objects, and mark the remaining feature points as , Then calculate the pixel offset of these marked feature points relative to the center of the visual axis at the current moment during the subsequent detection of this area. .

[0010] Furthermore, in step 3, The specific process of calculating the pixel offset is as follows: when the detection pod starts to detect a certain area, all the feature points in the first image are obtained by using the feature point algorithm, and the feature points are matched with the subsequent images. The transmission matrix of the subsequent image relative to the first image is solved through the same feature points of the two images, and the pixel offset of the feature points of the first image relative to the visual axis center of the subsequent image is calculated based on this; If a subsequent image does not have the same feature points as the first image, the pixel offset of the feature points in the first image relative to its own visual axis center is indirectly calculated by comparing it with a previous image that has the same feature points as the current image and has solved the pixel offset of the feature points in the first image relative to its own visual axis center; If an image does not have the same feature points as any previous image, the image and its attached data will be discarded; In this process, a motion detection algorithm is used to filter out the parts of the feature points in the first image that represent moving objects.

[0011] Furthermore, in step 4, The current position and attitude data of the aircraft at each moment, the current frame angle data of the pod, the current focal length value of the camera and the pixel offset of the marked feature point relative to the center of the current image are input into the iterative optimization function to calculate the installation error between the aircraft and the pod and the zero position error of the pod frame angle. and , and camera focal length error ; The installation error includes the heading component , pitch component , Roll component .

[0012] Furthermore, the calibration method further comprises: Step 6: When the pod's field of view leaves the current area, the iterative optimization program stops, and the error estimate calculated during this period is stored in the program; When the pod detects the next area, the previous error estimate is used as the initial value and substituted into the iterative optimization program to iterate out a more accurate error estimate. The updated error estimate is then passed to the positioning solution program and the servo control program.

[0013] A fully automatic pod calibration system that does not require known ground information: The calibration system includes a timing calibration module, an acquisition module, a feature point matching module, an error calculation module and an automatic calibration module: The timing calibration module is used to align the timing between the aircraft and the pod sensors; The acquisition module continuously acquires images and aircraft data and pod data at the same time when the pod is used to detect any area in any time period during the flight of the aircraft; The feature point matching module uses a feature point matching algorithm to perform feature point matching on images detected within a period of time selected by the acquisition module; The error calculation module calculates the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error through an iterative optimization function based on the data of the acquisition module and the feature point matching module; The automatic calibration module substitutes the error calculated by the error calculation module into the positioning solution program and the servo control program, and continuously estimates the true value of the attitude during the period when the next detected area still overlaps with the previous area, and transmits it to the positioning solution program and the servo control program at the same time, so as to perform automatic calibration.

[0014] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0015] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are implemented.

[0016] Beneficial effects of the present invention The present invention solves the limitation of previous methods that require some means to extract and obtain the precise geographical location of ground target points. It also does not require complicated detour routes and laser ranging, and does not require manual participation, special calibration instructions or time periods specifically used for calibration. It greatly improves the convenience, timeliness and concealment of pod error calibration during flight, and provides support for subsequent high-precision target positioning, geographic guidance and tracking functions during the flight. It not only greatly reduces the difficulty of calibration and the complexity of the process, but also brings great convenience to practical applications.

[0017] The present invention can be applied in the fields of error calibration of optoelectronic pods and target positioning and geographic guidance of optoelectronic pods, and can more accurately lock the region of interest (ROI) in line inspection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The experimental methods used in the following examples are conventional methods unless otherwise specified. The materials, reagents, methods and instruments used are conventional materials, reagents, methods and instruments in the art unless otherwise specified, and can be obtained through commercial channels by those skilled in the art.

[0021] like Figure 1 As shown, the present invention proposes a fully automatic pod calibration method without knowing ground information: The method specifically comprises the following steps: Step 1: Install the pod on the aircraft and align the timing between the aircraft and the pod’s sensors; The pod is mounted on the mounting base of the aircraft and the timing alignment is completed before installation on the aircraft; The differential GPS and IMU on the aircraft and the timestamp information of the encoder and camera inside the pod are exported through the communication protocol to align the timing between the sensors.

[0022] The output frequency of the pod encoder can be up to 1000Hz, so the timing alignment can be completed before installation; the output frequency of the differential GPS and IMU on the aircraft is relatively low and has a delay, so it is necessary to use the timestamp information to estimate the delay and find the aircraft position and attitude data that strictly corresponds to the current image formation; Step 2: When the aircraft is in flight, the pod is used to detect any area at any time period, and the image and the aircraft data and pod data at the same time are continuously collected; The aircraft data includes: the latitude output by the aircraft differential GPS ,longitude ,high , the heading output by the aircraft IMU , Pitch , roll ; The pod data includes: frame angle read by the pod encoder , , the camera's current focal length value data .

[0023] During the entire flight, there is no external input of pod calibration instructions, nor is there any pod calibration operation set at any time. The operator can freely control the pod at any time to complete the desired function.

[0024] At any time period, when the pod detects any area, the calibration function inside the pod is automatically activated without human intervention.

[0025] Step 3: Using a feature point matching algorithm, perform feature point matching on the image of a certain area detected within a period of time selected in step 2; Select the feature points in the first image as the points to be tested, and use the motion detection algorithm to delete the feature points representing the moving objects, and mark the remaining feature points as ,It should be noted that the number of marked feature points must be greater than 4; Then calculate the pixel offset of these marked feature points relative to the center of the visual axis at the current moment during the subsequent detection of this area. .

[0026] The specific process of calculating the pixel offset is as follows: when the detection pod starts to detect a certain area, all the feature points in the first image are obtained by using the feature point algorithm, and the feature points are matched with the subsequent images. The transmission matrix of the subsequent image relative to the first image is solved through the same feature points of the two images, and the pixel offset of the feature points of the first image relative to the visual axis center of the subsequent image is calculated based on this; If a subsequent image does not have the same feature points as the first image, the pixel offset of the feature points in the first image relative to its own visual axis center is indirectly calculated by comparing it with a previous image that has the same feature points as the current image and has solved the pixel offset of the feature points in the first image relative to its own visual axis center; If an image does not have the same feature points as any previous image, the image and its attached data will be discarded; In this process, a motion detection algorithm is used to filter out the parts of the feature points in the first image that represent moving objects.

[0027] The specific method of extracting feature points from images and motion detection is not limited. For example, the feature point matching algorithm can use traditional SIFT or SURF, or SuperGlue based on deep learning. When the image quality is average, the feature point matching algorithm can be used to detect feature points after improving the image clarity by using methods such as image fog penetration algorithm (such as dark channel method) and image enhancement algorithm (such as CLAHE). For example, the motion detection algorithm can use a mixed Gaussian model. Users can flexibly select algorithms according to hardware conditions, as long as they can meet the requirements of feature point matching and motion detection.

[0028] Step 4: Based on the data of steps 2 and 3, the installation error between the aircraft and the pod, the zero position error of the pod frame angle, and the camera focal length error are calculated by iterative optimization function; The current position and attitude data of the aircraft at each moment, the current frame angle data of the pod, the current focal length value of the camera and the pixel offset of the marked feature point relative to the center of the current image are input into the iterative optimization function to calculate the installation error between the aircraft and the pod and the zero position error of the pod frame angle. and , and camera focal length error ; The installation error includes the heading component , pitch component , Roll component .

[0029] Step 5: Substitute the error calculated in step 4 into the positioning solution program and the servo control program, and continue to estimate the true value of the attitude while the next detected area still overlaps with the previous area, and transmit it to the positioning solution program and the servo control program at the same time for automatic calibration.

[0030] Step 6: When the pod's field of view leaves the current area (i.e., the pod's field of view is significantly switched to another area, and the current image has no common feature points with the previous frame image), the iterative optimization program stops, and the error estimate calculated during this period is stored in the program; When the pod detects the next area, the previous error estimate is used as the initial value and substituted into the iterative optimization program to iterate out a more accurate error estimate. The updated error estimate is then passed to the positioning solution program and the servo control program.

[0031] A fully automatic pod calibration system that does not require known ground information: The calibration system includes a timing calibration module, an acquisition module, a feature point matching module, an error calculation module and an automatic calibration module: The timing calibration module is used to align the timing between the aircraft and the pod sensors; The acquisition module continuously acquires images and aircraft data and pod data at the same time when the pod is used to detect any area in any time period during the flight of the aircraft; The feature point matching module uses a feature point matching algorithm to perform feature point matching on images detected within a period of time selected by the acquisition module; The error calculation module calculates the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error through an iterative optimization function based on the data of the acquisition module and the feature point matching module; The automatic calibration module substitutes the error calculated by the error calculation module into the positioning solution program and the servo control program, and continuously estimates the true value of the attitude during the period when the next detected area still overlaps with the previous area, and transmits it to the positioning solution program and the servo control program at the same time, so as to perform automatic calibration.

[0032] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0033] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are implemented.

[0034] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, or a flash memory. The volatile memory may be a random access memory, RAM, which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory static RAM, SRAM, dynamic random access memory dynamic RAM, DRAM, synchronous dynamic random access memory synchronous DRAM, SDRAM, double data rate synchronous dynamic random access memory double data rate SDRAM, DDR SDRAM, enhanced synchronous dynamic random access memory enhanced SDRAM, ESDRAM, synchronous connection dynamic random access memory synchlink DRAM, SLDRAM and direct memory bus random access memory direct rambus RAM, DR RAM. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0035] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center through a wired method such as coaxial cable, optical fiber, digital subscriber line digital subscriber line, DSL or wireless such as infrared, wireless, microwave, etc. to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium such as a floppy disk, a hard disk, a tape, an optical medium such as a high-density digital video disc digital video disc, DVD, or a semiconductor medium such as a solid state hard disk solid state disc, SSD, etc.

[0036] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0037] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0038] The above is a detailed introduction to the fully automatic pod calibration method proposed in the present invention that does not require known ground information, and the principles and implementation methods of the present invention are explained. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A fully automatic pod calibration method without known ground information, characterized in that: The method specifically comprises the following steps: Step 1: Install the pod on the aircraft and align the timing between the aircraft and the pod’s sensors; Step 2: When the aircraft is in flight, the pod is used to detect any area at any time period, and the image and the aircraft data and pod data at the same time are continuously collected; Step 3: Use a feature point matching algorithm to perform feature point matching on the images detected within a period of time selected in step 2; Step 4: Based on the data of steps 2 and 3, the installation error between the aircraft and the pod, the zero position error of the pod frame angle, and the camera focal length error are calculated by iterative optimization function; Step 5: Substitute the error calculated in step 4 into the positioning solution program and the servo control program, and continue to estimate the true value of the attitude while the next detected area still overlaps with the previous area, and transmit it to the positioning solution program and the servo control program at the same time for automatic calibration.

2. The calibration method according to claim 1, characterized in that: In step 1, The pod is mounted on the mounting base of the aircraft and the timing alignment is completed before installation on the aircraft; The differential GPS and IMU on the aircraft and the timestamp information of the encoder and camera inside the pod are exported through the communication protocol to align the timing between the sensors.

3. The calibration method according to claim 2, characterized in that: In step 2 The aircraft data includes: the latitude output by the aircraft differential GPS ,longitude ,high , the heading output by the aircraft IMU , Pitch , roll ; The pod data includes: frame angle read by the pod encoder , , the camera's current focal length value data .

4. The calibration method according to claim 3, characterized in that: In step 3, Select the feature points in the first image as the points to be tested, and use the motion detection algorithm to delete the feature points representing the moving objects, and mark the remaining feature points as , Then calculate the pixel offset of these marked feature points relative to the center of the visual axis at the current moment during the subsequent detection of this area. .

5. The calibration method according to claim 4, characterized in that: In step 3, The specific process of calculating the pixel offset is as follows: when the detection pod starts to detect a certain area, all the feature points in the first image are obtained by using the feature point algorithm, and the feature points are matched with the subsequent images. The transmission matrix of the subsequent image relative to the first image is solved through the same feature points of the two images, and the pixel offset of the feature points of the first image relative to the visual axis center of the subsequent image is calculated based on this; If a subsequent image does not have the same feature points as the first image, the pixel offset of the feature points in the first image relative to its own visual axis center is indirectly calculated by comparing it with a previous image that has the same feature points as the current image and has solved the pixel offset of the feature points in the first image relative to its own visual axis center; If an image does not have the same feature points as any previous image, the image and its attached data will be discarded; In this process, a motion detection algorithm is used to filter out the parts of the feature points in the first image that represent moving objects.

6. The calibration method according to claim 5, characterized in that: In step 4, The current position and attitude data of the aircraft at each moment, the current frame angle data of the pod, the current focal length value of the camera and the pixel offset of the marked feature point relative to the center of the current image are input into the iterative optimization function to calculate the installation error between the aircraft and the pod and the zero position error of the pod frame angle. and , and camera focal length error ; The installation error includes the heading component , pitch component , Roll component .

7. The calibration method according to claim 6, characterized in that: The calibration method further comprises: Step 6: When the pod's field of view leaves the current area, the iterative optimization program stops, and the error estimate calculated during this period is stored in the program; When the pod detects the next area, the previous error estimate is used as the initial value and substituted into the iterative optimization program to iterate out a more accurate error estimate. The updated error estimate is then passed to the positioning solution program and the servo control program.

8. A calibration system for executing the fully automatic pod calibration method without known ground information as claimed in any one of claims 1 to 7, characterized in that: The calibration system includes a timing calibration module, an acquisition module, a feature point matching module, an error calculation module and an automatic calibration module: The timing calibration module is used to align the timing between the aircraft and the pod sensors; The acquisition module continuously acquires images and aircraft data and pod data at the same time when the pod is used to detect any area in any time period during the flight of the aircraft; The feature point matching module uses a feature point matching algorithm to perform feature point matching on images detected within a period of time selected by the acquisition module; The error calculation module calculates the installation error between the aircraft and the pod, the pod frame angle zero position error, and the camera focal length error through an iterative optimization function based on the data of the acquisition module and the feature point matching module; The automatic calibration module substitutes the error calculated by the error calculation module into the positioning solution program and the servo control program, and continuously estimates the true value of the attitude during the period when the next detected area still overlaps with the previous area, and transmits it to the positioning solution program and the servo control program at the same time, so as to perform automatic calibration.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images