Active and passive three-dimensional imaging real-time processing system and method

Abstract

The present disclosure provides a real-time processing system and method for active and passive three-dimensional imaging, including: a PS end, connected to a peripheral end through an external interface controller, using a serial processor to implement low-parallel operations, the PS end including: a system initialization module; a data preprocessing module; a laser ranging data error correction module; an optical image data processing module; a PL end, connected to the PS end through an AXI bus, using programmable logic to implement highly parallel operations; the PL end including: two DMA controllers, a first DMA controller and a second DMA controller; a geodetic coordinate solution module, connected to the first DMA controller; a Gaussian projection module, one end of which is connected to the geodetic coordinate solution module; and a collinear equation solution module, connected to the second DMA controller. The active and passive three-dimensional imaging processing methods and processes are reasonably divided.

Description

Technical Field

[0001] The present disclosure relates to the technical field of high-speed remote sensing data processing, and in particular to a system and method for real-time processing of active and passive three-dimensional imaging. Background Art

[0002] In recent years, the use of remote sensing technology to quickly and accurately obtain multi-dimensional attribute information such as space, texture, spectrum, and material of the observed target has become a research hotspot. As a new type of remote sensing detection technology, active and passive integrated imaging technology integrates active laser radar and passive optical imager through a common optical path. The two types of sensors share the same optical path and use the same observation scanning method. With the three-dimensional imaging processing system, the spatial and spectral dimension information of the observed area can be integrated and displayed in real time in three-dimensional imaging. Therefore, active and passive integrated imaging technology will be widely used in remote sensing technology fields such as urban planning, land use, environmental monitoring, and disaster assessment.

[0003] However, with the continuous increase in payload data acquisition capabilities, the amount of remote sensing data has exploded exponentially, which has put forward an urgent need for high-speed real-time processing capabilities for remote sensing data. On the other hand, active and passive three-dimensional imaging processing requires the simultaneous processing of optical payload image data, lidar ranging data, and corresponding position and posture data. The processing process involves a large amount of matrix operations and transcendental operations of double-precision floating-point data, which is both data-intensive and computationally intensive. In addition, the system's volume, power consumption and other restrictions must also be considered comprehensively. The above factors pose major challenges to the design of active and passive three-dimensional imaging systems.

[0004] The development and maturity of SoC (System on Chip) design technology has opened up new development space for low-power, high-speed data processing. It integrates high-performance processing cores, accelerator modules, dedicated modules, memory and other computing and storage devices into a chip, effectively reducing and increasing the data transmission bandwidth, and achieving higher system performance. Software and hardware collaborative design technology fully considers the inseparable relationship between software and hardware in SOC, so that software design and hardware design are designed in parallel as an organic whole, giving full play to the advantages of each software and hardware, so that the system can achieve efficient working capabilities.

[0005] However, the design of active and passive 3D imaging processing SOC system using software and hardware collaborative methods needs to focus on the following two problems: On the one hand, how to achieve a reasonable division of software and hardware tasks based on known hardware resources so that the system has good performance in terms of time consumption, hardware resource consumption and power consumption. On the other hand, how to achieve efficient interaction between software and hardware is the key to improving the overall performance of the system.

[0006] Public Content

[0007] 1. Technical issues to be resolved

[0008] Based on the above problems, the present disclosure provides an active and passive three-dimensional imaging real-time processing system and method to alleviate the technical problems of the active and passive three-dimensional imaging processing system and method in the prior art, such as poor performance and high power consumption.

[0009] (II) Technical solution

[0010] In one aspect of the present disclosure, there is provided an active and passive three-dimensional imaging real-time processing system, comprising:

[0011] The PS end is connected to the peripheral end through an external interface controller, and uses a serial processor to implement operations with low parallelism. The PS end includes:

[0012] System initialization module, used to complete the initialization configuration after the system is powered on;

[0013] The data preprocessing module is used to complete the precise time registration of optical image data, laser ranging data and position and attitude data, and use the position and attitude data to complete the calculation of the rotation matrix and translation parameters in the laser point cloud solution model, as well as the calculation of the exterior orientation elements of the optical image;

[0014] The laser ranging data error correction module is used to perform laser ranging data error correction processing on the ranging value;

[0015] The optical image data processing module uses the optical image data and the corresponding exterior orientation elements and laser point cloud data to perform data calculations;

[0016] The PL end is connected to the PS end through an AXI bus and uses programmable logic to implement highly parallel operations; the PL end includes:

[0017] Two DMA controllers, namely the first DMA controller and the second DMA controller, are used for high-speed data transmission between the PS side and the PL side;

[0018] A geodetic coordinate solution module is connected to the first DMA controller and solves the geodetic coordinates of each laser point based on a three-dimensional point cloud solution model of an array push-broom airborne laser radar;

[0019] A Gaussian projection module, one end of which is connected to the geodetic coordinate solution module, for implementing Gaussian plane rectangular coordinate solution of the laser point and transmitting the solution to the PS end memory through the first DMA controller;

[0020] The collinear equation solving module is connected to the second DMA controller and is used to solve the ground coordinates of the optical image points corresponding to the ground points.

[0021] In the embodiment of the present disclosure, the operations with lower parallelism include: data preprocessing, laser ranging data error correction, and elevation interpolation.

[0022] In the disclosed embodiment, the highly parallelized operations include: geodetic coordinate solution, Gaussian projection, and collinear equation solution.

[0023] In the disclosed embodiment, the laser ranging data error correction process includes: system error correction and gross error point filtering.

[0024] In the disclosed embodiment, the collinear equation solving module includes: a controller, an operation unit, an input data cache BRAM, an output data cache BRAM, and a parameter cache register.

[0025] In another aspect of the present disclosure, a method for real-time processing of active and passive three-dimensional imaging is provided, and the real-time processing of active and passive three-dimensional imaging is performed based on any of the above-mentioned real-time processing systems of active and passive three-dimensional imaging, and the real-time processing method of active and passive three-dimensional imaging comprises:

[0026] Step S1: accurately aligning the original data of the observed load through the time code to complete the matching of multiple load data in the time dimension; the original data of the observed load includes: optical image data, laser ranging data, and position and posture data;

[0027] Step S2: using the position and posture data, respectively calculating the rotation matrix and translation parameters for coordinate transformation from the laser sensor coordinate system to the earth coordinate system, and the exterior orientation elements of the optical line array image at the imaging moment;

[0028] Step S3: Use the calibration parameters to complete the system error correction of the laser ranging data and perform rapid gross error correction;

[0029] Step S4: using laser ranging data, the rotation matrix and translation parameters of coordinate transformation, based on the array push-broom airborne laser radar three-dimensional point cloud solution model, solve the geodetic coordinates of each laser point;

[0030] Step S5: solving the Gaussian plane rectangular coordinates of each laser point;

[0031] Step S6: Calculate the elevation value of the image point based on the correspondence between the optical camera image point and the laser point in the optical image data and in combination with the laser point cloud result;

[0032] Step S7: Calculate the ground coordinates of each image point corresponding to the ground point according to the collinear equation based on the coordinates of the optical image points and the corresponding elevation, the internal and external orientation elements of the optical image; and

[0033] Step S8: Assign a gray value to each pixel of the optical image to obtain a three-dimensional image, thereby completing real-time processing of active and passive three-dimensional imaging.

[0034] In the disclosed embodiment, in step S3, gross error points in the distance measurement value are filtered out according to the distance threshold, and the missing points are supplemented by an interpolation method, thereby achieving rapid gross error correction.

[0035] In the disclosed embodiment, the step S5 obtains the laser point cloud result after Gaussian projection.

[0036] In the disclosed embodiment, in step S6, the elevation values ​​of the image points are calculated using a linear interpolation method based on the correspondence between the optical camera image points in the optical image data and the laser points in the laser ranging data, combined with the laser point cloud results.

[0037] In the embodiment of the present disclosure, step S7 includes:

[0038] Sub-step S71: realizing message communication with the second DMA controller through the controller; realizing enabling control of the operation unit; realizing read and write control of the input data cache BRAM, the output data cache BRAM and the parameter cache register;

[0039] Sub-step S72: receiving the elevation data corresponding to the optical image point and the exterior orientation elements of the image through the second DMA controller, and caching them using the input data cache BRAM;

[0040] Sub-step S73: reading image exterior orientation element data from the input data buffer BRAM and writing it into the parameter buffer register for buffering;

[0041] Sub-step S74: The computing unit and FPGA use the image point coordinates, elevation values, and exterior orientation elements at the time of image formation, combined with interior orientation elements, to solve the XY coordinates of the image point according to the collinearity condition equation. The calculation formula is as follows:

[0042]

[0043] Among them, X A , Y A , Z A X is the object space 3D coordinate corresponding to the image point to be solved; S , Y S , Z S is the position of the optical center in the object coordinate system during imaging; a1, a2, a3, b1, b2, b3, c1, c2, c3 are the rotation matrix elements of the optical center relative to the object coordinate system at the time of imaging, generated by three exterior azimuth elements; x, y are the image coordinates of the image point; f is the principal distance of the photo.

[0044] Sub-step S75: enabling the output data cache BRAM to cache the ground coordinate data of the ground point corresponding to the optical image, and finally sending it to the PS end through the second DMA controller.

[0045] (III) Beneficial effects

[0046] It can be seen from the above technical solutions that the active and passive three-dimensional imaging real-time processing system and method disclosed in the present invention has at least one or part of the following beneficial effects:

[0047] (1) It can greatly improve the system computing performance and facilitate low-power and high-speed data processing;

[0048] (2) It overcomes the shortcomings of high system complexity and poor real-time performance caused by algorithm complexity and resource constraints under a single processor architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a schematic diagram of the composition architecture of the active and passive three-dimensional imaging processing system according to the embodiment of the present disclosure.

[0050] Figure 2 It is a flowchart of the active and passive three-dimensional imaging processing method according to an embodiment of the present disclosure.

[0051] Figure 3 It is a schematic diagram of the process architecture of the active and passive three-dimensional imaging processing method according to an embodiment of the present disclosure.

[0052] Figure 4 The figure is a schematic diagram of the working principle and processing flow of the collinear equation solving module in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] The present invention provides a real-time processing system and method for active and passive three-dimensional imaging. Based on the heterogeneous multi-core processor of ZYNQ, ARM and FPGA are used for hardware and software collaborative design, and the active and passive three-dimensional imaging processing algorithms and processes are reasonably divided to give full play to the respective advantages of software and hardware, so that the system can obtain the best performance.

[0054] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0055] In the embodiment of the present disclosure, a real-time processing system for active and passive three-dimensional imaging is provided, Figure 1 , Figure 3 , Figure 4 As shown, the active and passive three-dimensional imaging real-time processing system includes:

[0056] The PS (Processing System) end is connected to the peripheral end through an external interface controller, and uses a serial processor to implement low-parallel operations such as data preprocessing, laser ranging data error correction, and elevation interpolation. The PS end includes:

[0057] System initialization module, used to complete the initialization configuration after the system is powered on;

[0058] The data preprocessing module is used to complete the precise time registration of optical image data, laser ranging data and position and attitude data, and use the position and attitude data to complete the calculation of the rotation matrix and translation parameters in the laser point cloud solution model, as well as the calculation of the exterior orientation elements of the optical image;

[0059] The laser ranging data error correction module uses calibration parameters, distance thresholds, etc. to perform system error correction, gross error point filtering, and other laser ranging data error correction processes on the ranging value;

[0060] The optical image data processing module uses the optical image data and the corresponding exterior orientation elements and laser point cloud data to perform elevation interpolation, data fusion and other calculations;

[0061] The PL (Programmable Logic) end is connected to the PS end through the AXI bus, and uses programmable logic to implement highly parallel operations such as geodetic coordinate solution, Gaussian projection, and collinear equation solution; the PL end includes:

[0062] Two DMA (Direct Memory Access) controllers, namely the first DMA controller (AXI_DMA1) and the second DMA controller (AXI_DMA2), are used for high-speed data transfer between the PS side and the PL side;

[0063] A geodetic coordinate solution module is connected to the first DMA controller and solves the geodetic coordinates of each laser point based on a three-dimensional point cloud solution model of an array push-broom airborne laser radar;

[0064] A Gaussian projection module, one end of which is connected to the geodetic coordinate solution module, for implementing Gaussian plane rectangular coordinate solution of the laser point and transmitting the solution to the PS end memory through the first DMA controller;

[0065] The collinear equation solving module is connected to the second DMA controller and is used to solve the ground coordinates of the optical image points corresponding to the ground points.

[0066] The peripheral device is used to obtain the raw data of the observation load and the real-time display of the three-dimensional image, including:

[0067] Position and attitude measuring equipment, used to provide position and attitude data;

[0068] LiDAR, which provides laser ranging data;

[0069] an optical camera for providing optical image data, and

[0070] The image display device is used for real-time display of three-dimensional images; the system initialization module first implements the initialization configuration of external interfaces, interrupts, timers, etc.; and then completes the calibration parameter configuration of the PS end and the PL end.

[0071] The data preprocessing module first combines the interpolation method to complete the precise time alignment of the optical image data, laser ranging data, and position and attitude data; and uses the position and attitude data to complete the rotation matrix and translation parameter calculations for coordinate transformation from the laser sensor coordinate system to the WGS84 geodetic coordinate system, as well as the calculation of the external orientation elements of the optical image; the preprocessed data will be sent to the DDR3 cache. At the same time, in order to support the calculation of the geodetic coordinate solution module and the collinear equation solution module in the PL end, the PS end transmits the rotation matrix and translation parameters and the external orientation elements of the optical image to these two modules respectively.

[0072] The laser ranging data error correction module uses the calibration parameters to complete the ranging data calibration and complete the system error correction of the ranging value; it filters out the gross error points in the ranging data according to the distance threshold, and uses the interpolation method to supplement the missing points, thereby realizing the rapid gross error correction of the laser ranging data. At the same time, in order to realize the collaborative calculation with the geodetic coordinate solution module and Gaussian projection module in the PL end, this module will realize the interaction and control of laser ranging data, laser point coordinate data, etc.

[0073] The optical image data processing module performs elevation interpolation, data fusion and other calculations on the optical image data and the corresponding exterior orientation elements and laser point cloud data; at the same time, in order to achieve collaborative calculation with the PL-side collinear equation solving module, this module will realize the interaction and control of optical image geometric data, optical image exterior orientation elements and other data.

[0074] The DMA controller includes a first DMA controller (AXI_DMA1) and a second DMA controller (AXI_DMA2). The data interaction and control between the PS and PL ends adopts the DMA high-speed transmission method based on the AXI-4 protocol, and is implemented using the Xilinx official IP core AXI DMA (AXI Direct Memory Access). The AXI-4 protocol has independent read and write data channels, supports low-cost direct memory access DMA transmission, can realize data burst transmission without the intervention of the ARM core CPU, completes the high-speed movement of the entire block of data, and has extensive IP scalability. The PL end is equipped with DMA controllers AXI_DMA1 and AXI_DMA2 for the geodetic coordinate solution module, Gaussian projection module, and collinear equation solution module, respectively, to support high-speed transmission of data between the PS end memory and the PL end hardware accelerator.

[0075] The geodetic coordinate solution module is mainly based on the array push-broom airborne laser radar three-dimensional point cloud solution model, and carries out the geodetic coordinate solution of each laser point. It has strong parallelism, so the use of programmable logic resources is conducive to improving the computing performance. This module uses the pipeline parallel processing method to effectively optimize the running performance of the loop function. This module interacts with the PS-side laser ranging data error correction module and the data preprocessing module through AXI_DMA1 to obtain laser ranging data, rotation matrix and translation parameters. The processing results are transmitted to the Gaussian projection module using FIFO (First Input First Output).

[0076] The Gaussian projection module is mainly used to realize the conversion of laser points from the WGS84 geodetic coordinate system to the Gaussian plane rectangular coordinate system. During the calculation process, each laser point needs to be solved point by point to obtain the laser point cloud result after Gaussian projection. This module adopts a pipeline parallel method to realize multi-point data parallel processing. This module obtains the geodetic coordinate value through FIFO, and then after further completing the data processing, the solution result is transferred to the PS end memory through AXI_DMA1.

[0077] The collinear equation solving module is used to realize the ground coordinate solution of each image point corresponding to the ground point of the optical image. The module uses programmable logic resources, adopts pipeline parallelism, and reasonably uses registers to cache key data to improve data access rate. The module interacts with the optical image data processing module and the data preprocessing module on the PS side through AXI_DMA2 to obtain optical image data and the external orientation elements of the optical image, and returns the processing results to the optical image data processing module.

[0078] The external interface controller includes: controllers of 232 interface, 422 interface, CameraLink interface and Ethernet interface, among which the first three interfaces realize data communication with external observation payload equipment, and the external observation payload equipment includes: position / attitude measurement equipment, laser radar, optical camera and other equipment; the Ethernet interface realizes communication with image display equipment and supports real-time display and storage of three-dimensional images.

[0079] The present disclosure also provides a real-time processing method for active and passive 3D imaging. In the design of software and hardware division, not only the computing characteristics and processor characteristics should be considered, but also the cohesion and coupling characteristics of the processing algorithm should be analyzed. The active and passive 3D imaging processing algorithm is analyzed and decomposed into relatively independent and complete modules, including: time registration module, position and posture data processing, geodetic coordinate solution module, Gaussian projection module, elevation interpolation module, collinear equation solution module, data fusion module, combined with Figures 1 to 4 As shown, the active and passive three-dimensional imaging real-time processing method includes:

[0080] Step S1: The time configuration module accurately aligns the observed payload raw data through the time code to complete the matching of multiple payload data in the time dimension; the observed payload raw data includes: optical image data, laser ranging data, and position and posture data;

[0081] Step S2: The exterior orientation element solving module uses the position and posture data to respectively calculate the rotation matrix and translation parameters for coordinate transformation from the laser sensor coordinate system to the WGS84 geodetic coordinate system, as well as the exterior orientation elements of the optical linear array image at the imaging time;

[0082] Step S3: The ranging data error correction module uses the calibration parameters to complete the system error correction of the laser ranging data and perform rapid gross error correction;

[0083] Step S4: The geodetic coordinate solution module uses the laser ranging data, the rotation matrix and the translation parameters of the coordinate transformation, based on the array push-broom airborne laser radar three-dimensional point cloud solution model, to solve the geodetic coordinates of each laser point;

[0084] Step S5: The Gaussian projection module solves the Gaussian plane rectangular coordinates of each laser point;

[0085] Step S6: The elevation interpolation module calculates the elevation value of the image point based on the correspondence between the optical camera image point and the laser point in the optical image data and in combination with the laser point cloud result;

[0086] Step S7: the collinear equation solving module solves the ground coordinates of each image point corresponding to the ground point according to the collinear equation based on the coordinates of the optical image points and the corresponding elevation, the inner and outer orientation elements of the optical image;

[0087] Step S8: The data fusion module assigns a gray value to each pixel of the optical image to obtain a three-dimensional image, thereby completing real-time processing of active and passive three-dimensional imaging.

[0088] In step S1, the operation is simple, but it needs to perform three types of load data reading, time code extraction, data registration and packaging, etc., which puts a lot of pressure on data access. The multi-level cache design combining SRAM (Static Random-Access Memory) and Cache (high-speed cache memory) on the PS side is conducive to improving data access efficiency. It is allocated to the PS side for implementation.

[0089] In the steps S2 and S3, the calculation of the exterior orientation elements and the error correction of the laser ranging data are relatively simple, but there are many judgment operations and transcendental operations, etc., which are suitable for software implementation. Therefore, they are allocated to the PS end for implementation.

[0090] In step S3, gross error points in the distance measurement value are filtered out according to the distance threshold, and the missing points are supplemented by an interpolation method, thereby realizing rapid gross error correction.

[0091] In step S5, the laser point cloud result after Gaussian projection is obtained.

[0092] In step S6, the elevation values ​​of the image points are calculated using a linear interpolation method based on the correspondence between the optical camera image points in the optical image data and the laser points in the laser ranging data, combined with the laser point cloud results;

[0093] In step S6, the elevation interpolation process needs to determine the flag of each laser point. A large number of determination operations are suitable for implementation in software, and therefore, are allocated to the PS end for implementation.

[0094] In step S8, the data fusion process reads, assigns values, and stores the optical image point data point by point. Therefore, the data access pressure is large and suitable for software implementation, which is allocated to the PS end for implementation;

[0095] In the steps S4, S5 and S7, coordinates are solved point by point for the laser points or optical image points. There is no data dependency between different laser points and image points, and the degree of parallelism is high, which is in line with the characteristic of programmable logic being suitable for repeated operations. Therefore, it is allocated to the PL end for implementation.

[0096] In the embodiment of the present disclosure, in step S7, the collinear equation solving module uses the collinear equation to solve the geodetic coordinates of each optical image point, such as Figure 4As shown, the collinear equation solving module includes: a controller, a computing unit, an input data cache BRAM (Block Random Access Memory), an output data cache BRAM and a parameter cache register. The step S7 includes:

[0097] Sub-step S71: Implement task control, status feedback and other message communications with the second DMA controller through the controller; implement enable control of the operation unit; implement read and write control of the input data cache BRAM, output data cache BRAM and parameter cache register.

[0098] Sub-step S72: receiving the elevation data corresponding to the optical image point and the exterior orientation elements of the image through the second DMA controller, and caching them using the input data cache BRAM;

[0099] Sub-step S73: Read the image exterior orientation element data from the input data buffer BRAM and write it into the parameter buffer register for buffering. The amount of exterior orientation element data is small and is frequently called during the solution process. Using the parameter buffer register can improve access efficiency.

[0100] Sub-step S74: The XY coordinates of the image point are calculated by the computing unit and with the help of the DSP and LUT resources inside the FPGA, using the image point coordinates, elevation values, and the exterior orientation elements at the time of image formation, combined with the interior orientation elements, according to the collinearity condition equation. The calculation formula is as follows:

[0101]

[0102] Among them, (X A , Y A , Z A ) is the object space three-dimensional coordinate corresponding to the image point to be solved; (X S , Y S , Z S ) is the position of the optical center in the object coordinate system during imaging; a1, a2, a3, b1, b2, b3, c1, c2, c3 are the rotation matrix elements of the optical center relative to the object coordinate system at the moment of imaging, which are generated by three exterior azimuth elements; x, y are the image coordinates of the image point; f is the principal distance of the photo.

[0103] Sub-step S75: enabling the output data cache BRAM to cache the ground coordinate data of the ground point corresponding to the optical image, and finally sending it to the PS end through the second DMA controller.

[0104] Data communication between the PL side and the PS side is an important part of realizing hardware-software collaborative processing. Considering the characteristics of small local storage space and fast access speed on the PL side and large storage space and slow access speed on the PS side, in order to obtain the maximum transmission performance of the PS side and the PL side, a direct memory access high-speed transmission method based on the AXI-4 protocol is adopted to support the rapid movement of data blocks without CPU intervention. The DMA controller is controlled by both the PL side hardware accelerator and the PS side CPU, thereby realizing the interaction of data between the PL side BRAM and the PS side memory. It should be noted that DMA transmission inevitably introduces cache consistency issues, so attention should be paid to the maintenance of cache data in the design.

[0105] So far, the embodiments of the present disclosure have been described in detail in conjunction with the accompanying drawings. It should be noted that the implementation methods not shown or described in the drawings or the body of the specification are all forms known to ordinary technicians in the relevant technical field and are not described in detail. In addition, the above definitions of each element and method are not limited to the various specific structures, shapes or methods mentioned in the embodiments, and ordinary technicians in the field can simply change or replace them.

[0106] Based on the above description, those skilled in the art should have a clear understanding of the active and passive three-dimensional imaging real-time processing system and method disclosed in the present invention.

[0107] In summary, the present disclosure provides a real-time processing system and method for active and passive three-dimensional imaging, which divides the functional modules according to the processing characteristics and advantages of ARM and FPGA, and then carries out the design of data processing, caching, interface, task control, etc. on the PS and PL sides, so as to realize the efficient collaboration of software and hardware. It overcomes the shortcomings of high system complexity and poor real-time performance caused by algorithm complexity and resource limitation under a single processor architecture.

[0108] It should also be noted that the directional terms mentioned in the embodiments, such as "upper", "lower", "front", "back", "left", "right", etc., are only reference directions of the drawings and are not intended to limit the scope of protection of the present disclosure. Throughout the drawings, the same elements are represented by the same or similar reference numerals. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present disclosure.

[0109] Moreover, the shapes and sizes of the components in the figures do not reflect the real size and proportion, but only illustrate the contents of the embodiments of the present disclosure. In addition, in the claims, any reference symbols between brackets shall not be constructed as limiting the claims.

[0110] Unless otherwise indicated, the numerical parameters in this specification and the appended claims are approximate values ​​and can vary according to the desired properties obtained through the content of the present disclosure. Specifically, all numbers used in the specification and claims to express the content of the composition, reaction conditions, etc., should be understood to be modified by the term "about" in all cases. In general, the meaning of the expression is to include a variation of ±10% in some embodiments, ±5% in some embodiments, ±1% in some embodiments, and ±0.5% in some embodiments by a specific number.

[0111] Furthermore, the word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

[0112] The ordinal numbers used in the specification and claims, such as "first", "second", "third", etc., to modify the corresponding elements, do not themselves mean that the elements have any ordinal numbers, nor do they represent the order of one element and another element, or the order in the manufacturing method. The use of these ordinal numbers is only used to clearly distinguish a component with a certain name from another component with the same name.

[0113] In addition, unless the steps are specifically described or must occur in sequence, the order of the above steps is not limited to the above list, and can be changed or rearranged according to the required design. And the above embodiments can be mixed and matched with each other or with other embodiments based on design and reliability considerations, that is, the technical features in different embodiments can be freely combined to form more embodiments.

[0114] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose. Furthermore, in a unit claim that lists several devices, several of these devices may be embodied by the same hardware item.

[0115] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various disclosed aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: the claimed disclosure requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the disclosed aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present disclosure.

[0116] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above description is only a specific embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A real-time processing system for active and passive three-dimensional imaging, comprising: The PS end is connected to the peripheral end through an external interface controller, and uses a serial processor to implement operations with low parallelism. The operations with low parallelism include: data preprocessing, laser ranging data error correction, and elevation interpolation. The PS end includes: System initialization module, used to complete the initialization configuration after the system is powered on; The data preprocessing module is used to complete the precise time registration of optical image data, laser ranging data and position and attitude data, and use the position and attitude data to complete the calculation of the rotation matrix and translation parameters in the laser point cloud solution model, as well as the calculation of the exterior orientation elements of the optical image; The laser ranging data error correction module is used to perform laser ranging data error correction processing on the ranging value; The optical image data processing module uses the optical image data and the corresponding exterior orientation elements and laser point cloud data to perform data calculations; The PL end is connected to the PS end through an AXI bus, and uses programmable logic to implement highly parallel operations, including: geodetic coordinate solution, Gaussian projection, and collinear equation solution; the PL end includes: Two DMA controllers, namely the first DMA controller and the second DMA controller, are used for high-speed data transmission between the PS side and the PL side; A geodetic coordinate solution module is connected to the first DMA controller, and uses laser ranging data, a rotation matrix and translation parameters of coordinate conversion to solve the geodetic coordinates of each laser point based on a three-dimensional point cloud solution model of an array push-broom airborne laser radar; A Gaussian projection module, one end of which is connected to the geodetic coordinate solution module, for implementing Gaussian plane rectangular coordinate solution of the laser point and transmitting the solution to the PS end memory through the first DMA controller; The collinear equation solving module is connected to the second DMA controller and is used to solve the ground coordinates of the optical image points corresponding to the ground points.

2. According to the active and passive three-dimensional imaging real-time processing system of claim 1, the laser ranging data error correction process comprises: System error correction and gross error filtering.

3. According to the active and passive three-dimensional imaging real-time processing system of claim 1, the collinear equation solving module comprises: Controller, arithmetic unit, input data cache BRAM, output data cache BRAM, and parameter cache register.

4. A method for real-time processing of active and passive three-dimensional imaging, based on the real-time processing system for active and passive three-dimensional imaging according to any one of claims 1 to 3, the method for real-time processing of active and passive three-dimensional imaging comprising: Step S1: The original data of the observed load is accurately registered through the time code to complete the matching of multiple load data in the time dimension; The observed payload raw data includes: optical image data, laser ranging data, and position and posture data; Step S2: using the position and posture data, respectively calculating the rotation matrix and translation parameters for coordinate transformation from the laser sensor coordinate system to the earth coordinate system, and the exterior orientation elements of the optical line array image at the imaging moment; Step S3: Use the calibration parameters to complete the system error correction of the laser ranging data and perform rapid gross error correction; Step S4: using laser ranging data, the rotation matrix and translation parameters of coordinate transformation, based on the array push-broom airborne laser radar three-dimensional point cloud solution model, solve the geodetic coordinates of each laser point; Step S5: solving the Gaussian plane rectangular coordinates of each laser point; Step S6: Calculate the elevation value of the image point based on the correspondence between the optical camera image point and the laser point in the optical image data and in combination with the laser point cloud result; Step S7: Calculate the ground coordinates of each image point corresponding to the ground point according to the collinear equation based on the coordinates of the optical image points and the corresponding elevation, the internal and external orientation elements of the optical image; and Step S8: Assign a gray value to each pixel of the optical image to obtain a three-dimensional image, thereby completing real-time processing of active and passive three-dimensional imaging.

5. According to the real-time processing method for active and passive three-dimensional imaging of claim 4, in step S3, gross error points in the ranging value are filtered out according to the distance threshold, and the missing points are supplemented by an interpolation method, thereby achieving rapid gross error correction.

6. According to the real-time processing method for active and passive three-dimensional imaging according to claim 4, the step S5 obtains the laser point cloud result after Gaussian projection.

7. According to the active and passive three-dimensional imaging real-time processing method of claim 4, in step S6, based on the correspondence between the optical camera image points in the optical image data and the laser points in the laser ranging data, combined with the laser point cloud result, the elevation value of the image point is calculated using a linear interpolation method.

8. The active and passive three-dimensional imaging real-time processing method according to claim 4, wherein step S7 comprises: Sub-step S71: implementing message communication with the second DMA controller through the controller; Realize the enable control of the operation unit; Implement read and write control of input data cache BRAM, output data cache BRAM and parameter cache register; Sub-step S72: receiving the elevation data corresponding to the optical image point and the exterior orientation elements of the image through the second DMA controller, and caching them using the input data cache BRAM; Sub-step S73: reading image exterior orientation element data from the input data buffer BRAM and writing it into the parameter buffer register for buffering; Sub-step S74: The computing unit and FPGA use the image point coordinates, elevation values, and exterior orientation elements at the time of image formation, combined with interior orientation elements, to solve the XY coordinates of the image point according to the collinearity condition equation. The calculation formula is as follows: ； Among them, X A , Y A , Z A X is the object space 3D coordinates corresponding to the image point to be solved; S , Y S , Z S is the position of the optical center in the object coordinate system at the time of imaging; a1, a2, a3, b1, b2, b3, c1, c2, c3 are the rotation matrix elements of the optical center relative to the object coordinate system at the time of imaging, generated by three exterior azimuth elements; x, y are the image coordinates of the image point; f is the principal distance of the photo; and Sub-step S75: enabling the output data cache BRAM to cache the ground coordinate data of the ground point corresponding to the optical image, and finally sending it to the PS end through the second DMA controller.

Images