Time domain resampling data fusion method and system based on multi-task communication scheduling

Abstract

The invention provides a time domain resampling data fusion method and system based on multi-task communication scheduling, and the method comprises the steps: carrying out the matching according to the obtained camera data and the imaging sampling time data added to the position posture data, and removing the unmatched data; generating M rows and N columns of fusion point data according to the adapted position attitude data and camera data, and calculating a normal vector of a projection plane and coordinates of camera pixels in an earth-fixed coordinate system; according to the M-row N-column fusion point data, the normal vector of the projection plane and the coordinate of the pixel in the earth-fixed coordinate system, clustering the projection pixel scatter point to the grid angular point, namely the fusion point, and storing the fusion point data; and fusing the scatter information and the fusion point position information in the fusion point data into required information. The method can be conveniently deployed in a line-on-satellite embedded system, and the clustering fusion algorithm is realized by adopting the FPGA to process the fusion algorithm which belongs to a calculation-intensive fusion algorithm, so that the method has good real-time performance.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a time domain resampling data fusion method and system based on multi-task communication scheduling. Background Art

[0002] Satellites are equipped with multiple imaging systems that scan and image the ground in different ways, and then send the pixels together with the imaging time information in the form of data packets to information processing equipment. Clustering pixels imaged at the same location, different imaging devices, different bands, and different times, performing information fusion, and generating different data products is a common application scenario. Generally, the images of each imaging system are transmitted back to the ground, and then pixel clustering and information fusion are performed. With the continuous increase of image data and the limitation of satellite communication bandwidth, clustering and fusion need to be processed in real time on the satellite, and the clustering algorithms of existing ground systems are often not applicable to satellite-borne embedded systems.

[0003] Implementing clustering and fusion algorithms on embedded hardware requires solving two problems: one is how to make the algorithm itself lightweight; the other is how to deeply couple with embedded hardware to fully utilize resources. The imaging data of various imaging systems and satellite position and attitude data often have different sampling frequencies and transmission delays. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a time domain resampling data fusion method and system based on multi-task communication scheduling.

[0005] A method for fusion of spatial domain data based on heterogeneous computing time domain resampling provided by the present invention includes:

[0006] Step S1: matching the acquired camera data and the imaging sampling time data attached to the position and posture data, and removing the unmatched data;

[0007] Step S2: Generate M rows and N columns of fusion point data according to the adapted position and posture data and camera data, and calculate the normal vector of the projection plane and the coordinates of the camera pixel in the ground-fixed coordinate system;

[0008] Step S3: clustering the projected pixel scattered points to the grid corner points, i.e., the fusion points, according to the M rows and N columns of fusion point data, the projection plane normal vector and the coordinates of the pixels in the ground-fixed coordinate system, and storing the fusion point data;

[0009] Step S4: Fusing the scattered point information and the fusion point position information in the fusion point data into required information.

[0010] Preferably, in step S3, a two-dimensional array with a length of M*N is used to store the fusion point data, and the array elements record pixel scatter point information or pixel scatter point information and fusion point position information.

[0011] Preferably, the step S2 includes a fusion point generation step, a projection plane calculation step, a camera pixel scattered point position posture data time resampling step and a camera pixel scattered point positioning calibration step;

[0012] Fusion point generation step: Calculate the sub-satellite point position at the sampling time according to the adapted position and attitude data, calculate the coordinates of N corner points on the earth's spherical surface according to the set vertical track resolution and the vertical track field angles of all cameras, and the set vertical track resolution, and obtain M rows and N columns of grid corner point data;

[0013] Projection plane calculation steps: cache M rows of fusion point data and calculate the normal vector of the projection plane;

[0014] The time resampling step of the camera pixel scattered point position and attitude data: according to the adapted position and attitude data and all camera data, the position and attitude data of the camera data i pixel scattered points are calculated before and after the imaging sampling time of the camera data i;

[0015] Camera pixel scatter point positioning and calibration step: Calculate the coordinates of the pixel in the ground-fixed coordinate system based on the position and posture data of the pixel scatter point i of the camera data.

[0016] Preferably, step S3 comprises:

[0017] The fusion points are projected onto the projection plane to form a grid in the projection plane. These fusion points are called projection fusion points, these grids are called projection grids, and the area where they are located is the projection grid area; the pixel scattered points are projected onto the projection plane, and these points are called projection pixel scattered points;

[0018] Establish an affine auxiliary grid, the row length of the auxiliary grid is equal to the row length of the projection grid at the corresponding position near the center line of the projection grid area; the column length of the auxiliary grid is equal to the column length of the projection grid near the center position of the projection grid area or equal to the column length of the projection grid at the corresponding position near the center line of the projection grid area;

[0019] Clustering the projected pixel scatter points onto an auxiliary grid;

[0020] The projected pixel scatter points are corrected to the projected grid, and the corresponding pixel scatter points are clustered to the corresponding spherical grid.

[0021] Preferably, the synchronous adaptation step S1 is implemented in the form of a task, the fusion point generation step and the projection plane calculation step are combined into one task, the camera data i pixel scattered point position posture data time area resampling step and the camera data i scattered point positioning calibration step are combined into one module, and the grid clustering step S3 and the data fusion step S4 are combined into one task;

[0022] A data-driven multi-task communication scheduling architecture is adopted, and the data communication interface between tasks is in the form of a queue.

[0023] According to the present invention, a system based on heterogeneous computing time domain resampling space domain data fusion is provided, comprising:

[0024] Module M1: Match the acquired camera data and the imaging sampling time data attached to the position and posture data, and remove the mismatched data;

[0025] Module M2: Generate M rows and N columns of fusion point data based on the adapted position and posture data and camera data, and calculate the normal vector of the projection plane and the coordinates of the camera pixel in the ground-fixed coordinate system;

[0026] Module M3: clustering the projected pixel scattered points to the grid corner points, i.e., the fusion points, according to the M rows and N columns of fusion point data, the projection plane normal vector and the coordinates of the pixels in the ground-fixed coordinate system, and storing the fusion point data;

[0027] Module M4: Fusing the scattered point information and the fusion point position information in the fusion point data into the required information.

[0028] Preferably, a two-dimensional array with a length of M*N is used in the module M3 to store the fusion point data, and the array elements record pixel scatter point information or pixel scatter point information and fusion point position information.

[0029] Preferably, the module M2 includes a fusion point generation module, a projection plane calculation module, a camera pixel scattered point position posture data time resampling module and a camera pixel scattered point positioning calibration module;

[0030] Fusion point generation module: Calculate the sub-satellite point position at the sampling time according to the adapted position and attitude data, calculate the coordinates of N corner points on the earth's spherical surface according to the set vertical track resolution and the vertical track field angles of all cameras, and the set vertical track resolution, and obtain M rows and N columns of grid corner point data;

[0031] Projection plane calculation module: cache M rows of fusion point data and calculate the normal vector of the projection plane;

[0032] Camera pixel scattered point position and attitude data time resampling module: calculates the position and attitude data of the scattered points of camera data i pixel based on the adapted position and attitude data and all camera data before and after the imaging sampling time of camera data i;

[0033] Camera pixel scatter point positioning and calibration module: calculates the coordinates of the pixel in the ground-fixed coordinate system based on the position and posture data of the pixel scatter point i of the camera data.

[0034] Preferably, the module M3 comprises:

[0035] The fusion points are projected onto the projection plane to form a grid in the projection plane. These fusion points are called projection fusion points, these grids are called projection grids, and the area where they are located is the projection grid area; the pixel scattered points are projected onto the projection plane, and these points are called projection pixel scattered points;

[0036] Establish an affine auxiliary grid, the row length of the auxiliary grid is equal to the row length of the projection grid at the corresponding position near the center line of the projection grid area; the column length of the auxiliary grid is equal to the column length of the projection grid near the center position of the projection grid area or equal to the column length of the projection grid at the corresponding position near the center line of the projection grid area;

[0037] Clustering the projected pixel scatter points onto an auxiliary grid;

[0038] The projected pixel scatter points are corrected to the projected grid, and the corresponding pixel scatter points are clustered to the corresponding spherical grid.

[0039] Preferably, the synchronous adaptation module M1 is implemented in the form of a task, the fusion point generation module and the projection plane calculation module are combined into one task implementation, the camera data i pixel scattered point position posture data time area resampling module and the camera data i scattered point positioning and calibration module are combined into one module implementation, and the grid clustering module M3 and the data fusion module M4 are combined into one task implementation;

[0040] A data-driven multi-task communication scheduling architecture is adopted, and the data communication interface between tasks is in the form of a queue.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention adopts a data-driven multi-tasking software architecture and is deeply coupled with multi-core processor hardware, which can be easily deployed in a satellite-borne embedded system. The clustering fusion algorithm is realized by using FPGA to process the computationally intensive fusion algorithm and has good real-time performance.

[0043] 2. The clustering algorithm involved in the present invention is a memory-intensive calculation suitable for processing by a CPU processor. The processed result uses the data structure of the present invention to store necessary information, which is suitable for subsequent calculation-intensive data fusion calculations on FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0045] Figure 1 It is a schematic diagram of the structure of the system modules of the present invention.

[0046] Figure 2 This is the corner point section relationship diagram.

[0047] Figure 3 It is a top view of the projection plane. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0049] Embodiment 1

[0050] According to a time domain resampling data fusion method based on multi-task communication scheduling provided by the present invention, Figures 1 to 3 As shown, including:

[0051] Step S1: Match the acquired camera data and the imaging sampling time data attached to the position and posture data, and remove the unmatched data; the camera data 1 to n and the position and posture data arrive at the synchronous adaptation module in sequence according to the sampling time, and the imaging sampling time intervals of the camera data 1 to n and the position and posture data are different, and the transmission delay is different. Match the imaging sampling time data attached to the data 1 to n and the position and posture data, and remove the data whose sampling time is too early to match with other data for grid clustering. The adapted position and posture data generates fusion points, and the adapted position and posture data and camera data 1 perform time and area resampling of the pixel scattered position and posture data of camera data 1, and the adapted position and posture data and camera data 2 perform time and area resampling of the pixel scattered position and posture data of camera data 2, and so on, until the adapted position and posture data and camera data n perform time and area resampling of the pixel scattered position and posture data of camera data n.

[0052] Step S2: Generate M rows and N columns of fusion point data according to the adapted position and attitude data and camera data, and calculate the projection plane normal vector and the coordinates of the camera pixel in the ground-fixed coordinate system; Step S2 includes a fusion point generation step, a projection plane calculation step, a camera pixel scattered point position and attitude data time resampling step, and a camera pixel scattered point positioning and calibration step.

[0053] Fusion point generation step: Calculate the sub-satellite point position at the sampling time according to the adapted position and attitude data, calculate the coordinates of N corner points on the earth's spherical surface according to the set vertical track resolution and the field of view of all cameras in the vertical track direction, and the set vertical track resolution, and obtain M rows and N columns of grid corner point data. Calculate the sub-satellite point position at the sampling time according to the position and attitude data, calculate the coordinates of N fusion points on the earth's spherical surface according to the set vertical track resolution and the field of view of cameras 1 to n in the vertical track direction, and the set vertical track resolution setting, and the fusion point spacing is equal or equal to the field of view formed by the satellite. Send the M rows and N columns of fusion point data to the projection plane step.

[0054] Projection plane calculation steps: cache M rows of fusion point data, calculate the normal vector of the projection plane; the projection plane passes through the center of the sphere and is perpendicular to the line connecting the center of the sphere and the center position of the fusion point area in the spherical surface.

[0055] The time resampling step of the camera pixel scattered point position and attitude data: the position and attitude data of the camera data i pixel scattered points are calculated based on the adapted position and attitude data and all camera data before and after the imaging sampling time of the camera data i. The position and attitude data of the camera data i (i from 1 to n) pixel scattered points are calculated based on the position and attitude data before and after the imaging sampling time of the camera data i (i from 1 to n). The calculated position and attitude data together with the pixel scattered point grayscale value in the camera data and the imaging sampling time of the camera are sent to the camera data n pixel scattered point positioning and calibration step.

[0056] Camera pixel scatter point positioning and calibration step: Calculate the coordinates of the pixel in the ground-fixed coordinate system based on the position and attitude data of the pixel scatter points of the camera data i. Calculate the coordinates of the pixel in the ground-fixed coordinate system based on the position of the pixel scatter points in the imaging plane, the direction of the camera optical axis relative to the satellite body, the attitude of the satellite body, the position of the satellite, and the imaging sampling time. Perform calibration calculation based on the coordinates in the ground-fixed coordinate system and the set parameters

[0057] Step S3: According to the M rows and N columns of fused point data, the normal vector of the projection plane and the coordinates of the pixel in the ground-fixed coordinate system, the projected pixel scattered points are clustered to the grid corner points, i.e., the fused points, and the fused point data are stored; in step S3, a two-dimensional array with a length of M*N is used to store the fused point data, and the array elements record the pixel scattered point information or the pixel scattered point information and the fused point position information. The step S3 includes:

[0058] 1. Project the fusion points to the projection plane to form a grid in the projection plane. These fusion points are called projection fusion points, these grids are called projection grids, and the area where they are located is the projection grid area; project the pixel scattered points to the projection plane, and call these points as projection pixel scattered points; establish a two-dimensional coordinate system in the projection plane, with the Y axis along the track and the horizontal axis X perpendicular to the track. Project the fusion points of the M rows and N columns of the grid to the projection plane. The M rows of fusion points form (M-1)*(N-1) target grids, and record the coordinates of the fusion points in the above two-dimensional coordinate system. Project the pixel scattered points to the projection plane and record the coordinates of the pixel scattered points in the above two-dimensional coordinate system. The calculation steps of the projection step in the two-dimensional coordinate system are as follows:

[0059] Step 1. In the earth-fixed coordinate system, calculate the coordinates of the unit vectors of the X-axis and Y-axis of the two-dimensional coordinate system, respectively, and record them as (x xn ,y xn ,z xn ) and (x yn ,y yn ,z yn ).

[0060] Step 2. Note that the coordinates of the point to be projected in the ground-fixed coordinate system are (x r0 ,y r0 ,z r0 ), the projection point coordinates on the projection plane (x p ,y p ) is calculated using the following formula:

[0061] x p =x r0 x xn +y r0 y xn +z r0 z xn

[0062] y p =x r0 x yn +y r0 y yn +z r0 z yn

[0063] Establish an affine auxiliary grid, the row length of the auxiliary grid is equal to the row length of the projection grid at the corresponding position near the center line of the projection grid area row direction; the column length of the auxiliary grid is equal to the column length of the projection grid near the center position of the projection grid area or equal to the column length of the projection grid at the corresponding position near the center line of the projection grid area column direction; establish an auxiliary grid on the projection plane, the auxiliary grid consists of M rows and N columns of fusion points, one-to-one correspondence with the target grid, and the center coincides. Let the generation time of the fusion point of the first row be t1, the generation time of the fusion point of the Mth row be t2, and define t m =(t1+t2) / 2. The distance between adjacent fusion points in the Y-axis direction of the auxiliary grid is equal to t m -Δt / 2 time and t m +Δt / 2, the distance between the subsatellite point and the projection point on the projection plane is denoted as d. Generate t according to the method in the fusion point generation step. m The point data is fused at each moment and projected onto the auxiliary plane to form a point column L consisting of N points. This point column corresponds one-to-one to the fused point column of each row of the auxiliary grid, and the corresponding points have the same X coordinates.

[0064] Cluster the projected pixel scatter points to the auxiliary grid; determine which row of the grid the pixel belongs to by calculating the ratio of the y coordinate in the pixel's projected coordinates to d, and determine which column the pixel is in by comparing the x coordinate value in the point column L.

[0065] Correct the projected pixel scatter points to the projection grid, and cluster the corresponding pixel scatter points to the corresponding spherical grid. Use the point sequence L as the boundary to divide the projection plane into two parts, the upper and lower parts. The specific steps are as follows:

[0066] 1. For the pixels in the upper (lower) half, find the corresponding target grid based on the auxiliary grid where it is located, and take out the coordinates of the fusion point above (below) the grid;

[0067] 2. Determine whether the pixel point is above (below) the straight line connecting the above fusion points. If yes, execute step 3; if not, the correction ends;

[0068] 3. Correct the pixels to the target grid directly above (below) and return to step 1 to execute.

[0069] The step of determining whether the pixel point is above (below) the straight line connecting the above fusion points is as follows: Calculate A = (x0-x1)(y1-x p )-(y0-y1)(x1-x p ), where (x0, y0) and (x1, y1) are the pixel scatter points (x p ,y p ) The left and right fusion points. Judge A. If A is negative, (x p ,y p) is below the line connecting (x0,y0) and (x1,y1); if positive, it is above.

[0070] Step S4: Fuse the scattered point information and the fusion point position information in the fusion point data into the required information. The calculation logic of a single fusion point uses a pipeline method to increase the processing throughput; multiple fusion points use parallel hardware circuits to increase the processing speed.

[0071] A data-driven multi-task communication scheduling architecture is adopted, and the data communication interface between tasks adopts the form of queues. The synchronization adaptation step S1 is implemented in the form of a task, the fusion point generation step and the projection plane calculation step are combined into one task, the camera data i pixel scattered point position posture data time area resampling step and the camera data i scattered point positioning calibration step are combined into one module implementation, and the grid clustering step S3 and the data fusion step S4 are combined into one task implementation. The fusion point generation step and the projection plane calculation step are respectively implemented as two independent tasks, the camera data i pixel scattered point position posture data time area resampling step and the camera data i scattered point positioning calibration step are combined into one step implementation (where i ranges from 1 to n) are respectively implemented as two independent tasks, and the grid clustering step S3 and the data fusion step S4 are respectively implemented as two independent tasks. The data-driven multi-task communication scheduling architecture adopts a multi-core CPU deployment method, and the task input and output data are one or more data objects, and the buffer structure of the data object is a queue.

[0072] Specifically, the multi-task scheduling method includes:

[0073] Step 1: Initialization.

[0074] Step 2: Set the initialization completion flag.

[0075] Step 3: Detect the initialization completion flags of other cores.

[0076] Step 4: Determine whether all other core initializations are completed. If completed, go to step 5. If not, go to step 3.

[0077] Step 5: Query the next task input queue buffer according to the set rules. The querying the next task input queue according to the rules in step 5 includes the following steps: Step 5.1: Increment by 1 with N as modulo n, where N is the number of tasks deployed in the core. Step 5.2: Increment the query counter variable to which task n belongs by 1. Step 5.3: Determine whether the task query counter is equal to the set value. If it is equal to the set value, go to step 4. If it is not equal to the set value, go to step 1. Step 5.4: Clear the counter variable, read the task n queue information, and the querying the next task input queue according to the rules is completed.

[0078] Step 6: Determine whether the conditions for executing the task are met. If so, go to step 7. If not, go to step 5.

[0079] Step 7: Execute the task and go to step 5 after completion.

[0080] A method for configuring a large-capacity off-chip memory space for a multi-core CPU includes dividing the large-capacity off-chip memory space into a multi-core shared memory space and a multi-core private memory space. A multi-core multi-task deployment method includes: the average execution time of all tasks in each core is less than the average generation interval of the core source queue data. The source queue data comes from the CPU external interface or other core task outputs; the queue buffer capacity should be set to be larger than the capacity requirement in the case of data bursts; the writing and reading tasks are respectively deployed in queues of different cores and deployed on shared memory. Among them, the large-capacity off-chip memory includes DDR and HBM, the queue data is implemented using an array, the queue read and write pointers are respectively implemented using array index variables, the queue elements in the queue are aligned with the cache block boundaries, the private memory cache is turned on, the shared memory cache is turned off, the queue pointer is deployed on the CPU chip shared memory, and the task reads data from other cores and copies it to the private memory of this core for processing.

[0081] The shared memory cache used to deploy queue data is opened, and the task does not copy data from other cores, but is processed directly on the shared memory. Before processing, the relevant parts of the local cache are invalidated, and the write-back operation is performed after the processing is completed. The multi-core multi-task deployment method includes a single image loading method, which loads the program image into the shared memory, and all cores read the program from the same memory. The program in the program image contains a judgment statement for comparing the core ID number, and jumps to execute the program that matches the ID number. The single image loading method loads the program image into the private memory of each core respectively, and all cores read the program from each sub-private memory.

[0082] Embodiment 2

[0083] The present invention also provides a system based on heterogeneous computing time domain resampling and spatial domain data fusion. Those skilled in the art can implement the system based on heterogeneous computing time domain resampling and spatial domain data fusion by executing the step flow of the method based on heterogeneous computing time domain resampling and spatial domain data fusion. That is, the method based on heterogeneous computing time domain resampling and spatial domain data fusion can be understood as a preferred implementation of the system based on heterogeneous computing time domain resampling and spatial domain data fusion.

[0084] According to the present invention, a system based on heterogeneous computing time domain resampling space domain data fusion is provided, comprising:

[0085] Module M1: Match the acquired camera data and the imaging sampling time data attached to the position and posture data, and remove the mismatched data;

[0086] Module M2: Generate M rows and N columns of fusion point data according to the adapted position and attitude data and camera data, calculate the projection plane normal vector and the coordinates of the camera pixel in the ground-fixed coordinate system; the module M2 includes a fusion point generation module, a projection plane calculation module, a camera pixel scattered point position and attitude data time resampling module and a camera pixel scattered point positioning and calibration module; the fusion point generation module: calculates the sub-satellite point position at the sampling time according to the adapted position and attitude data, calculates the coordinates of N corner points on the earth's spherical surface according to the set vertical track direction resolution and the vertical track direction field angle of all cameras, the set vertical track direction resolution, and obtains M rows and N columns of grid corner point data; the projection plane calculation module: caches M rows of fusion point data and calculates the projection plane normal vector; the camera pixel scattered point position and attitude data time resampling module: calculates the position and attitude data of the camera data i pixel scattered points according to the adapted position and attitude data and all camera data, before and after the imaging sampling time of the camera data i; the camera pixel scattered point positioning and calibration module: calculates the coordinates of the pixel in the ground-fixed coordinate system according to the position and attitude data of the camera data i pixel scattered points.

[0087] Module M3: According to the M rows and N columns of fusion point data, the projection plane normal vector and the coordinates of the pixel in the ground-fixed coordinate system, the projected pixel scattered points are clustered to the grid corner points, i.e., the fusion points, and the fusion point data are stored; in module M3, a two-dimensional array with a length of M*N is used to store the fusion point data, and the array elements record the pixel scatter point information or the pixel scatter point information and the fusion point position information. The module M3 includes: projecting the fusion points onto the projection plane to form a grid in the projection plane, calling these fusion points as projection fusion points, calling these grids as projection grids, and the area where they are located is the projection grid area; projecting the pixel scattered points onto the projection plane, calling these points as projection pixel scattered points; establishing an affine auxiliary grid, the row length of the auxiliary grid is equal to the row length of the projection grid at the corresponding position near the center line of the row direction of the projection grid area; the column length of the auxiliary grid is equal to the column length of the projection grid near the center position of the projection grid area or equal to the column length of the projection grid at the corresponding position near the center line of the column direction of the projection grid area; clustering the projection pixel scattered points into the auxiliary grid; correcting the projection pixel scattered points to the projection grid, and clustering the corresponding pixel scattered points into the corresponding spherical grid.

[0088] Module M4: Fusing the scattered point information and the fusion point position information in the fusion point data into the required information.

[0089] The synchronization adaptation module M1 is implemented in the form of a task, the fusion point generation module and the projection plane calculation module are merged into one task, the camera data i pixel scattered point position posture data time area resampling module and the camera data i scattered point positioning and calibration module are merged into one module, and the grid clustering module M3 and the data fusion module M4 are merged into one task; a data-driven multi-task communication scheduling architecture is adopted, and the data communication interface between tasks adopts a queue form. The calculation logic of a single fusion point uses a pipeline method to increase the processing throughput; multiple fusion points use parallel hardware circuits to increase the processing speed.

[0090] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0091] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for data fusion in time domain resampling and spatial domain based on heterogeneous computing, characterized in that: include: Step S1: matching the acquired camera data and the imaging sampling time data attached to the position and posture data, and removing the unmatched data; Step S2: Generate M rows and N columns of fusion point data according to the adapted position and posture data and camera data, and calculate the normal vector of the projection plane and the coordinates of the camera pixel in the ground-fixed coordinate system; Step S3: clustering the projected pixel scattered points to the grid corner points, i.e., the fusion points, according to the M rows and N columns of fusion point data, the projection plane normal vector and the coordinates of the pixels in the ground-fixed coordinate system, and storing the fusion point data; Step S4: Fusing the scattered point information and the fusion point position information in the fusion point data into required information.

2. The method for heterogeneous computing-based time domain resampling and spatial domain data fusion according to claim 1 is characterized in that: In step S3, a two-dimensional array with a length of M*N is used to store the fusion point data, and the array elements record pixel scatter point information or pixel scatter point information and fusion point position information.

3. The method for data fusion based on heterogeneous computing time domain resampling and spatial domain according to claim 1 is characterized in that: The step S2 includes a fusion point generation step, a projection plane calculation step, a camera pixel scattered point position posture data time resampling step and a camera pixel scattered point positioning and calibration step; Fusion point generation step: Calculate the sub-satellite point position at the sampling time according to the adapted position and attitude data, calculate the coordinates of N corner points on the earth's spherical surface according to the set vertical track resolution and the vertical track field angles of all cameras, and the set vertical track resolution, and obtain M rows and N columns of grid corner point data; Projection plane calculation steps: cache M rows of fusion point data and calculate the normal vector of the projection plane; The time resampling step of the camera pixel scattered point position and attitude data: according to the adapted position and attitude data and all camera data, the position and attitude data of the camera data i pixel scattered points are calculated before and after the imaging sampling time of the camera data i; Camera pixel scatter point positioning and calibration step: Calculate the coordinates of the pixel in the ground-fixed coordinate system based on the position and posture data of the pixel scatter point i of the camera data.

4. The method for data fusion based on heterogeneous computing time domain resampling and spatial domain according to claim 1 is characterized in that: The step S3 comprises: The fusion points are projected onto the projection plane to form a grid in the projection plane. These fusion points are called projection fusion points, these grids are called projection grids, and the area where they are located is the projection grid area; the pixel scattered points are projected onto the projection plane, and these points are called projection pixel scattered points; Establish an affine auxiliary grid, the row length of the auxiliary grid is equal to the row length of the projection grid at the corresponding position near the center line of the projection grid area; the column length of the auxiliary grid is equal to the column length of the projection grid near the center position of the projection grid area or equal to the column length of the projection grid at the corresponding position near the center line of the projection grid area; Cluster the projected pixel scatter points onto an auxiliary grid; The projected pixel scatter points are corrected to the projected grid, and the corresponding pixel scatter points are clustered to the corresponding spherical grid.

5. The method for heterogeneous computing-based time domain resampling and spatial domain data fusion according to claim 3 is characterized in that: The synchronous adaptation step S1 is implemented in the form of a task, the fusion point generation step and the projection plane calculation step are combined into one task, the camera data i pixel scattered point position posture data time area resampling step and the camera data i scattered point positioning and calibration step are combined into one module, and the grid clustering step S3 and the data fusion step S4 are combined into one task; It adopts a data-driven multi-task communication scheduling architecture, and the data communication interface between tasks adopts the queue form; The computing logic of a single fusion point uses pipeline method to increase processing throughput; Multiple fusion points use parallel hardware circuits to increase processing speed.

6. A system based on heterogeneous computing time domain resampling space domain data fusion, characterized in that: include: Module M1: Match the acquired camera data and the imaging sampling time data attached to the position and posture data, and remove the mismatched data; Module M2: Generate M rows and N columns of fusion point data based on the adapted position and posture data and camera data, and calculate the normal vector of the projection plane and the coordinates of the camera pixel in the ground-fixed coordinate system; Module M3: clustering the projected pixel scattered points to the grid corner points, i.e., the fusion points, according to the M rows and N columns of fusion point data, the projection plane normal vector and the coordinates of the pixels in the ground-fixed coordinate system, and storing the fusion point data; Module M4: Fusing the scattered point information and the fusion point position information in the fusion point data into the required information.

7. The system based on heterogeneous computing time domain resampling space domain data fusion according to claim 6 is characterized in that: In module M3, a two-dimensional array with a length of M*N is used to store fusion point data, and the array elements record pixel scatter information or pixel scatter information and fusion point position information.

8. The system based on heterogeneous computing time domain resampling space domain data fusion according to claim 6 is characterized in that: The module M2 includes a fusion point generation module, a projection plane calculation module, a camera pixel scattered point position posture data time resampling module and a camera pixel scattered point positioning and calibration module; Fusion point generation module: Calculate the sub-satellite point position at the sampling time according to the adapted position and attitude data, calculate the coordinates of N corner points on the earth's spherical surface according to the set vertical track resolution and the vertical track field angles of all cameras, and the set vertical track resolution, and obtain M rows and N columns of grid corner point data; Projection plane calculation module: cache M rows of fusion point data and calculate the normal vector of the projection plane; Camera pixel scattered point position and attitude data time resampling module: calculates the position and attitude data of the scattered points of camera data i pixel based on the adapted position and attitude data and all camera data before and after the imaging sampling time of camera data i; Camera pixel scatter point positioning and calibration module: calculates the coordinates of the pixel in the ground-fixed coordinate system based on the position and posture data of the pixel scatter point i of the camera data.

9. The system based on heterogeneous computing time domain resampling space domain data fusion according to claim 6 is characterized in that: The module M3 comprises: The fusion points are projected onto the projection plane to form a grid in the projection plane. These fusion points are called projection fusion points, these grids are called projection grids, and the area where they are located is the projection grid area; the pixel scattered points are projected onto the projection plane, and these points are called projection pixel scattered points; Establish an affine auxiliary grid, the row length of the auxiliary grid is equal to the row length of the projection grid at the corresponding position near the center line of the projection grid area; the column length of the auxiliary grid is equal to the column length of the projection grid near the center position of the projection grid area or equal to the column length of the projection grid at the corresponding position near the center line of the projection grid area; Cluster the projected pixel scatter points onto an auxiliary grid; The projected pixel scatter points are corrected to the projected grid, and the corresponding pixel scatter points are clustered to the corresponding spherical grid.

10. The system based on heterogeneous computing time domain resampling space domain data fusion according to claim 8, characterized in that: The synchronous adaptation module M1 is implemented in the form of a task, the fusion point generation module and the projection plane calculation module are merged into one task, the camera data i pixel scattered point position posture data time area resampling module and the camera data i scattered point positioning and calibration module are merged into one module, and the grid clustering module M3 and the data fusion module M4 are merged into one task; It adopts a data-driven multi-task communication scheduling architecture, and the data communication interface between tasks adopts the queue form; The computing logic of a single fusion point uses pipeline method to increase processing throughput; Multiple fusion points use parallel hardware circuits to increase processing speed.

Images