Multi-stage parallel processing system for medium and high orbit SAR (Synthetic Aperture Radar) oversized data volume

Abstract

The invention relates to a multi-stage parallel processing system for ultra-large data volume of medium and high orbit synthetic aperture radar (SAR), and the system comprises a data storage device which is used for storing a plurality of pieces of full-scene observation data returned by the medium and high orbit SAR within a preset period; the distributed computing device comprises a data exchange node and a plurality of data processing nodes connected to the data exchange node; the data exchange node is connected with the data storage equipment and is used for acquiring multiple pieces of full-scene observation data, decomposing each piece of full-scene observation data into multiple task blocks and distributing the multiple task blocks to the multiple data processing nodes; and the plurality of data processing nodes are used for carrying out parallel processing on the received task blocks by utilizing a multi-thread model and feeding back corresponding processing results to the data exchange node. The device can effectively improve the processing throughput of mass data through a data-task-algorithm multi-stage joint parallel mode, and meets the real-time or near-real-time imaging requirement.

Description

Technical Field

[0001] The present invention belongs to the technical field of radar signal processing, and in particular relates to a multi-level parallel processing system for very large amounts of data for medium and high orbit SAR. Background Art

[0002] The medium- and high-orbit synthetic aperture radar (SAR) is a high-resolution radar system installed on a satellite or other flying platform, capable of all-weather, all-day observations of the earth in a high orbit. Specifically, by moving the aircraft or changing the direction of the antenna beam, the relative movement between the radar antenna and the ground target is achieved, so that the same area can be observed at multiple time points, and multiple observation signals are synthesized to form an image of geographic coordinates and radiation intensity with rich time series information. In the medium- and high-orbit SAR imaging technology, the captured single-scene data reaches 500G to 800G, and up to 16 observations can be made in a single day / single loop, and the echo data flow is huge.

[0003] However, the existing serial processing method has the problem of low data processing efficiency when processing massive medium and high orbit SAR data, and it is difficult to meet the real-time or near real-time imaging requirements. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a multi-level parallel processing system for large amounts of medium and high orbit SAR data. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] The present invention provides a multi-level parallel processing system for medium- and high-orbit SAR with very large data volume, comprising: a data storage device, used to store multiple full-scene observation data returned by a medium- and high-orbit synthetic aperture radar within a preset period, and synchronously send the multiple full-scene observation data to a distributed computing device, and receive multiple image processing results sent by the distributed computing device; the distributed computing device comprises a data exchange node, and multiple data processing nodes connected to the data exchange node; the data exchange node is connected to the data storage device, used to obtain the multiple full-scene observation data, and decompose each full-scene observation data into multiple task blocks, and synchronously distribute the multiple task blocks to the multiple data processing nodes; the multiple data processing nodes are used to perform parallel processing on the received task blocks using a multi-threaded model, and feed back the corresponding image processing results to the data exchange node.

[0006] In some embodiments, each data processing node corresponds to a node number; the data exchange node is also used to: obtain the total number of the multiple data processing nodes; divide the number of the multiple task blocks by the total number of the multiple data processing nodes to obtain a basic task amount; based on the basic task amount, split the multiple task blocks to obtain multiple basic task blocks; and synchronously distribute the multiple basic task blocks to the multiple data processing nodes in the order of the multiple node numbers.

[0007] In some embodiments, when the total number of the plurality of data processing nodes cannot divide the number of the plurality of task blocks evenly, the plurality of task blocks are subjected to a complement process.

[0008] In some embodiments, the threads in the multi-threaded model include: a read data thread, a processing data thread, and a write data thread that operate in parallel; the trigger time of the read data thread is earlier than the trigger time of the processing data thread and the trigger time of the write data thread, and the trigger time of the processing data thread is earlier than the trigger time of the write data thread.

[0009] In some embodiments, the multithreading model is a producer-consumer model.

[0010] In some embodiments, the data exchange node uses a BP algorithm to decompose each full-scene observation data to obtain the multiple task blocks.

[0011] In some embodiments, each data processing node is provided with a wireless interaction port for establishing a connection with a mobile terminal so that the mobile terminal can obtain the processing result of the data processing node.

[0012] In some embodiments, the data storage device is a disk array.

[0013] In some embodiments, the data exchange node is a fiber optic switch.

[0014] In some embodiments, the plurality of data processing nodes connected to the data exchange node are in a star topology.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] In view of the problem that the existing serial processing method has too low data processing efficiency and is difficult to meet the real-time or near real-time imaging requirements when processing massive medium and high orbit SAR data, the present invention provides a multi-level parallel processing system for medium and high orbit SAR ultra-large data volumes. The system stores multiple full-scene observation data returned by the medium and high orbit SAR within a preset period, and decomposes each full-scene observation data into tasks to obtain multiple task blocks. The system uses multiple distributed data processing nodes that do not interfere with each other to process the multiple task blocks in parallel to obtain image processing results. On the basis of ensuring high-resolution imaging, the throughput of the overall system can be effectively improved, the data processing efficiency can be greatly improved, and the timeliness of data processing can be ensured to meet the real-time or near real-time imaging requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural block diagram of a multi-level parallel processing system for medium and high orbit SAR ultra-large data volume provided by the present invention;

[0018] Figure 2 is an example diagram of task decomposition provided by an embodiment of the present invention;

[0019] Figure 3 This is an example diagram of a multi-threaded model processing multiple task blocks provided by an embodiment of the present invention;

[0020] Figure 4 This is a diagram showing an application example of a multi-level parallel processing system for extremely large amounts of data for medium and high-orbit SAR provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0022] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0023] A multi-level parallel processing system for very large amounts of data for medium and high orbit SAR proposed by the present invention is now described in detail with reference to the accompanying drawings.

[0024] Figure 1This is a block diagram of the multi-level parallel processing system for medium and high orbit SAR with very large data volume provided by the present invention. Figure 1 As shown, the multi-level parallel processing system includes: a data storage device and a distributed computing device; the data storage device is used to store multiple full-scene observation data returned by the medium and high orbit synthetic aperture radar within a preset period, and synchronously send the multiple full-scene observation data to the distributed computing device, and receive multiple image processing results sent by the distributed computing device; the distributed computing device includes a data exchange node, and multiple data processing nodes connected to the data exchange node; the data exchange node is connected to the data storage device, and is used to obtain multiple full-scene observation data, and decompose each full-scene observation data into multiple task blocks, and synchronously distribute the multiple task blocks to multiple data processing nodes; multiple data processing nodes are used to use a multi-threaded model to perform parallel processing on the received task blocks, and feed back the corresponding image processing results to the data exchange node.

[0025] Here, the full-scene observation data is the SAR image data returned by the medium- and high-orbit synthetic aperture radar. The data storage device is a disk array. The disk array combines multiple hard disks together to increase storage capacity and read and write speeds, and has data redundancy and fault recovery capabilities. When a hard disk fails, the disk array can restore the data on the failed hard disk through redundant data, avoid data loss, and effectively improve the reliability of the system and data security. In addition, the disk array also supports parallel read and write operations, which can speed up data access and meet the storage needs of massive data.

[0026] Here, the data exchange node is a fiber optic switch, and the multiple data processing nodes connected to the data exchange node are in a star topology. The data exchange node and the multiple data processing nodes, as well as the data exchange node and the data storage device are connected via communication optical cables. The multiple data processing nodes do not affect each other, and can synchronously process the received task blocks, which can improve the parallelism of data processing. It should be understood that the number of data processing nodes can be set according to actual needs. This method of synchronously processing data through distributed nodes can effectively overcome the problem of low efficiency of data distribution and reception due to bandwidth limitations of a single data processing node.

[0027] It should be noted that there may be other node connection modes between multiple data processing nodes, such as a tree structure.

[0028] Figure 2 is an example diagram of task decomposition provided by an embodiment of the present invention. Figure 2As shown, the data exchange node uses the BP algorithm to decompose each full-scene observation data to obtain multiple task blocks. Specifically, the BP algorithm uses a matched filtering method to perform pulse compression on a full-scene observation data in the distance direction to obtain the full-scene data after pulse compression, and then meshes the full-scene data after pulse compression, decomposes the compressed full-scene data in the azimuth direction with a preset step size, establishes a node screening algorithm in the distance direction through the array element normal, classifies and screens the generated grids, and constructs a neural network structure, resets the loss function and activation function to optimize the grid generation efficiency and quality. Through the BP algorithm, a set of full-scene observation data can be decomposed into m in the azimuth direction and n in the distance direction. Both m and n are positive integers, and the values ​​of the two can be the same or different. It should be noted that the content in each task block is not a complete full-scene image, but a part of the full-scene image. When processing a single full-scene task, compared with the method of processing a complete full-scene image, the method of processing only a part of the full-scene image can improve the throughput of the overall system on the basis of ensuring high-resolution imaging, while ensuring the timeliness of the processing process and speeding up the processing efficiency.

[0029] Here, each data processing node corresponds to a node number; the data exchange node is also used to: obtain the total number of multiple data processing nodes; use the total number of multiple data processing nodes to divide the number of multiple task blocks to obtain a basic task volume; based on the basic task volume, split the multiple task blocks to obtain multiple basic task blocks; distribute the multiple basic task blocks to the multiple data processing nodes in the order of the multiple node numbers.

[0030] For example, the total number of data processing nodes is 10, and the number of task blocks corresponding to multiple full-scene observation data is 1200. Then the basic task volume that each data processing node needs to process is 120. In the order of node numbers 1 to 10, the first 1 to 120 task blocks are sent to data processing node 1, and 121 to 240 task blocks are sent to data processing node 2, and so on.

[0031] Here, when the total number of multiple data processing nodes cannot divide the number of multiple task blocks, the multiple task blocks are supplemented so that the number of task blocks after the supplementation can be divided by the total number of data processing nodes. For example, the total number of data processing nodes is 10, the number of task blocks is 1197, and the number of task blocks is increased to 1200 by adding empty data, and then divided; or, the number of task blocks is 1205, and the number of task blocks is increased to 1210 by adding empty data, so that it can be divided by the total number of data processing nodes.

[0032] In a possible implementation, after each data processing node receives multiple task blocks, in order to speed up data processing efficiency, a multi-thread model is used to process data on the multiple task blocks. The multi-thread model is a producer-consumer model. The threads in the multi-thread model include: a parallel computing data reading thread, a data processing thread, and a data writing thread; the trigger time of the data reading thread is earlier than the trigger time of the data processing thread and the trigger time of the data writing thread, and the trigger time of the data processing thread is earlier than the trigger time of the data writing thread.

[0033] Figure 3 1 is an example diagram of a multi-threaded model processing multiple task blocks provided by an embodiment of the present invention. Figure 3 As shown, a data processing node receives R task blocks, where R is a positive integer, triggers the data reading thread, reads the first task block, processes the first task block while reading the second task block, and writes the image processing result corresponding to the first task block to the data storage device when starting to process the second task block. Compared with the method of uniformly reading several task blocks, then uniformly processing and writing, this pipeline data processing process, that is, the method of parallel processing of three threads, can significantly shorten the imaging time from (R*t_r+t_d+R*t_wn) to (t_r+t_d+t_wn), where t_r refers to the time to read a task block, t_d refers to the time to process R task blocks, and t_wn refers to the time to write an image processing result.

[0034] Here, the data exchange node uses the BP algorithm to process the task block to obtain the SAR image corresponding to each task block. It should be understood that the process of processing the task block by the BP algorithm is an existing data processing process and will not be repeated here.

[0035] In a possible implementation, each data processing node is provided with a wireless interaction port for establishing a connection with a mobile terminal so that the mobile terminal obtains the image processing result of the data processing node. Exemplarily, the mobile terminal accesses the data processing node through a router to obtain the image processing result on the data processing node.

[0036] In one possible implementation, to speed up data processing, each data node uses a CPU-GPU heterogeneous computing architecture. By distributing computing tasks between the CPU and GPU, the CPU implements multi-threading technology to allow multiple computing threads to run simultaneously, and the GPU executes CUDA Streams to achieve parallel processing of multiple tasks. Through the collaborative work of the CPU and GPU, efficient use of heterogeneous computing resources is achieved, greatly improving the speed of data processing.

[0037] Figure 41 is an application example diagram of a multi-level parallel processing system for medium and high orbit SAR with very large data volume provided by an embodiment of the present invention. Figure 4 As shown in the figure, in actual operation, the medium-high orbit SAR performs multiple high-frequency revisit observations of the same area on a single orbit, and collects full-scene observation data of different time periods, for example, full-scene image data at time t1 and full-scene image data at time t2; the full-scene observation data of different time periods are stored in real time in the disk array, and the disk array synchronously obtains the image processing results of multiple data processing nodes, and merges the image processing results to achieve data parallelism; the optical fiber switch (not shown in the figure) obtains the full-scene observation data of different time periods, and grids each full-scene observation data to obtain multiple task blocks (i.e., subtasks in the figure), and according to the total number of data processing nodes, the multiple task blocks are evenly distributed to each data processing node to achieve task parallelism; each data processing node relies on the CPU+GPU architecture and supports the multi-threaded model to process the received task blocks in parallel, and obtain the image data corresponding to each task block to achieve algorithm parallelism. Through the data parallelism-task parallelism-algorithm parallelism method, the system can effectively improve the processing throughput of massive data and meet the real-time or near real-time imaging requirements.

[0038] In view of the problem that the existing serial processing method has too low data processing efficiency and is difficult to meet the real-time or near real-time imaging requirements when processing massive medium and high orbit SAR data, the present invention provides a multi-level parallel processing system for medium and high orbit SAR ultra-large data volumes, which stores multiple full-scene observation data returned by the medium and high orbit SAR within a preset period, and decomposes each full-scene observation data into tasks to obtain multiple task blocks. By using multiple distributed data processing nodes that do not interfere with each other to process the multiple task blocks in parallel, an image processing result is obtained. On the basis of ensuring high-resolution imaging, the throughput of the overall system can be effectively improved, the data processing efficiency can be greatly improved, and the timeliness of data processing can be ensured to meet the real-time or near real-time imaging requirements.

[0039] In order to verify the multi-level parallel processing system for medium and high orbit SAR with very large data volume provided by the present invention, Table 1 is a simulation comparison between the existing distributed system provided by the embodiment of the present invention and the multi-level parallel processing system provided by the present invention, wherein the number of data processing nodes in each system is 2, and the input data used in the simulation is 30K*30K SAR simulation echo data, and the simulation process includes four steps: data distribution, pulse compression, data imaging, and merging processing results. As shown in Table 1, compared with the existing distributed system, the multi-level parallel processing system provided by the present invention achieves accelerated processing in these four steps by using a multi-level parallel approach, which increases the overall processing speed by 78%.

[0040] Table 1

[0041]

[0042] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A multi-level parallel processing system for very large amounts of data for medium and high orbit SAR, characterized in that: include: A data storage device, used to store multiple full-scene observation data returned by the medium- and high-orbit synthetic aperture radar within a preset period, and synchronously send the multiple full-scene observation data to the distributed computing device, and receive multiple image processing results sent by the distributed computing device; The distributed computing device includes a data exchange node and multiple data processing nodes connected to the data exchange node; the data exchange node is connected to the data storage device, and is used to obtain the multiple full-scene observation data, and decompose each full-scene observation data into multiple task blocks, and synchronously distribute the multiple task blocks to the multiple data processing nodes; the multiple data processing nodes are used to use a multi-threaded model to parallel process the received task blocks, and feed back the corresponding image processing results to the data exchange node.

2. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 1 is characterized in that: Each data processing node corresponds to a node number; the data exchange node is also used to: Obtaining a total number of the plurality of data processing nodes; Dividing the number of the plurality of task blocks by the total number of the plurality of data processing nodes to obtain a basic task amount; Based on the basic task amount, split the multiple task blocks to obtain multiple basic task blocks; The multiple basic task blocks are synchronously distributed to the multiple data processing nodes in the order of the multiple node numbers.

3. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 2 is characterized in that: When the total number of the plurality of data processing nodes cannot divide the number of the plurality of task blocks evenly, a padding process is performed on the plurality of task blocks.

4. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 1 is characterized in that: The threads in the multi-threaded model include: a read data thread, a processing data thread and a write data thread for parallel computing; the trigger time of the read data thread is earlier than the trigger time of the processing data thread and the trigger time of the write data thread, and the trigger time of the processing data thread is earlier than the trigger time of the write data thread.

5. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 1 is characterized in that: The multithreading model is a producer-consumer model.

6. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 1 is characterized in that: The data exchange node uses the BP algorithm to decompose each full-scene observation data to obtain the multiple task blocks.

7. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 1 is characterized in that: Each data processing node is provided with a wireless interaction port for establishing a connection with a mobile terminal so that the mobile terminal can obtain the processing result of the data processing node.

8. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 1 is characterized in that: The data storage device is a disk array.

9. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 1, characterized in that: The data exchange node is a fiber optic switch.

10. The multi-level parallel processing system for medium and high orbit SAR very large data volume according to claim 1, characterized in that: The multiple data processing nodes connected to the data exchange node are in a star topology.

Images