Multi-stage parallel processing system for large data volume of middle-high orbit SAR
By using a multi-level parallel processing system for medium- and high-orbit SAR, data decomposition and distributed computing are employed to solve the problem of low data processing efficiency for medium- and high-orbit SAR, thereby achieving efficient data processing and real-time imaging.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2024-12-30
- Publication Date
- 2026-06-09
AI Technical Summary
Existing serial processing methods have low data processing efficiency when processing medium and high orbit SAR data, making it difficult to meet the requirements of real-time or near real-time imaging.
A multi-level parallel processing system for medium- and high-orbit SAR is adopted, including data storage devices, distributed computing devices and multi-threaded models. The observation data of the whole scene is decomposed into task blocks through data exchange nodes and distributed to multiple data processing nodes in parallel for processing. The CPU-GPU heterogeneous computing architecture is used to accelerate data processing.
While ensuring high-resolution imaging, it significantly improves data processing efficiency, meets real-time or near-real-time imaging requirements, and enhances the system's throughput and timeliness.
Smart Images
Figure CN119828140B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of radar signal processing technology, specifically relating to a multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR. Background Technology
[0002] Medium- and high-orbit synthetic aperture radar (SAR) is a high-resolution radar system mounted on satellites or other flight platforms, capable of all-weather, all-time observation of the Earth from high orbit. Specifically, by moving the aircraft or changing the direction of the antenna beam, the relative motion between the radar antenna and the ground target is achieved, allowing for observations of the same area at multiple time points. These multiple observation signals are then synthesized to form an image with rich time-series information, including geographic coordinates and radiation intensity. In medium- and high-orbit SAR imaging technology, single-scene data can reach 500G to 800G, and up to 16 observations can be performed per day / per orbit, resulting in a massive echo data throughput.
[0003] However, existing serial processing methods suffer from low data processing efficiency when dealing with massive amounts of medium and high orbit SAR data, making it difficult to meet the requirements of real-time or near real-time imaging. Summary of the Invention
[0004] To address the aforementioned problems in existing technologies, this invention provides a multi-level parallel processing system for ultra-large data volumes in medium- and high-orbit SAR. The technical problem to be solved by this invention is achieved through the following technical solution:
[0005] This invention provides a multi-level parallel processing system for massive data volumes of medium- and high-orbit SAR, comprising: a data storage device for storing multiple full-scene observation data returned by medium- and high-orbit synthetic aperture radar within a preset period, and synchronously sending the multiple full-scene observation data to a distributed computing device, and receiving multiple image processing results sent by the distributed computing device; the distributed computing device includes a data exchange node and multiple data processing nodes connected to the data exchange node; the data exchange node is connected to the data storage device and is used to acquire the multiple full-scene observation data, decompose each full-scene observation data into multiple task blocks, and synchronously distribute the multiple task blocks to the multiple data processing nodes; the multiple data processing nodes are used to perform parallel processing on the received task blocks using a multi-threaded model and feed back the corresponding image processing results to the data exchange node.
[0006] In some embodiments, each data processing node corresponds to a node number; the data exchange node is further configured to: obtain the total number of the plurality of data processing nodes; divide the total number of the plurality of data processing nodes by the number of the plurality of task blocks to obtain a basic task quantity; based on the basic task quantity, split the plurality of task blocks to obtain a plurality of basic task blocks; and synchronously distribute the plurality of basic task blocks to the plurality of data processing nodes in the order of the plurality of node numbers.
[0007] In some embodiments, when the total number of the plurality of data processing nodes is not divisible by the number of the plurality of task blocks, the plurality of task blocks are padded.
[0008] In some embodiments, the threads in the multithreaded model include: a data reading thread, a data processing thread, and a data writing thread for parallel computation; the triggering time of the data reading thread is earlier than the triggering time of the data processing thread and the triggering time of the data writing thread, and the triggering time of the data processing thread is earlier than the triggering time of the data writing thread.
[0009] In some embodiments, the multithreaded model is a producer-consumer model.
[0010] In some embodiments, the data exchange node uses the BP algorithm to decompose each full-scene observation data to obtain the multiple task blocks.
[0011] In some embodiments, each data processing node is provided with a wireless interaction port for establishing a connection with a mobile terminal so that the mobile terminal can obtain the processing results of the data processing node.
[0012] In some embodiments, the data storage device is a disk array.
[0013] In some embodiments, the data exchange node is a fiber optic switch.
[0014] In some embodiments, the plurality of data processing nodes connected to the data exchange node are arranged in a star topology.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0016] To address the problem that existing serial processing methods suffer from low data processing efficiency when handling massive amounts of medium- and high-orbit SAR data, making it difficult to meet real-time or near-real-time imaging requirements, this invention provides a multi-level parallel processing system for ultra-large volumes of medium- and high-orbit SAR data. This system stores multiple full-scene observation data returned by medium- and high-orbit SAR within a preset period, decomposes each full-scene observation data into multiple task blocks, and processes these task blocks in parallel using multiple non-interfering distributed data processing nodes to obtain image processing results. This effectively improves the overall system throughput while ensuring high-resolution imaging, greatly enhances data processing efficiency, and ensures the timeliness of data processing to meet real-time or near-real-time imaging requirements. Attached Figure Description
[0017] Figure 1 This is a structural block diagram of the multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR provided by the present invention;
[0018] Figure 2 This is a task decomposition example diagram provided in an embodiment of the present invention;
[0019] Figure 3 This is an example diagram illustrating the multi-threaded model for processing multiple task blocks provided in this embodiment of the invention;
[0020] Figure 4 This is an application example diagram of the multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR provided in the embodiments of the present invention. Detailed Implementation
[0021] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0022] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.
[0023] The multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR proposed in this invention will now be described in detail with reference to the accompanying drawings.
[0024] Figure 1This is a block diagram of the multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR provided by the present invention. Figure 1 As shown, the multi-level parallel processing system includes: a data storage device and a distributed computing device; the data storage device is used to store multiple full-scene observation data returned by medium- and high-orbit synthetic aperture radar within a preset period, and synchronously send the multiple full-scene observation data to the distributed computing device, and receive multiple image processing results sent by the distributed computing device; the distributed computing device includes a data exchange node and multiple data processing nodes connected to the data exchange node; the data exchange node is connected to the data storage device and is used to acquire multiple full-scene observation data, decompose each full-scene observation data into multiple task blocks, and synchronously distribute the multiple task blocks to the multiple data processing nodes; the multiple data processing nodes are used to perform parallel processing on the received task blocks using a multi-threaded model, and feed back the corresponding image processing results to the data exchange node.
[0025] Here, the full-scene observation data consists of SAR image data returned by medium- and high-orbit synthetic aperture radar. The data storage device is a disk array. This disk array, by combining multiple hard drives, increases storage capacity and read / write speed, and provides data redundancy and fault recovery capabilities. When a hard drive fails, the disk array can recover data from the failed drive using redundant data, preventing data loss and effectively improving system reliability and data security. Furthermore, the disk array supports parallel read / write operations, accelerating data access speed and meeting the storage needs of massive amounts of data.
[0026] Here, the data exchange node is a fiber optic switch, and the multiple data processing nodes connected to the data exchange node form a star topology. The data exchange node is connected to the multiple data processing nodes, and also to the data storage device, via communication fiber optic cables. The multiple data processing nodes operate independently and can process received task blocks synchronously, improving the parallelism of data processing. It should be understood that the number of data processing nodes can be set according to actual needs. This method of synchronously processing data through distributed nodes effectively overcomes the problem of low data distribution and reception efficiency caused by the bandwidth limitations of a single data processing node.
[0027] It should be noted that there can be other ways to connect multiple data processing nodes, such as a tree structure.
[0028] Figure 2 This is an example diagram of task decomposition provided in an embodiment of the present invention. For example... Figure 2As shown, the data exchange node uses the BP algorithm to decompose each full-scene observation data into multiple task blocks. Specifically, the BP algorithm employs matched filtering to perform pulse compression on a full-scene observation data in the range direction, resulting in pulse-compressed full-scene data. Then, the pulse-compressed full-scene data is divided into grids. In the azimuth direction, the compressed full-scene data is decomposed with a preset step size. In the range direction, a node filtering algorithm is established using the element normal to classify and filter the generated grids. A neural network structure is constructed, and the loss and activation functions are reset to optimize grid generation efficiency and quality. Through the BP algorithm, a set of full-scene observation data can be decomposed into m elements in the azimuth direction and n elements in the range direction. Both m and n are positive integers, and their values can be the same or different. It should be noted that the content in each task block is not a complete full-scene image, but a part of it. When processing a single full-scene task, compared to processing a complete full-scene image, processing only a part of the full-scene image can improve the overall system throughput while ensuring high-resolution imaging, and simultaneously ensure the timeliness of the processing, thus accelerating processing efficiency.
[0029] Here, each data processing node corresponds to a node number; the data exchange node is also used to: obtain the total number of multiple data processing nodes; divide the total number of multiple data processing nodes by the number of multiple task blocks to obtain the basic task quantity; based on the basic task quantity, split the multiple task blocks to obtain multiple basic task blocks; and distribute the multiple basic task blocks to multiple data processing nodes in the order of the multiple node numbers.
[0030] For example, if the total number of data processing nodes is 10 and the number of task blocks corresponding to multiple full-scene observation data is 1200, then the basic task volume that each data processing node needs to process is 120. According to the node number 1 to 10, the first 1 to 120 task blocks are sent to data processing node 1, the 121 to 240 task blocks are sent to data processing node 2, and so on.
[0031] Here, when the total number of data processing nodes is not divisible by the number of task blocks, the task blocks are padded to ensure that the number of padded task blocks is divisible by the total number of data processing nodes. For example, if the total number of data processing nodes is 10 and the number of task blocks is 1197, the number of task blocks is increased to 1200 by adding empty data, and then the result is divided. Alternatively, if the number of task blocks is 1205, the number of task blocks is increased to 1210 by adding empty data, making it divisible by the total number of data processing nodes.
[0032] In one possible implementation, after each data processing node receives multiple task blocks, a multi-threaded model is used to process the data from these task blocks to accelerate data processing efficiency. This multi-threaded model is a producer-consumer model. The threads in this multi-threaded model include: parallel data reading threads, data processing threads, and data writing threads; the trigger time of the data reading thread is earlier than that of the data processing thread and the data writing thread, and the trigger time of the data processing thread is earlier than that of the data writing thread.
[0033] Figure 3 This is an example diagram illustrating the multi-threaded model for processing multiple task blocks provided in this embodiment of the invention. For example... Figure 3 As shown, a data processing node receives R task blocks (R being a positive integer), triggering a data read thread to read the first task block. While reading the second task block, the first task block is processed simultaneously. When processing the second task block begins, the image processing result corresponding to the first task block is written to the data storage device. Compared to reading several task blocks all at once, then processing and writing them all at once, this pipelined data processing flow—that is, the parallel processing by three threads—significantly reduces the imaging time from (R*t_r + t_d + R*t_wn) to (t_r + t_d + t_wn), where t_r is the time to read one task block, t_d is the time to process R task blocks, and t_wn is the time to write one image processing result.
[0034] Here, the data exchange nodes use the BP algorithm to process the task blocks, obtaining the SAR image corresponding to each task block. It should be understood that the BP algorithm's processing of task blocks is an existing data processing procedure, and will not be elaborated upon here.
[0035] In one possible implementation, each data processing node is equipped with a wireless interaction port for establishing a connection with a mobile terminal, enabling the mobile terminal to obtain the image processing results from the data processing node. For example, the mobile terminal accesses the data processing node through a router to obtain the image processing results from that node.
[0036] In one possible implementation, to accelerate data processing, each data node adopts a CPU-GPU heterogeneous computing architecture. Computational tasks are distributed between the CPU and GPU. The CPU implements multithreading technology to allow multiple computing threads to run simultaneously, while the GPU executes CUDA Streams to achieve parallel processing of multiple tasks. Through the collaborative work of the CPU and GPU, efficient utilization of heterogeneous computing resources is achieved, significantly improving data processing speed.
[0037] Figure 4This is an application example diagram of the multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR provided in this embodiment of the invention. For example... Figure 4 As shown, in actual operation, medium-high orbit SAR performs multiple high-frequency revisit observations of the same area on a single orbit, acquiring full-scene observation data at different time periods, such as full-scene image data at time t1 and full-scene image data at time t2. This full-scene observation data at different time periods is stored in real-time on a disk array. The disk array simultaneously acquires image processing results from multiple data processing nodes and merges these results to achieve data parallelism. A fiber optic switch (not shown in the figure) acquires full-scene observation data at different time periods and divides each full-scene observation data into grids to obtain multiple task blocks (i.e., sub-tasks in the figure). Based on the total number of data processing nodes, these task blocks are evenly distributed to each data processing node to achieve task parallelism. Each data processing node relies on a CPU+GPU architecture, supporting a multi-threaded model to process the received task blocks in parallel, obtaining the image data corresponding to each task block, thus achieving algorithm parallelism. Through data parallelism, task parallelism, and algorithm parallelism, the system's throughput for processing massive amounts of data can be effectively improved, meeting real-time or near-real-time imaging requirements.
[0038] To address the problem that existing serial processing methods suffer from low data processing efficiency when handling massive amounts of medium- and high-orbit SAR data, making it difficult to meet real-time or near-real-time imaging requirements, this invention provides a multi-level parallel processing system for ultra-large volumes of medium- and high-orbit SAR data. This system stores multiple full-scene observation data returned by medium- and high-orbit SAR within a preset period, decomposes each full-scene observation data into multiple task blocks, and processes these task blocks in parallel using multiple non-interfering distributed data processing nodes to obtain image processing results. This effectively improves the overall system throughput while maintaining high-resolution imaging, significantly increases data processing efficiency, and ensures the timeliness of data processing to meet real-time or near-real-time imaging requirements.
[0039] To verify the multi-level parallel processing system for large-scale medium- and high-orbit SAR data provided by this invention, Table 1 shows a simulation comparison between the existing distributed system and the multi-level parallel processing system provided by this invention. Each system has two data processing nodes, and the input data used in the simulation is 30K*30K SAR simulation echo data. The simulation process includes four steps: data distribution, pulse compression, data imaging, and result merging. As shown in Table 1, compared to the existing distributed system, the multi-level parallel processing system provided by this invention accelerates these four operations through multi-level parallelism, resulting in a 78% improvement in overall processing speed.
[0040] Table 1
[0041]
[0042] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A multi-stage parallel processing system for large data volume of SAR in medium-high orbit, characterized in that, include: A data storage device is used to store multiple full-scene observation data returned by a medium-high orbit synthetic aperture radar within a preset period, and simultaneously send the multiple full-scene observation data to a distributed computing device, and receive multiple image processing results sent by the distributed computing device; wherein, the preset period refers to a single day / single orbit, and a maximum of 16 observations are performed within the preset period. The distributed computing device includes a data exchange node and multiple data processing nodes connected to the data exchange node. The data exchange node is connected to the data storage device and is used to acquire the multiple full-scene observation data, decompose each full-scene observation data into multiple task blocks, and synchronously distribute the multiple task blocks to the multiple data processing nodes. The multiple data processing nodes are used to perform parallel processing on the received task blocks using a multi-threaded model and feed back the corresponding image processing results to the data exchange node. The threads in the multi-threaded model include: a data reading thread, a data processing thread, and a data writing thread for parallel computation; the trigger time of the data reading thread is earlier than the trigger time of the data processing thread and the trigger time of the data writing thread, and the trigger time of the data processing thread is earlier than the trigger time of the data writing thread. This multi-level parallel processing system processes multiple full-scene observation data through a multi-level parallel architecture that combines data parallelism, task parallelism, and algorithm parallelism. Specifically, the data storage device and the distributed computing device work together to achieve concurrent storage and retrieval of the multiple full-scene observation data and the multiple image processing results, thus constituting data parallelism. The data exchange node decomposes each full-scene observation data into multiple task blocks and distributes them to the multiple data processing nodes, thus constituting task parallelism. Each data processing node uses a multi-threaded model to perform parallel imaging operations on the received task blocks, thus constituting algorithm parallelism.
2. The multi-stage parallel processing system for SAR super large data volume in the medium-high orbit according to claim 1, characterized in that, Each data processing node corresponds to a node number; the data exchange node is also used for: Obtain the total number of the plurality of data processing nodes; The basic task quantity is obtained by dividing the total number of the multiple data processing nodes by the number of the multiple task blocks; Based on the aforementioned basic task volume, the multiple task blocks are split to obtain multiple basic task blocks; The multiple basic task blocks are synchronously distributed to the multiple data processing nodes in the order of their node numbers.
3. The multi-stage parallel processing system for SAR super large data volume in medium-high orbit according to claim 2, characterized in that, When the total number of the multiple data processing nodes is not divisible by the number of the multiple task blocks, the multiple task blocks are padded.
4. The multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR according to claim 1, characterized in that, The multi-threaded model is a producer-consumer model.
5. The multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR according to claim 1, characterized in that, The data exchange node uses the BP algorithm to decompose each full-scene observation data to obtain the multiple task blocks.
6. The multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR according to claim 1, characterized in that, Each data processing node is equipped with a wireless interaction port for establishing a connection with a mobile terminal, so that the mobile terminal can obtain the processing results of the data processing node.
7. The multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR according to claim 1, characterized in that, The data storage device is a disk array.
8. The multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR according to claim 1, characterized in that, The data exchange node is a fiber optic switch.
9. The multi-level parallel processing system for ultra-large data volumes of medium- and high-orbit SAR according to claim 1, characterized in that, The plurality of data processing nodes connected to the data exchange node are arranged in a star topology.