Intelligent processing method of satellite data

Abstract

The application provides an intelligent processing method for satellite data, which decomposes a huge task into a large number of parallel processing subtasks, fully utilizes the cluster computing power to shorten the product production time from the hour / day level to the minute / hour level, and meets the high timeliness application demand; dynamically schedules the tasks according to the data content logic and the real-time load of the cluster, avoids the idle or overload of the nodes; a single node failure only affects a small number of subtasks, and the system can automatically reschedule the subtasks to healthy nodes, realizes efficient fault-tolerant processing; supports fine processing based on geographical area, spectral characteristics and other dimensions, and can easily realize flexible business requirements such as priority processing of key areas; logical splitting based on domain knowledge ensures that the data processed by each subtask is complete in physics or logic; each node reads a specific part of a large file on demand, changes the centralized large-flow sequential I / O into distributed small-flow random I / O, significantly reduces the single-point pressure of the storage system, and improves the overall data throughput capacity.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and specifically to an intelligent processing method for satellite data. Background Technology

[0002] The current mainstream solutions for processing massive amounts of satellite data on the ground can be divided into two categories: Single-file serial processing: The entire data file transmitted from the satellite (typically hundreds of GB or even TB in size) is treated as a single processing unit. During processing, a high-performance server or workstation sequentially loads the entire file into memory and executes a series of algorithms, including Level 0 processing, radiometric calibration, geometric correction, and atmospheric correction, to ultimately generate the final product. The entire processing is completed on a single computing node.

[0003] General-purpose distributed frameworks often employ brute-force chunking: To utilize cluster resources, some systems use general-purpose distributed computing frameworks (such as Hadoop and Spark). These typically use simple methods like "splitting by fixed byte size" or "splitting by line" to physically divide large files into multiple equal-sized chunks, which are then distributed to different computing nodes for processing using the same algorithm. After processing, the chunks are simply merged.

[0004] The aforementioned existing technical solutions have the following inherent defects: Poor processing timeliness: The serial solution takes a very long time to process a very large file, and it relies entirely on the performance of a single machine, which cannot meet the high timeliness requirements of applications such as disaster emergency response and military reconnaissance.

[0005] Low and uneven resource utilization: Serial solutions cannot utilize the massive parallel computing resources of modern data centers, resulting in most nodes in the cluster being idle. Although the crude partitioning solution utilizes multiple nodes, its partitioning method is severely disconnected from the internal logical structure of satellite data (such as geographic continuity, band correlation, and scene unit integrity). This may cause a single complete ground object to be cut into different pieces, causing subsequent processing algorithms to fail or produce errors, resulting in low practical feasibility.

[0006] The system has weak robustness: In serial processing, failure at any stage leads to the entire task being redone, which is costly. In the brute-force block-based scheme, although node failures can be restarted, the lack of logical semantics in the blocks makes the retry and result aggregation logic complex.

[0007] The process is rigid and lacks flexibility: it cannot intelligently schedule tasks based on data content (such as prioritizing specific key areas) or dynamic resource conditions. Even when only a small portion of the data of interest needs to be processed, the entire file still needs to be processed, resulting in a huge waste of computing resources.

[0008] I / O becomes a bottleneck: Whether it is loading an entire large file on a single machine or reading physical fragments concurrently from multiple nodes, it puts enormous random or sequential I / O pressure on the shared storage system, slowing down the overall processing flow.

[0009] Therefore, an intelligent processing method that can efficiently and evenly process satellite data is needed. Summary of the Invention

[0010] This invention addresses the problems of poor timeliness, low and uneven resource utilization, weak system robustness, rigid processes, lack of flexibility, and I / O bottlenecks in satellite data processing. It provides an intelligent satellite data processing method that decomposes massive tasks into numerous parallel-processable subtasks, fully utilizing cluster computing power to reduce production time from hours / days to minutes / hours, meeting the demands of high-timeliness applications. Dynamic task scheduling based on data content logic and real-time cluster load avoids node idleness or overload, maximizing and balancing the utilization of computing, storage, and network resources. A single node failure only affects a few subtasks, and the system can automatically reschedule them to healthy nodes, achieving efficient fault tolerance. It supports fine-grained processing based on dimensions such as geographic region and spectral features, easily realizing flexible business needs such as "prioritizing key areas"; logical decomposition based on domain knowledge ensures that the data processed by each subtask is physically or logically complete (such as a complete city or a complete scene), avoiding geographic errors or information loss that may be caused by rough block partitioning, and guaranteeing the quality of the final product from a mechanism perspective; each node reads specific parts of large files on demand, changing centralized high-volume sequential I / O to distributed low-volume random I / O, significantly reducing the single-point pressure on the storage system and improving the overall data throughput capacity.

[0011] This invention provides an intelligent processing method for satellite data, comprising the following steps: S1. Perform data feature analysis on the raw satellite data to obtain the input data analysis result. Based on the input data analysis result, the user-specified processing target, and the real-time cluster resource status of the distributed processing node cluster, make a logical splitting decision on the raw satellite data and generate task descriptors corresponding to multiple sub-tasks that can be processed in parallel. S2. Continuously monitor and calculate the real-time health status and resource utilization of each node in the distributed processing node cluster, and dynamically and evenly distribute the task descriptors of subtasks to idle or low-load nodes. S3. After receiving the assigned task descriptor, the node reads the data block it is responsible for in the original satellite data through the file access interface according to the logical index in the task descriptor, without waiting for the entire subtask file to be transferred or physically cut. S4. The node determines the maximum amount of data it can process based on the hardware parameter information. Using the maximum amount of data as a benchmark, the data blocks are processed according to the satellite category to obtain data units. The node performs binary reading on the data units, performs image enhancement on the optical data and / or extracts satellite data files from the SAR data and locates the original data to obtain intermediate result files. S5. Monitor all running subtasks in the entire distributed processing node cluster throughout their entire lifecycle, collect status in real time, and automatically trigger a retry mechanism when a task failure is detected, putting the failed task back into the scheduling queue and assigning it to other healthy nodes for execution. S6. After all subtasks are successfully completed, the intermediate result files distributed across the nodes are aggregated based on the logic and geographic index recorded when the task was generated. For optical data, pixel-level geographic mosaicking and color equalization are performed during aggregation; for SAR data, phase stitching and consistency correction are performed during aggregation; finally, the final aggregated product is subjected to quality checks and a processing report is generated.

[0012] In the intelligent processing method for satellite data described in this invention, as a preferred embodiment, in step S1, the data feature analysis module extracts the geographical range, projection, number of bands and wavelength, and imaging time of the optical data in the original satellite data, and analyzes the imaging mode, polarization mode, orbital parameters and organization format of the original data file of the SAR data in the original satellite data. The imaging mode is strip, scanning or spotting. The adaptive task planning engine makes logical splitting decisions. The adaptive task planning engine integrates a domain knowledge base for optical and SAR data. The domain knowledge base includes a variety of predefined splitting rules, which can be any one or more of the following: splitting by geographic grid, splitting by administrative division, splitting by spectral subset, and splitting by distance / azimuth. The task descriptor includes the data index and processing parameters.

[0013] The intelligent processing method for satellite data described in this invention, as a preferred embodiment, includes the following steps in step S1: S11. Reset the runtime environment and clear unnecessary data cache; S12. Access the raw satellite data in binary read mode, decode it, and extract the mission flag; S13. Match the task flag with the built-in policy mapping table, lock the splitting rules, and then enter the main processing loop. S14. The main processing loop reads the subsequent content of the source file through the buffer in a streaming manner and detects the split boundary of the data stream in real time according to the selected strategy. When the predefined boundary judgment condition is met, the current buffered data is immediately encapsulated into a complete sub-file and written to the storage system. The naming of the sub-file follows a standardized paradigm. This iterative process continues until the source file data stream terminates. S15. After all logical splits are completed, release all file handles and generate a manifest document.

[0014] In the intelligent processing method for satellite data described in this invention, as a preferred embodiment, in step S13, the splitting rules are: based on fixed-size data block division and / or logical segmentation according to delimiters and / or dynamic segmentation following the embedded data length field within the file; the delimiter is a protocol header, a marker, or a newline sequence; In step S14, the boundary determination conditions include: the buffer reaches the predetermined capacity, a complete delimiter sequence is identified, or the preset number of data units is calculated; In step S15, the manifest document records the metadata of all sub-files in a structured format, including filename, byte size, checksum, and logical order in the original file.

[0015] In the intelligent processing method for satellite data described in this invention, as a preferred embodiment, in step S2, subtask allocation is performed based on a queue and resource-aware scheduling algorithm, wherein the queue and resource-aware scheduling algorithm is either minimum load priority or data locality priority.

[0016] In a preferred embodiment of the intelligent satellite data processing method described in this invention, step S4 involves performing scene segmentation processing on the optical data acquired by optical and multispectral sensors based on time sequence and fixed swath width. Based on imaging physical constraints, SAR data is segmented into scenes. Then, the coordinates of the four corner points of each segment are calculated. The coordinates of the pitch and roll angles corresponding to the four corner points are transformed to obtain the latitude and longitude of the four corner points.

[0017] The intelligent processing method for satellite data described in this invention, in a preferred embodiment, includes the following steps in step S4: S41. Read the hardware parameter information of the node and combine it with the computational amount of the algorithm to be executed in the future to comprehensively evaluate the maximum amount of data that can be processed. Based on the maximum amount of data, the data blocks are processed according to the category to obtain data units. S42. The node reads the data unit in binary format and extracts key operating parameters according to the preset data protocol format. Key operating parameters include information such as satellite type, position, speed, and attitude. S43. Perform scene segmentation operation. Based on the time series and fixed swath width, and according to the acquired time series and satellite orbit parameters, combined with the preset global scene reference grid system, the continuous data stream is segmented into scenes with independent time markers and complete spatial coverage. The scenes are geographically seamless or only have limited systematic overlap preset to ensure global continuous coverage. For SAR data, based on imaging physical constraints, an overlapping observation zone with time length is introduced when dividing the scene along the azimuth. The length of the overlapping observation zone is calculated by the sensor parameters and imaging mode to meet the minimum requirements of synthetic aperture time. S44. Perform image enhancement on the scene-by-scene files of the optical data respectively; For each SAR data scene file, the coordinates of the four corner points of each file are calculated. Based on the coordinates of the satellite in the celestial coordinate system and the attitude angles corresponding to the four corner points, the coordinates of the satellite in the orbital coordinate system are obtained. Then, the SAR data parsing results of the scene file are obtained through coordinate transformation.

[0018] In a preferred embodiment of the intelligent satellite data processing method described in this invention, the aperture synthesis time in step S43 is: ; Among them, 0.886λR / L a The azimuth beamwidth is projected onto the ground. When the angle of inclination is not zero, the projected length in the azimuth direction needs to be multiplied by a coefficient. .

[0019] The intelligent processing method for satellite data described in this invention, as a preferred embodiment, includes the following steps in step S44: The method for processing SAR data scene files: Read satellite positions from the segmented data file and satellite attitude , These are the yaw angle, pitch angle, and roll angle of the satellite platform, respectively. The satellite azimuth beamwidth is obtained from the satellite position information. and pitch beamwidth The attitude angles of the four corner points corresponding to the scene-separated data file are as follows: ; according to The satellite's coordinates in the orbital coordinate system are obtained by combining the attitude angles of the four corner points. Perform coordinate transformation: ; in ; Then, the satellite coordinates are transformed from the orbital coordinate system to the inertial coordinate system and then to the Earth-fixed coordinate system to obtain the satellite Earth-fixed coordinates. ; Based on the transformation relationships between the celestial coordinate system, orbital coordinate system, inertial coordinate system, and fixed coordinate system, the Earth-fixed system vectors corresponding to the beam pointing at the four corner points are obtained. The equation of the line is: ; Among them, satellite position Given that t is time; Solve the equations of the straight line and the Earth simultaneously, and take the closest value to obtain the position of the intersection point of the satellite beam pointing to the Earth in the Earth-fixed coordinate system. Then convert the position in the Earth-fixed coordinate system to geodetic latitude and longitude to obtain the SAR data analysis results of the scene file.

[0020] The intelligent satellite data processing method described in this invention, as a preferred embodiment, uses an architecture deployed between a ground data receiving station and a standard product production system, including an intelligent parsing and planning layer, a dynamic scheduling and execution layer, and an aggregation and control layer set from top to bottom; The intelligent parsing and planning layer includes a data feature parsing module for data feature parsing and an adaptive task planning engine for logical decomposition and decision-making; The dynamic scheduling and execution layer includes a task distributor and load balancer that serves as the dynamic scheduling center, and a distributed processing node cluster consisting of multiple physical or virtual machines with independent computing and storage capabilities. The aggregation and control layer includes a task status monitor for monitoring subtasks and a result aggregation and quality manager for aggregating intermediate result files and performing quality checks.

[0021] This invention provides a process for processing satellite data on the ground, an intelligent processing system architecture for intelligent splitting of big data, relevant processing algorithms for file splitting, and an algorithm for positioning based on relevant satellite attitudes.

[0022] The present invention has the following advantages: (1) This invention greatly improves processing efficiency and timeliness: it decomposes a large number of sub-tasks that can be processed in parallel, makes full use of the cluster computing power, and shortens the product production time from "hours / days" to "minutes / hours", meeting the needs of high timeliness applications.

[0023] (2) The present invention realizes intelligent resource utilization and load balancing: dynamic task scheduling is performed according to the data content logic and the real-time load of the cluster, avoiding node idleness or overload, so that computing, storage and network resources are maximized and balanced.

[0024] (3) This invention enhances the robustness and flexibility of the system: a single node failure only affects a few subtasks, and the system can automatically reschedule them to healthy nodes, achieving efficient fault-tolerant processing. It supports refined processing based on dimensions such as geographical region and spectral features, and can easily realize flexible business requirements such as "prioritizing key areas".

[0025] (4) The present invention can guarantee the correctness of the processing logic and the quality of the product: the logical decomposition based on domain knowledge ensures that the data processed by each subtask is physically or logically complete (such as a complete city or a complete scene), avoiding geographical errors or missing information that may be caused by rough block division, and ensuring the quality of the final product from the mechanism.

[0026] (5) The present invention effectively alleviates the I / O bottleneck: each node reads a specific part of a large file as needed, changing centralized high-volume sequential I / O to distributed low-volume random I / O, which significantly reduces the single-point pressure of the storage system and improves the overall data throughput. Attached Figure Description

[0027] Figure 1 A flowchart of an intelligent processing method for satellite data; Figure 2 S4 is a flowchart of the steps in an intelligent satellite data processing method. Figure 3 A scene-by-scene flowchart of an intelligent satellite data processing method; Figure 4 A flowchart of SAR data processing after scene segmentation, which is a smart processing method for satellite data; Figure 5 This is a schematic diagram of the direct imaging result without scene division; Figure 6 This is a flowchart of the image stitching result after scene segmentation, which is a smart processing method for satellite data. Detailed Implementation

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Example 1

[0029] like Figures 1-4 As shown, an intelligent processing method for satellite data can perform parallel processing of ground blocks through "content perception, adaptive splitting, and dynamic scheduling". Its core components and process are as follows: The system of this invention adopts a layered architecture design, deployed between the ground data receiving station and the standard product production system, and mainly includes the following three layers from top to bottom: Intelligent Analysis and Planning Layer: Data Feature Analysis Module: Responsible for automatically analyzing the raw satellite data entered into the database. For optical data, it extracts its geographical extent, projection, number of bands and wavelength, imaging time, etc.; for SAR data, it analyzes its imaging mode (strip / scan / spotlight), polarization, orbital parameters, and the organization format of the raw data file. This module is the foundation of the system's "sensory content".

[0030] The adaptive task planning engine, acting as the system's "brain," integrates a domain knowledge base for optical and SAR data (containing various predefined splitting rules, such as by geographic grid, administrative division, spectral subset, range / azimuth partitioning, etc.). The engine synthesizes the parsing results of the input data, user-specified processing objectives (such as rapid mapping, precise interferometry), and real-time cluster resource status (CPU / memory load, GPU utilization, network and storage I / O) obtained from the resource monitor. Through built-in strategy algorithms, it determines the optimal logical splitting scheme and generates a series of task descriptors carrying explicit data indexes and processing parameters.

[0031] Dynamic scheduling and execution layer: Task Distributor and Load Balancer: Receives task queues from the planning engine. It is a dynamic scheduling center that continuously monitors the real-time health status and resource utilization of each node in the computing cluster. Employing queue-based and resource-aware scheduling algorithms (such as minimum load priority and data locality priority), it dynamically and evenly distributes tasks to the most suitable idle or low-load nodes, ensuring overall cluster load balance and avoiding single-point overload.

[0032] A distributed processing node cluster consists of multiple physical or virtual machines with independent computing and storage capabilities. Each node deploys a unified processing executor. After receiving a task descriptor, the executor reads the data block it is responsible for from the original large file through an efficient file access interface, based on the logical index within it, without waiting for the entire file to be transferred or physically split. Subsequently, it calls the corresponding processing algorithm library (such as radiometric calibration, SAR focusing, orthorectification, etc.) in local memory to complete the entire calculation process.

[0033] Aggregation and Control Layer: Task Status Monitor: Monitors all running subtasks throughout the entire cluster, collecting their status (waiting, running, successful, failed) in real time. When a task failure is detected, a retry mechanism is automatically triggered, putting the task back into the scheduling queue and having it distributed to another healthy node for execution by the dispatcher, achieving system-level fault tolerance.

[0034] Results Aggregation and Quality Manager: This module starts after all subtasks have successfully completed. Based on the precise logical and geographic indexes recorded during task generation, it automatically aggregates the intermediate result files distributed across various nodes. For optical imagery, it performs pixel-level geomapping and color equalization; for SAR data (such as interferometric phase maps), it performs precise phase stitching and consistency correction. Finally, it performs quality checks on the aggregated final product and generates a processing report.

[0035] i. Core Algorithm Flow The core algorithm of this solution revolves around data partitioning and parallel processing, with the key steps as follows: Data file parsing and extraction algorithms: The algorithm parses the satellite data received by the receiving station according to the data transmission and payload data format, and extracts relevant auxiliary information.

[0036] Data intelligent splitting algorithm: Since the information received by the ground receiving station may contain data from multiple orbits and multiple missions, and all the data is in one file, it cannot be processed directly. Therefore, this algorithm is needed to intelligently split the file.

[0037] This module is divided into two parts: the first part is the file splitting algorithm, and the second part is the scene segmentation algorithm.

[0038] File splitting algorithm: During the algorithm initialization phase, the runtime environment needs to be reset, and unnecessary data cache needs to be cleared to prepare for the next processing. Subsequently, the algorithm accesses the source file in binary read mode, and then decodes the data according to the pre-agreed data protocol, extracting the data at the corresponding offset and storing it in the pre-set parameters.

[0039] The algorithm then extracts task flags from pre-defined parameters. These flags are then matched against a built-in policy mapping table. The policy table explicitly defines the splitting rules corresponding to various flags, such as: data block partitioning based on fixed sizes, logical segmentation based on specific delimiters (e.g., protocol headers, markers, or newline sequences), or dynamic segmentation following the data length field embedded within the file.

[0040] After locking the splitting rules, the algorithm enters the main processing loop. It reads the subsequent content of the source file in a buffered stream and detects the splitting boundaries of the data stream in real time according to the selected strategy. When the predefined boundary judgment conditions are met (such as the buffer reaching a predetermined capacity, identifying a complete delimiter sequence, or calculating a preset number of data units), the algorithm immediately encapsulates the current buffered data into a complete sub-file and writes it to the storage system. The naming of the sub-file follows a standardized paradigm, and this iterative process continues until the source file data stream terminates.

[0041] The algorithm concludes with cleanup operations, including releasing all file handles and optionally generating a manifest document. This manifest records the metadata of all subfiles in a structured format (such as JSON), including filenames, byte sizes, checksums, and their logical order within the original files, ensuring the integrity and traceability of the dataset. Finally, the algorithm outputs the storage location of the subfile set and its manifest, marking the completion of the processing task.

[0042] Scene segmentation algorithm Based on the differences in sensor imaging principles, this algorithm automatically segments the file after the file splitting algorithm into data units ("scenes") that conform to standard specifications, and provides accurate geometric and radiometric reference information for subsequent processing.

[0043] In the initial stage of the algorithm, the computation process is divided into two main parts.

[0044] The first part reads hardware parameter information, including the size of the running memory and the number of threads. Then, it combines this information with the computational load of the algorithms to be executed (such as CS imaging algorithms and image enhancement algorithms) to comprehensively evaluate the maximum amount of data that can be processed. For example, if the hardware running memory is 32GB and the data expands to twice its original size during algorithm processing, then the maximum amount of original data that can be supported is 16GB. Subsequent scene-by-scene processing will be completed based on 16GB.

[0045] The second part is responsible for reading the data in binary format and extracting key working parameters, including satellite type, position, speed, attitude and other information, according to the preset data protocol format.

[0046] The algorithm then determines the type of satellite. For data acquired by non-synthetic aperture radar (non-SAR) sensors (such as optical and multispectral sensors), the scene segmentation operation mainly follows the principles of time sequence and fixed swath width. Based on the precise time series of the acquired data and satellite orbital parameters, combined with a pre-set global scene reference grid system, the algorithm segments the continuous data stream into scenes with independent time markers and complete spatial coverage. The segmentation boundaries are usually clear and well-defined, and the scenes are geographically seamless or only have limited systematic overlap pre-set to ensure continuous global coverage.

[0047] For Synthetic Aperture Radar (SAR) sensor data, the scene segmentation strategy strictly adheres to its imaging physical constraints. SAR's azimuth high resolution depends on a continuous observation time, known as the synthetic aperture time. To ensure that even the edge regions of the generated image achieve nominal resolution, the algorithm actively introduces overlapping observation zones of a specific time length when segmenting along the azimuth. The length of the overlapping zone is precisely calculated using sensor parameters and imaging modes to meet the minimum requirements of the synthetic aperture time. This design ensures the geometric quality consistency of single-scene data and preserves necessary data redundancy and phase continuity for subsequent multi-scene collaborative processing such as interferometry and temporal analysis. The synthetic aperture time is calculated as follows: ; In the above formula The azimuth beamwidth is projected onto the ground. When the angle of inclination is not zero, this projection length in the azimuth direction needs to be multiplied by a coefficient. .

[0048] When performing data segmentation, the data is segmented according to time, and the time of the original data and related auxiliary data is matched one-to-one. Then, these data are packaged into a file.

[0049] After the data segmentation operation is completed, the location calculation is performed on each file according to the number of files in the segment to obtain the coordinates of the four corner points of each file.

[0050] This scene segmentation algorithm integrates intelligent sensor discrimination and adaptive scene segmentation logic to achieve standardized and high-quality preprocessing of multi-source heterogeneous satellite remote sensing data, effectively improving the usability of data products and the automation level of downstream advanced product production.

[0051] Data location algorithm: Based on the satellite attitude information and satellite orbit information extracted from the satellite data, the original data is located.

[0052] For satellites observing Earth, the beam has a certain azimuth beamwidth and pitch beamwidth. For the observation area, the four corner points correspond to different pitch and roll angles. The specific calculation flowchart is shown in the figure below.

[0053] First, the algorithm reads the satellite positions from the segmented data file. and the satellite's attitude ,in, These refer to the three attitude angles of the satellite platform: yaw, pitch, and roll.

[0054] At the same time, the azimuth beamwidth of the satellite can be obtained based on the satellite's wave position information. and pitch beamwidth This allows us to obtain the attitude angles of the four corner points corresponding to the data. ; Based on the satellite's coordinates in the celestial coordinate system And the attitude angles corresponding to the four corner points. The satellite's coordinates in the orbital coordinate system can then be obtained as follows: According to the coordinate transformation formula, we can obtain: ; in ; Then, the satellite coordinates are transformed from the orbital coordinate system to the inertial coordinate system and then to the Earth-fixed coordinate system to obtain the satellite's coordinates in the Earth-fixed coordinate system. .

[0055] Simultaneously, based on the transformation relationships between the celestial coordinate system, orbital coordinate system, inertial coordinate system, and fixed coordinate system, the beam pointing vectors corresponding to the four corner points of the image in the Earth-fixed system can be obtained. In addition to satellite position Given that, the equation of the line can be obtained as: ; Solving the above equation simultaneously with the Earth's equations and taking the closest value gives us the location of the intersection point between the satellite beam and the Earth in the Earth-fixed coordinate system. Then, converting this location to geodetic latitude and longitude yields the final calculation result.

[0056] Dynamic task allocation and image stitching algorithm: The algorithm calls different hardware to process the data according to the input requirements and file splitting requirements, and then uses different methods to stitch the results based on the different data. For optical data, a feather-based seamless mosaicking algorithm is used; for SAR data (such as SLC or interferometric phase), a weighted average or phase gradient integral algorithm based on the overlapping area is used to stitch the data, ensuring the spatial continuity of amplitude and phase.

[0057] The image stitching result after scene segmentation in this embodiment is as follows: Figure 6 As shown, compared with the results of direct imaging without scene division Figure 5 The differences are minimal; that is, the stitched image after scene segmentation is essentially the same as the direct image without scene segmentation. However, direct imagery without scene segmentation has high hardware requirements and cannot be processed when the data volume is large. The scene segmentation and stitching algorithm of this invention can split large files into multiple smaller files, process them on a distributed server, and then stitch them together, greatly improving efficiency while maintaining consistent stitching results.

[0058] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for intelligent processing of satellite data, characterized in that: Includes the following steps: S1. Perform data feature parsing on the raw satellite data to obtain input data parsing results. Based on the input data parsing results, the user-specified processing target, and the real-time cluster resource status of the distributed processing node cluster, make a logical splitting decision on the raw satellite data to generate task descriptors corresponding to multiple sub-tasks that can be processed in parallel. S2. Continuously monitor and calculate the real-time health status and resource utilization of each node in the distributed processing node cluster, and dynamically and evenly distribute the task descriptors of the subtasks to idle or low-load nodes. S3. After receiving the assigned task descriptor, the node reads the data block it is responsible for in the original satellite data through the file access interface according to the logical index in the task descriptor, without waiting for the entire sub-task file to be transferred or physically cut. S4. The node determines the maximum amount of data it can support processing based on hardware parameter information. Using the maximum amount of data as a benchmark, the node performs scene processing on the data block according to satellite category to obtain data units. The node performs binary reading on the data units, performs image enhancement on optical data and / or extracts satellite data files from SAR data and locates the original data to obtain intermediate result files. S5. Monitor all running subtasks in the entire distributed processing node cluster throughout their entire lifecycle, collect status in real time, and automatically trigger a retry mechanism when a task failure is detected, putting the failed task back into the scheduling queue and assigning it to other healthy nodes for execution. S6. After all the sub-tasks are successfully completed, the intermediate result files distributed across the nodes are aggregated according to the logic and geographic index recorded when the task is generated. For optical data, pixel-level geographic mosaicking and color equalization are performed during aggregation; for SAR data, phase stitching and consistency correction are performed during aggregation; finally, the final product after aggregation is subjected to quality inspection and a processing report is generated.

2. The intelligent processing method for satellite data according to claim 1, characterized in that: In step S1, the data feature analysis module extracts the geographical range, projection, number of bands and wavelength, and imaging time of the optical data in the original satellite data, and analyzes the imaging mode, polarization mode, orbital parameters and organization format of the original data file of the SAR data in the original satellite data. The imaging mode is strip, scan or spotlight. An adaptive task planning engine makes logical splitting decisions. The adaptive task planning engine integrates a domain knowledge base for optical and SAR data. The domain knowledge base includes a variety of predefined splitting rules, which are any one or more of the following: splitting by geographic grid, splitting by administrative division, splitting by spectral subset, and splitting by distance / azimuth. The task descriptor includes a data index and processing parameters.

3. The intelligent processing method for satellite data according to claim 1, characterized in that: In step S1, the logical decomposition method includes the following steps: S11. Reset the runtime environment and clear unnecessary data cache; S12. Access the raw satellite data in binary read mode and decode and extract the mission flag; S13. Match the task flag with the built-in strategy mapping table, lock the splitting rule, and then enter the main processing loop. S14. The main processing loop reads the subsequent content of the source file through the buffer in a streaming manner, and detects the split boundary of the data stream in real time according to the selected strategy. When the predefined boundary judgment condition is met, the current buffered data is immediately encapsulated into a complete sub-file and written to the storage system. The naming of the sub-file follows a standardized paradigm. This iterative process continues until the source file data stream terminates. S15. After all logical splits are completed, release all file handles and generate a manifest document.

4. The intelligent processing method for satellite data according to claim 3, characterized in that: In step S13, the splitting rules are: based on fixed-size data block division and / or logical segmentation according to delimiters and / or dynamic segmentation following the embedded data length field within the file; The delimiter is a protocol header, a marker, or a newline sequence; In step S14, the boundary determination conditions include: the buffer reaches the predetermined capacity, a complete delimiter sequence is identified, or the preset number of data units is calculated; In step S15, the manifest document records the metadata of all sub-files in a structured format, including filename, byte size, checksum, and logical order in the original file.

5. The intelligent processing method for satellite data according to claim 1, characterized in that: In step S2, subtasks are allocated based on a queue- and resource-aware scheduling algorithm, which is either minimum load priority or data locality priority.

6. The intelligent processing method for satellite data according to claim 1, characterized in that: In step S4, the optical data acquired by the optical and multispectral sensors are processed by scene segmentation according to the time sequence and fixed swath width; Based on imaging physical constraints, SAR data is segmented, and then the coordinates of the four corner points of each segment are calculated. The coordinates of the pitch and roll angles corresponding to the four corner points are transformed to obtain the latitude and longitude of the four corner points.

7. The intelligent processing method for satellite data according to claim 1, characterized in that: Step S4 includes the following steps: S41. Read the hardware parameter information of the node and combine it with the computational amount of the algorithm to be executed in the future to comprehensively evaluate the maximum amount of data that can be processed. Based on the maximum amount of data, the data block is processed according to the category to obtain data units. S42. The node performs binary reading on the data unit and extracts key operating parameters according to a preset data protocol format. The key operating parameters include information such as satellite type, position, speed, and attitude. S43. Perform scene segmentation operation. Based on the time series and fixed swath width, and according to the acquired time series and satellite orbit parameters, combined with the preset global scene reference grid system, the continuous data stream is segmented into scenes with independent time markers and complete spatial coverage. The scenes are geographically seamless or only have limited systematic overlap preset to ensure global continuous coverage. For SAR data, based on imaging physical constraints, when dividing the scene along the azimuth, an overlapping observation area with a time length is introduced. The length of the overlapping observation area is calculated by the sensor parameters and imaging mode to meet the minimum requirements of synthetic aperture time. S44. Perform image enhancement on the scene-by-scene files of the optical data respectively; For each SAR data scene file, the coordinates of the four corner points of each file are calculated. Based on the coordinates of the satellite in the celestial coordinate system and the attitude angles corresponding to the four corner points, the coordinates of the satellite in the orbital coordinate system are obtained. Then, the SAR data parsing results of the scene file are obtained through coordinate transformation.

8. The intelligent processing method for satellite data according to claim 7, characterized in that: In step S43, the time for synthesizing the pore size is: ； Among them, 0.886λR / L a The azimuth beamwidth is projected onto the ground. When the angle of inclination is not zero, the projected length in the azimuth direction needs to be multiplied by a coefficient. .

9. The intelligent processing method for satellite data according to claim 7, characterized in that: In step S44, the method for processing the scene files of SAR data includes the following steps: Read satellite positions from the segmented data file and satellite attitude , These are the yaw angle, pitch angle, and roll angle of the satellite platform, respectively. The satellite azimuth beamwidth is obtained from the satellite position information. and pitch beamwidth The attitude angles of the four corner points corresponding to the scene-separated data file are as follows: ； according to The satellite's coordinates in the orbital coordinate system are obtained by combining the attitude angles of the four corner points. Perform coordinate transformation: ； in: ； Then, the satellite coordinates are transformed from the orbital coordinate system to the inertial coordinate system and then to the Earth-fixed coordinate system to obtain the satellite Earth-fixed coordinates. ; Based on the transformation relationships between the celestial coordinate system, orbital coordinate system, inertial coordinate system, and fixed coordinate system, the Earth-fixed system vectors corresponding to the beam pointing at the four corner points are obtained. The equation of the line is: ； Among them, satellite position Given that t is time; Solve the equations of the straight line and the Earth simultaneously, and take the closest value to obtain the position of the intersection of the satellite beam pointing and the Earth in the Earth-fixed coordinate system. Then convert the position in the Earth-fixed coordinate system into geodetic latitude and longitude to obtain the SAR data analysis result of the scene file.

10. The intelligent processing method for satellite data according to claim 1, characterized in that: The architecture used in the intelligent processing method of satellite data is deployed between the ground data receiving station and the standard product production system, including a top-down intelligent parsing and planning layer, a dynamic scheduling and execution layer, and an aggregation and control layer; The intelligent parsing and planning layer includes a data feature parsing module for data feature parsing and an adaptive task planning engine for logical decomposition decision-making. The dynamic scheduling and execution layer includes a task distributor and load balancer that serves as the dynamic scheduling center, and a distributed processing node cluster consisting of multiple physical or virtual machines with independent computing and storage capabilities. The aggregation and control layer includes a task status monitor for monitoring subtasks and a result aggregation and quality manager for aggregating and quality checking intermediate result files.

Images