A remote sensing satellite data processing method, device and electronic equipment based on Spark

Through the Spark-based data processing method, the remote sensing satellite data is blocked and parallelized, which solves the problem of computing and storage resource limitation in traditional methods, and improves the processing efficiency of remote sensing satellite data and system fault tolerance.

CN118694429BActive Publication Date: 2024-11-05CHINA SURVEY SURVEYING & MAPPING TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411178304.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-05
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

Traditional remote sensing data processing methods are limited by computing and storage resources, and cannot effectively process raw data of large-scale remote sensing satellites, resulting in inefficient processing.

Method used

The Spark-based data processing method is adopted to block the original data through data reading configuration files, and combined with Spark's Union operator, FlatMap function and virtual channel identifier for channel merging, AOS frame synchronization, partitioning and sorting. Improve data processing efficiency.

Benefits of technology

Through blocking and parallel processing, the preprocessing efficiency of remote sensing satellite data is significantly improved, the data processing burden is reduced, the processing efficiency is significantly improved, and the system's fault tolerance is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118694429B_ABST
    Figure CN118694429B_ABST
Patent Text Reader

Abstract

The present application discloses a remote sensing satellite data processing method, device and electronic device based on Spark. The method is based on a data reading configuration file in Spark. In the process of reading the original data collected by the remote sensing satellite, the original data is processed in blocks, and the original file with a large data volume is divided into file blocks with a small data volume for processing; secondly, AOS frame synchronization processing is performed on each block data in a parallel processing manner; then, the valid frame data in the block data is partitioned, the valid frame data in each partition is sorted in a parallel processing manner, and the BPDU bit stream data of each frame data is extracted from the sorted valid frame data, and the effective load data corresponding to each partition is obtained, which can improve the preprocessing efficiency of satellite remote sensing data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of remote sensing satellite technology, and in particular to a remote sensing satellite data processing method, device, electronic equipment and medium based on Spark. Background Art

[0002] Remote sensing satellites generally use dual-channel polarization multiplexing to transmit data to the ground. Since the recording rates of each payload data are different, the channels assigned to the recorded data during playback are not fixed, and there are slight differences in the time when the two channels start transmission, the data type (virtual channel) and sorting method (VCDU count) of the data frames transmitted in the two channels are not fixed, that is, "mixed data transmission". Under this transmission characteristic, a large amount of data processing work needs to be completed when processing data on the ground. In addition, the raw data of remote sensing satellites are usually stored in binary file format, and have the characteristics of high resolution and large capacity. Processing the raw data of large-scale remote sensing satellites requires a lot of computing and storage resources.

[0003] Traditional remote sensing data processing methods are limited by computing and storage resources and cannot meet the needs of processing large-scale remote sensing satellite raw data, resulting in low processing efficiency. Summary of the invention

[0004] To this end, the present application provides a remote sensing satellite data processing method, device and electronic device based on Spark to solve the problem of low processing efficiency of traditional remote sensing data processing methods in the prior art due to the limitations of computing and storage resources.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides: a remote sensing satellite data processing method based on Spark, the method comprising: based on the data reading configuration file in Spark, in the process of reading the original data collected by the remote sensing satellite, the original data is processed in blocks to obtain the first channel data after blocking and the second channel data after blocking; the original data includes the first channel data and the second channel data; the first channel data after blocking and the second channel data after blocking are processed by channel merging to obtain the merged block data; the block data includes multiple AOS frames, and the AOS frame includes a virtual channel identifier; each block data is synchronized by AOS frame Processing is performed to obtain processed block data; based on the virtual channel identifier, valid frame data is obtained from the processed block data; based on the virtual channel identifier, the valid frame data is partitioned to obtain valid frame data in different partitions; for the valid frame data in each partition, the valid frame data is sorted by AOS frame to obtain sorted valid frame data in different partitions; for each partition, the BPDU bit stream data of each frame data is extracted from the sorted valid frame data to obtain the effective load data corresponding to each partition; for each partition, the BPDU bit stream data of each frame data is extracted from the sorted valid frame data to obtain the effective load data corresponding to each partition.

[0006] According to the first aspect, or any implementation of the first aspect above, the reading configuration file includes a preset data amount; based on the data reading configuration file in Spark, in the process of reading the original data collected by the remote sensing satellite, the original data is processed in blocks to obtain the first channel data after blocking and the second channel data after blocking, including: in the process of reading the original data collected by the remote sensing satellite, the original data is blocked according to the size of the preset data amount.

[0007] According to the first aspect, or any implementation of the first aspect above, the first channel data after blocking and the second channel data after blocking are channel-merged to obtain the merged block data, including: based on the Union operator in Spark, the first channel data after blocking and the second channel data after blocking are channel-merged to obtain the merged block data; wherein the merged block data is the data of one channel.

[0008] According to the first aspect, or any implementation method of the first aspect above, the AOS frame synchronization processing is performed on each block data to obtain the processed block data, including: for each block data, calling the FlatMap function in Spark, and dividing the block data into independent frame data based on the synchronization header.

[0009] According to the first aspect, or any implementation of the first aspect above, obtaining valid frame data from the processed block data based on the virtual channel identifier includes: filtering out data whose virtual channel identifier is a fill frame from the processed block data to obtain valid frame data.

[0010] According to the first aspect, or any implementation method of the first aspect above, the AOS frame includes a file number and a frame count; for the valid frame data in each partition, AOS frame sorting processing is performed on the valid frame data to obtain valid frame data sorted in different partitions, including: for the valid frame data in each partition, AOS frame sorting processing is performed on the valid frame data based on the file number and the frame count to obtain valid frame data sorted in different partitions.

[0011] According to the first aspect, or any implementation of the first aspect above, the method further includes: obtaining invalid frame data from the processed block data based on the virtual channel identifier; and performing bit error rate analysis on the invalid frame data based on the Spark data analysis tool.

[0012] The second aspect of the present application provides a remote sensing satellite data processing device based on Spark, which includes: a reading module, which is used to read a configuration file based on data in Spark, and in the process of reading the original data collected by the remote sensing satellite, the original data is processed in blocks to obtain the first channel data after block and the second channel data after block; the original data includes the first channel data and the second channel data; a merging module, which is used to perform channel merging processing on the first channel data after block and the second channel data after block to obtain the merged block data; the block data includes multiple AOS frames, and the AOS frames include virtual channel identifiers; a frame synchronization module, which is used to perform AOS frame synchronization processing on each block data to obtain the processed block data; a filtering module, which is used to obtain valid frame data from the processed block data based on the virtual channel identifier; a classification module, which is used to partition the valid frame data based on the virtual channel identifier to obtain the valid frame data under different partitions; a sorting module, which is used to perform AOS frame sorting processing on the valid frame data in each partition to obtain the sorted valid frame data under different partitions.

[0013] The third aspect of the present application provides an electronic device, which includes: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement any one of the methods in the first aspect; one or more I / O interfaces, connected between the processor and the memory, and configured to implement information interaction between the processor and the memory.

[0014] A third aspect of the present application provides a computer-readable medium having a computer program stored thereon, which implements any one of the methods in the first aspect when the program is executed by a processor.

[0015] The present application has the following advantages: The embodiment of the present application provides a remote sensing satellite data processing method based on Spark. In the method, firstly, based on the data reading configuration file in Spark, in the process of reading the original data collected by the remote sensing satellite, the original data is processed in blocks to obtain the first channel data and the second channel data after blocking. This step divides the original file with a large amount of data into file blocks with a small amount of data for processing, which can reduce the data processing burden and improve the preprocessing efficiency of satellite remote sensing data; secondly, the first channel data after blocking and the second channel data after blocking are channel merged to obtain the merged block data, and the AOS frame synchronization processing is performed on each block data to obtain the processed block data. This step is parallel processing. AOS frame synchronization processing is performed on each block data in a rational manner, which can further improve the preprocessing efficiency of satellite remote sensing data; then, based on the virtual channel identifier, valid frame data is obtained from the processed block data, and based on the virtual channel identifier, the valid frame data is partitioned to obtain valid frame data in different partitions. For the valid frame data in each partition, AOS frame sorting processing is performed on the valid frame data to obtain sorted valid frame data in different partitions. For each partition, BPDU bit stream data of each frame data is extracted from the sorted valid frame data to obtain the payload data corresponding to each partition. This step sorts and extracts the valid frame data in each partition in a parallel processing manner, which can further improve the preprocessing efficiency of satellite remote sensing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide further understanding of the present application and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present application, but do not constitute a limitation to the present application.

[0017] Figure 1 A flowchart of a remote sensing satellite data processing method based on Spark provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of another remote sensing satellite data processing method based on Spark provided in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of an application effect provided by an embodiment of the present application;

[0020] Figure 4 A schematic diagram of the structure of a remote sensing satellite data processing device based on Spark provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0023] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0024] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0025] The terms used in this application are only used to describe specific embodiments and are not intended to limit the present application.As used in this application, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0026] When the terms “comprising” and / or “made of…” are used in this application, it specifies the existence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0027] The embodiments described in the present application may be described with reference to plan views and / or cross-sectional views by means of ideal schematic diagrams of the present application. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used in this application have the same meaning as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this application.

[0029] Remote sensing satellites generally use dual-channel polarization multiplexing to transmit data to the ground. Since the recording rates of each payload data are different, the channels assigned to the recorded data during playback are not fixed, and there are slight differences in the time when the two channels start transmission, the data type (virtual channel) and sorting method (VCDU count) of the data frames transmitted in the two channels are not fixed, that is, "mixed data transmission". Under this transmission characteristic, a large amount of data processing work needs to be completed when processing data on the ground. In addition, the raw data of remote sensing satellites are usually stored in binary file format, and have the characteristics of high resolution and large capacity. Processing the raw data of large-scale remote sensing satellites requires a lot of computing and storage resources.

[0030] Traditional remote sensing data processing methods are limited by computing and storage resources and cannot meet the needs of processing large-scale remote sensing satellite raw data, and the processing efficiency is low. In order to improve the efficiency of remote sensing satellite data processing, this application provides a remote sensing satellite data processing method, device, electronic device and medium based on Spark (a data processing engine).

[0031] In the first aspect, an embodiment of the present application provides a remote sensing satellite data processing method based on Spark, which combines the distributed computing framework Spark with the remote sensing satellite raw data preprocessing in the hybrid transmission mode and uses Spark to preprocess the remote sensing satellite data to improve the processing efficiency of the remote sensing satellite data.

[0032] Figure 1 A flowchart of a remote sensing satellite data processing method based on Spark provided in an embodiment of the present application. The method is applied to a remote sensing satellite data processing device based on Spark, which is deployed in an electronic device. Figure 1 The remote sensing satellite data processing method based on Spark includes: the following steps S101 to S106.

[0033] Step S101: Based on the data reading configuration file in Spark, in the process of reading the original data collected by the remote sensing satellite, the original data is processed into blocks to obtain the first channel data and the second channel data after the blocks.

[0034] The data reading configuration file is a pre-configured file used to define the behavior of Spark reading data. The custom configuration of the data reading configuration file can be implemented through InputFormat (a component that parses the input file into a string) in Spark.

[0035] The original data is data in a mixed transmission mode, and therefore, the original data includes data of multiple channels, namely, first channel data and second channel data in the embodiment of the present application.

[0036] In one embodiment, a custom read configuration file is defined based on InputFormat in Spark, and the read configuration file includes a preset data volume, which is used to indicate the block size when reading raw data. For example, the preset data volume can be set to 512M or 256M.

[0037] In an embodiment of the present application, a data processing device performs block processing on the raw data collected by a remote sensing satellite based on a data reading configuration file in Spark, and obtains first channel data after block processing and second channel data after block processing, including: in the process of reading the raw data collected by a remote sensing satellite, the raw data is divided into blocks according to a preset data amount.

[0038] In this embodiment, the amount of original data is generally large, and the original data is divided into blocks according to the size of the preset data amount. Each block can be processed in parallel during the subsequent data processing process, thereby improving the processing efficiency of satellite remote sensing data.

[0039] In an embodiment of the present application, in the process of reading the original data collected by the remote sensing satellite based on Spark, the data processing device converts the original data from underlying (Hadoop Distributed File System, HDFS) data into resilient distributed (Resilient Distributed Dataset, RDD) data.

[0040] In one embodiment, the remote sensing satellite sends the collected raw data to the ground station. Before reading the raw data collected by the remote sensing satellite, the data processing device (distributed file system) obtains and stores the raw data collected by the remote sensing satellite from the ground station to facilitate subsequent data reading and data processing and analysis.

[0041] Step S102: performing channel merging processing on the blocked first channel data and the blocked second channel data to obtain merged block data.

[0042] The channel merging process is used to merge multi-channel data into data of one channel.

[0043] In one embodiment, a data processing device performs channel merging processing on the first channel data after blocking and the second channel data after blocking to obtain merged block data, including: based on the Union operator in Spark, performing channel merging processing on the first channel data after blocking and the second channel data after blocking to obtain merged block data; wherein the merged block data is data of one channel.

[0044] In an embodiment of the present application, a data processing device performs channel merging processing on the first channel data after blocking and the second channel data after blocking, and obtains the block data merged into one channel, so as to facilitate the subsequent overall processing of the data. It should be noted that the merging processing in this step only refers to channel merging. For example, the first channel data after blocking contains eight blocks, and the second channel data after blocking contains eight blocks. Then, after the first channel data after blocking and the second channel data after blocking are channel merged, the merged channel can be considered as the third channel, and the third channel data contains sixteen blocks. The data type before and after the channel merging remains unchanged, and the first channel data, the second channel data and the third channel data are all RDD data.

[0045] It should also be noted that each block data includes multiple AOS frames, and each AOS frame includes at least a virtual channel identifier, a file number, a frame count, and a Bridge Protocol Data Unit (BPDU) data. Among them, the virtual channel identifier is the identifier of each virtual channel. For example, the virtual channel identifier can be "100000" or "100001", and the "100000" can represent image compression data 1 (camera 1~4 channels), and the "100001" can represent image compression data 2 (camera 5~8 channels). The file number is used to represent the identifier of the file to which the AOS frame data belongs. The frame count is used to represent the number of frames of the file to which the AOS frame data belongs. The Bridge Protocol Data Unit contains the payload data in the frame data. For example, the Bridge Protocol Data Unit contains remote sensing satellite image data.

[0046] Step S103: Perform AOS frame synchronization processing on each block of data to obtain processed block data.

[0047] The AOS frame synchronization process is used to clear abnormal frame data in the block data, such as frame data with part of the information missing.

[0048] In one embodiment, the data processing device performs AOS frame synchronization processing on each block data to obtain processed block data, including: for each block data, calling the FlatMap function in Spark, and dividing the block data into independent frame data based on the synchronization header.

[0049] The synchronization header is a special header structure used to identify and synchronize data frames, which can ensure the correctness and integrity of data frames during transmission and avoid data loss or disorder.

[0050] In one embodiment, frame loss or bit error may occur during data transmission. Therefore, in the embodiment of the present application, the data processing device performs AOS frame synchronization processing on each block of data, and further includes: for each block of data, calling the FlatMap function in Spark to clear the AOS frames without synchronization headers.

[0051] In the embodiment of the present application, the data processing device performs AOS frame synchronization processing on each block of data, which can ensure the correctness and integrity of the data frame during transmission, avoid data loss or disorder, and facilitate data processing and analysis in subsequent processes.

[0052] Moreover, in the embodiment of the present application, the AOS frame synchronization processing of the block data is a parallel processing step, and the data processing device can perform distributed processing and simultaneously perform AOS frame synchronization processing on different block data to improve the processing efficiency of satellite remote sensing data.

[0053] Step S104: Based on the virtual channel identifier, obtain valid frame data from the processed block data.

[0054] The valid frame data is used to indicate that the virtual channel identifier is not the frame data of the filling frame. When the virtual channel identifier is an empty frame or a guide code, such as "001111", the virtual channel identifier is considered to be a filling frame.

[0055] In one embodiment, the data processing device obtains valid frame data from the processed block data based on the virtual channel identifier, including: filtering out data whose virtual channel identifier is a filler frame from the processed block data to obtain the valid frame data.

[0056] In one embodiment, the data processing device filters out data whose virtual channel identifier is a filler frame from the processed block data to obtain valid frame data, including the following steps a and b.

[0057] Step a: Call the filter operator in Spark to obtain different virtual channel identifiers from the processed block data.

[0058] Step b, extracting data whose virtual channel identifier is a filling frame, and the remaining data is valid frame data; or, extracting data whose virtual channel identifier is not a filling frame as the valid frame data.

[0059] Step S105: partition the valid frame data based on the virtual channel identifier to obtain valid frame data in different partitions.

[0060] The pseudo-channel identifier is an identifier of each virtual channel.

[0061] In the embodiment of the present application, based on the virtual channel identifier, the valid frame data is partitioned, which can effectively distinguish the frame data corresponding to different virtual channels, classify the valid frame data, and facilitate the subsequent processing of the frame data corresponding to different virtual channels. Taking the virtual channel identifier as "100000" or "100001" as an example, the frame data with the virtual channel identifier of "100000" is divided into the data of the first partition, and the frame data with the virtual channel identifier of "100001" is divided into the data of the second partition.

[0062] In an embodiment of the present application, the data processing system calls the repartition operator in Spark to partition the valid frame data based on the virtual channel identifier to obtain the valid frame data under different partitions.

[0063] Step S106: For the valid frame data in each partition, perform AOS frame sorting processing on the valid frame data to obtain sorted valid frame data in different partitions.

[0064] The AOS frame includes a file number and a frame count. The file number is used to indicate the identifier of the file to which the AOS frame data belongs. The frame count is used to indicate the number of frame data of the file to which the AOS frame data belongs.

[0065] In one embodiment, the data processing device performs AOS frame sorting processing on the valid frame data in each partition to obtain the valid frame data sorted in different partitions, including: for the valid frame data in each partition, performing AOS frame sorting processing on the valid frame data based on the file number and the frame count to obtain the valid frame data sorted in different partitions.

[0066] In an embodiment of the present application, the data processing system can call the SortWithinPartitions operator to perform AOS frame sorting on the valid frame data in each partition based on the file number and frame count to obtain sorted valid frame data in different partitions.

[0067] In this embodiment, in the step of performing AOS frame sorting processing on valid frame data based on file number and frame count, the AOS frames in the valid frame data are first sorted from small to large according to the file number, and then the AOS frames belonging to the same file number are sorted from small to large based on the frame count.

[0068] For example, in a certain partition, the file number contained in AOS frame data 1 is 001, and the frame count is 0001; the file number contained in AOS frame data 2 is 002, and the frame count is 0001; the file number contained in AOS frame data 3 is 001, and the frame count is 0002; the file number contained in AOS frame data 4 is 001, and the frame count is 0003… then in this partition, the sorted valid frame data are: AOS frame data 1, AOS frame data 3, AOS frame data 4, AOS frame data 2… It can be understood that the file number and frame count here are only exemplary and can be set according to actual applications.

[0069] It should be noted that, for the valid frame data in each partition, the step of performing AOS frame sorting processing on the valid frame data is a parallel step for multiple partitions, which can improve the processing efficiency of satellite remote sensing data.

[0070] Step S107: For each partition, extract the BPDU bit stream data of each frame data from the sorted valid frame data to obtain the effective load data corresponding to each partition.

[0071] The BPDU bit stream data includes the payload data in the frame data, for example, the BPDU bit stream data includes the image data collected by the remote sensing satellite. The effective payload data includes the BPDU bit stream data of all effective frame data in a partition.

[0072] In the embodiment of the present application, for each partition, the valid frame data is first sorted, and then the BPDU bit stream data of each frame data is extracted from the sorted valid frame data, so that the sorted BPDU bit stream data can be obtained, thereby combining it into the effective load data corresponding to the partition. Taking the BPDU bit stream data containing image data as an example, the sorted BPDU bit stream data can effectively represent the order of the image, which is convenient for subsequent analysis and processing of the image data.

[0073] In one embodiment, after obtaining the payload data corresponding to each partition, the payload data corresponding to each partition is stored in the distributed file system. It is understandable that the payload data corresponding to each partition includes image data of multiple files under one virtual channel.

[0074] The embodiment of the present application provides a remote sensing satellite data processing method based on Spark. In the method, firstly, based on the data reading configuration file in Spark, in the process of reading the original data collected by the remote sensing satellite, the original data is processed in blocks to obtain the first channel data and the second channel data after the blocks. This step divides the original file with a large amount of data into file blocks with a small amount of data for processing, which can reduce the data processing burden and improve the preprocessing efficiency of satellite remote sensing data; secondly, the first channel data after the blocks and the second channel data after the blocks are processed by channel merging to obtain the merged block data, and the AOS frame synchronization processing is performed on each block data to obtain the processed block data. This step processes the first channel data and the second channel data in a parallel processing manner. Performing AOS frame synchronization processing on each block data can further improve the preprocessing efficiency of satellite remote sensing data; then, based on the virtual channel identifier, obtaining valid frame data from the processed block data, partitioning the valid frame data based on the virtual channel identifier, obtaining valid frame data in different partitions, performing AOS frame sorting processing on the valid frame data for the valid frame data in each partition, obtaining sorted valid frame data in different partitions, extracting BPDU bit stream data of each frame data from the sorted valid frame data for each partition, obtaining the payload data corresponding to each partition, this step sorts and extracts the valid frame data in each partition in a parallel processing manner, which can further improve the preprocessing efficiency of satellite remote sensing data.

[0075] Figure 2 Another remote sensing satellite data processing method based on Spark is provided in an embodiment of the present application. The method is applied to a data processing device, and the data processing device is configured in an electronic device. Figure 2 As shown, after performing AOS frame synchronization processing on each block data to obtain the processed block data (the above step S103), the method further includes: the following steps S108-S109.

[0076] Step S108: Obtain invalid frame data from the processed block data based on the virtual channel identifier.

[0077] The invalid frame data is used to indicate that the virtual channel identifier is frame data of a filler frame. When the virtual channel identifier is an empty frame or a guide code, such as "001111", the virtual channel identifier is considered to be a filler frame.

[0078] In one embodiment, obtaining invalid frame data from processed block data based on the virtual channel identifier includes: the following steps c and d.

[0079] Step c: Call the filter operator in Spark to obtain different virtual channel identifiers from the processed block data.

[0080] Step d: extracting the data of the filling frame whose virtual channel identifier is used as the invalid frame data.

[0081] It should be noted that step S108 may be the same step as the aforementioned step S104. For example, the data of the filling frame whose virtual channel identifier is extracted is used as the invalid data, and the remaining data is the valid frame data.

[0082] Step S109: performing bit error rate analysis on the invalid frame data based on the Spark data analysis tool.

[0083] The invalid frame data may be input into a Spark data analysis tool to obtain a bit error rate output by the Spark data analysis tool.

[0084] In the embodiment of the present application, by statistics on the bit error rate, the bit error rate indicator can be used to evaluate the quality of the original data. For example, different bit error rates correspond to different original data qualities. The lower the bit error rate, the better the original data quality.

[0085] Figure 3 A schematic diagram of an application effect provided by an embodiment of the present application. In an implementation scenario, the original data collected by the remote sensing satellite used in the embodiment of the present application is: 5-circle original data of the GF7 satellite in a mixed satellite-to-ground transmission mode.

[0086] In this implementation scenario, the raw data collected by the remote sensing satellite is processed by a traditional stand-alone processing system, and the raw data collected by the remote sensing satellite is processed by the remote sensing satellite data processing method based on Spark provided in this application, and the processing time of the two is obtained. By comparing the processing time of the two, the effect of the remote sensing satellite data processing method based on Spark provided in the embodiment of this application is demonstrated.

[0087] Figure 3 Including satellite name, circle number, ground station ID, channel number, amount of data collected, processing time of each circle data by the data processing system of this application and processing time of each circle data by the traditional stand-alone system. Figure 3 It can be seen that the data processing time of the traditional stand-alone system is longer than the data processing time of the data processing system of the present application. Taking the original data with circle number 19017 as an example, the data processing time of the data processing system of the present application is 12 minutes (min), while the data processing time of the traditional stand-alone system is 20 minutes (min).

[0088] Judging from the experimental results, compared with the traditional single-machine processing system, this application has greatly improved the preprocessing efficiency of remote sensing satellite raw data in the mixed transmission mode, significantly reduced the difficulty of large-scale data processing, and enhanced the system's fault tolerance. In addition, this method has the characteristics of short development cycle and strong applicability. Therefore, this method can be applied to the processing of remote sensing satellite raw data in large-scale mixed transmission mode.

[0089] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0090] In a second aspect, an embodiment of the present application provides a remote sensing satellite data processing device based on Spark.

[0091] Figure 4 The present invention provides a schematic diagram of a remote sensing satellite data processing device based on Spark. Figure 4 As shown, the data processing device includes: a reading module 41, a merging module 42, a frame synchronization module 43, a filtering module 44, a classification module 45, a sorting module 46 and an extraction module 47.

[0092] Among them, the reading module 41 is used to read the data configuration file in Spark. In the process of reading the original data collected by the remote sensing satellite, the original data is processed in blocks to obtain the first channel data and the second channel data after blocking; the original data includes the first channel data and the second channel data.

[0093] In the embodiment of the present application, the reading module 41 is also used to divide the original data into blocks according to the size of preset data volume during the process of reading the original data collected by the remote sensing satellite.

[0094] The merging module 42 is used to perform channel merging processing on the blocked first channel data and the blocked second channel data to obtain merged block data; the block data includes multiple AOS frames, and the AOS frame includes a virtual channel identifier.

[0095] In an embodiment of the present application, the merging module 42 is also used to perform channel merging processing on the first channel data after block and the second channel data after block based on the Union operator in Spark to obtain merged block data; wherein the merged block data is the data of one channel.

[0096] The frame synchronization module 43 is used to perform AOS frame synchronization processing on each block data to obtain processed block data.

[0097] In the embodiment of the present application, the frame synchronization module 43 is also used to call the FlatMap function in Spark for each block data, and divide the block data into independent frame data based on the synchronization header.

[0098] The filtering module 44 is configured to obtain valid frame data from the processed block data based on the virtual channel identifier.

[0099] In the embodiment of the present application, the filtering module 44 is further configured to filter out data whose virtual channel identifier is a filler frame from the processed block data to obtain valid frame data.

[0100] The classification module 45 is used to partition the valid frame data based on the virtual channel identifier to obtain the valid frame data in different partitions.

[0101] The sorting module 46 is used to perform AOS frame sorting processing on the valid frame data in each partition to obtain sorted valid frame data in different partitions.

[0102] In the embodiment of the present application, the sorting module 46 is also used to perform AOS frame sorting processing on the valid frame data in each partition based on the file number and the frame count to obtain sorted valid frame data in different partitions.

[0103] The extraction module 47 is used to extract the BPDU bit stream data of each frame data from the sorted valid frame data for each partition, and obtain the effective load data corresponding to each partition.

[0104] In one embodiment, the data processing device further includes: a quality analysis module. The aforementioned filtering module is further used to obtain invalid frame data from the processed block data based on the virtual channel identifier; after the filtering module obtains the invalid data, the quality analysis module is used to perform bit error rate analysis on the invalid frame data based on the Spark data analysis tool.

[0105] It should be clear that the present application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0106] In the remote sensing satellite data processing device based on Spark provided in the embodiment of the present application, the reading module 41 is used to read the configuration file based on the data in Spark. In the process of reading the original data collected by the remote sensing satellite, the original data is processed in blocks to obtain the first channel data after block and the second channel data after block. This process divides the original file with a large amount of data into file blocks with a small amount of data for processing, which can reduce the data processing burden and improve the preprocessing efficiency of satellite remote sensing data; secondly, the merging module 42 is used to perform channel merging processing on the first channel data after block and the second channel data after block to obtain the merged block data, and the frame synchronization module 43 is used to perform AOS frame synchronization processing on each block data to obtain the processed block data. This process processes each block data in a parallel processing manner. The AOS frame synchronization processing is performed on the data, which can further improve the preprocessing efficiency of satellite remote sensing data; the filtering module 44 is used to obtain valid frame data from the processed block data based on the virtual channel identifier, and the classification module is used to partition the valid frame data based on the virtual channel identifier to obtain the valid frame data in different partitions. The sorting module is used to perform AOS frame sorting processing on the valid frame data in each partition to obtain the sorted valid frame data in different partitions. The extraction module 47 is used to extract the BPDU bit stream data of each frame data from the sorted valid frame data for each partition to obtain the payload data corresponding to each partition. This process sorts and extracts the valid frame data in each partition in a parallel processing manner, which can further improve the preprocessing efficiency of satellite remote sensing data.

[0107] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 The electronic device includes: one or more processors 501; a memory 502, on which one or more programs are stored. When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned remote sensing satellite data processing methods based on Spark; one or more I / O interfaces 503, connected between the processor and the memory, and configured to realize information interaction between the processor and the memory.

[0108] Among them, the processor 501 is a device with data processing capabilities, including but not limited to a central processing unit (CPU) and the like; the memory 502 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 503 is connected between the processor 501 and the memory 502, and can realize information interaction between the processor 501 and the memory 502, including but not limited to a data bus (Bus) and the like.

[0109] In some embodiments, the processor 501 , the memory 502 , and the I / O interface 503 are connected to each other through a bus, and further connected to other components of the computing device.

[0110] This embodiment also provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the remote sensing satellite data processing method based on Spark provided in this embodiment is implemented. To avoid repeated description, the specific steps of the remote sensing satellite data processing method based on Spark are not repeated here.

[0111] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods invented above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0112] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0113] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.

[0114] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A remote sensing satellite data processing method based on Spark data analysis tools, It is characterized in that The method comprises: Based on the data reading configuration file in Spark, in the process of reading the raw data collected by the remote sensing satellite, the raw data is processed into blocks to obtain the first channel data and the second channel data after the blocks; the raw data includes the first channel data and the second channel data; Perform channel merging processing on the blocked first channel data and the blocked second channel data to obtain merged block data; the block data includes a plurality of AOS frames, and the AOS frames include a virtual channel identifier; Perform AOS frame synchronization processing on each block data to obtain processed block data; Based on the virtual channel identifier, obtaining valid frame data from the processed block data; Partitioning the valid frame data based on the virtual channel identifier to obtain valid frame data in different partitions; For the valid frame data in each partition, perform AOS frame sorting processing on the valid frame data to obtain sorted valid frame data in different partitions; For each partition, the BPDU bit stream data of each frame data is extracted from the sorted valid frame data to obtain the effective load data corresponding to each partition.

2. The method according to claim 1, It is characterized in that The reading configuration file includes a preset data amount; the data reading configuration file based on Spark, in the process of reading the original data collected by the remote sensing satellite, performs block processing on the original data to obtain the first channel data after block processing and the second channel data after block processing, including: In the process of reading the original data collected by the remote sensing satellite, the original data is divided into blocks according to the size of the preset data volume.

3. The method according to claim 2, It is characterized in that The step of performing channel merging processing on the first channel data after block and the second channel data after block to obtain merged block data includes: Based on the Union operator in the Spark, the first channel data after block and the second channel data after block are subjected to channel merging processing to obtain merged block data; wherein the merged block data is data of one channel.

4. The method according to claim 3, It is characterized in that The performing AOS frame synchronization processing on each block data to obtain processed block data includes: For each block of data, the FlatMap function in Spark is called to split the block data into independent frame data based on the synchronization header.

5. The method according to claim 4, It is characterized in that The acquiring valid frame data from the processed block data based on the virtual channel identifier comprises: The data whose virtual channel identifier is a filler frame is filtered out from the processed block data to obtain the valid frame data.

6. The method according to claim 5, It is characterized in that The AOS frame includes a file number and a frame count; and for the valid frame data in each partition, performing AOS frame sorting processing on the valid frame data to obtain sorted valid frame data in different partitions, including: For the valid frame data in each partition, AOS frame sorting processing is performed on the valid frame data based on the file number and the frame count to obtain sorted valid frame data in different partitions.

7. The method according to claim 1, It is characterized in that The method further comprises: Based on the virtual channel identifier, obtaining invalid frame data from the processed block data; The invalid frame data is subjected to bit error rate analysis based on the Spark data analysis tool.

8. A remote sensing satellite data processing device based on Spark data analysis tools, It is characterized in that The device comprises: A reading module is used for performing block processing on the raw data collected by the remote sensing satellite based on the data reading configuration file in Spark, so as to obtain the first channel data and the second channel data after block processing; the raw data includes the first channel data and the second channel data; A merging module, used for performing channel merging processing on the first channel data after block and the second channel data after block to obtain merged block data; the block data includes a plurality of AOS frames, and the AOS frame includes a virtual channel identifier; A frame synchronization module is used to perform AOS frame synchronization processing on each block data to obtain processed block data; A filtering module, configured to obtain valid frame data from the processed block data based on the virtual channel identifier; A classification module, used for partitioning the valid frame data based on the virtual channel identifier to obtain valid frame data in different partitions; A sorting module, for performing AOS frame sorting processing on the valid frame data in each partition to obtain sorted valid frame data in different partitions; and The extraction module is used to extract the BPDU bit stream data of each frame data from the sorted valid frame data for each partition, and obtain the effective load data corresponding to each partition.

9. An electronic device, It is characterized in that include: one or more processors; A storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7; One or more I / O interfaces are connected between the processor and the memory and are configured to implement information interaction between the processor and the memory.

10. A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.