Parallel processing method and device for multi-stage data transmission and data calculation

Abstract

The invention relates to a parallel processing method and device for multi-stage data transmission and data calculation of big data, and comprises an interactive concurrent method for multi-stage data transmission and data calculation. The method and the device comprise a system structure, a parallel mechanism, a system function and a system module. In order to solve the timeliness problem of big data multi-access data transmission and multi-task calculation, an application mode of three multi-level site remote or short-range or local remote and local short-range data communication systems and a super computer cluster of independently distributed calculation resources is established. According to the interactive concurrent method, a data block forwarding calculation mechanism is established, big data multi-level station concurrent transmission can be completed by using relatively less time, a large amount of calculation power is collected to complete a big data calculation task, and very high data transmission and data calculation benefits are achieved. A parallel mechanism and an application mode are provided for big data multi-access transmission and multi-task calculation, and different types of requirements of super computing power of big data multi-access data communication and big data multi-task calculation are efficiently met.

Description

Technical Field

[0001] The present invention relates to a parallel processing method and device for multi-level data transmission and data calculation, and belongs to the technical field of big data, computer network, data communication and data calculation. Technical Background

[0002] Current communication technology and computing technology are trying their best to cater to and promote the human need for rapid information processing. The information data collected and counted manually, the business data generated by the technology platform, and the information data obtained by the collector are all growing explosively. For example, mobile phones collect voice, cameras collect images, recorders and cameras collect audio and video, radars detect targets, satellites detect the ground, and spacecraft detect the universe. Various collectors are collecting information all the time, and information data distribution and sharing require transmission and calculation. According to the Shannon principle, the speed of information transmission is related to the channel spectrum bandwidth. In order to improve the timeliness of data transmission, both wired and wireless channels must rely on abundant spectrum resources. At present, if data transmission is carried out on the ground, optical fiber is mostly used, and if satellite communication is used, microwave channels are mostly rented. In order to improve the data processing speed, it is necessary to equip a computer with strong data processing capabilities. In order to improve the data calculation speed, a faster CPU (central processing unit) or GPU (graphics processor) and memory components are needed. Increasing memory capacity is conducive to big data processing. Improving the speed of peripheral data reading and storage requires the use of external memory and its interface circuits with faster read and write access, such as solid-state drives and their supporting interfaces. Currently, parallel transmission and parallel computing systems have achieved great success. In order to further improve data transmission and computing efficiency, more costly parallel transmission and parallel computing systems can be used.

[0003] The multi-task processing of business data and the hierarchical setting of business organizations require multi-level data transmission and data calculation. Whether data needs multi-level transmission depends on many factors, such as the need for the front end to process the received data and then forward it; the cost of the ground dedicated line is related to the distance, or a sub-center is set up for forwarding; the cost of satellite receiving equipment is high, and after receiving the data sent by the satellite, it is transferred to the ground dedicated line for further transmission; and so on. Multi-level data transmission mode, such as the multi-level transmission of satellite remote sensing data, the ground station receives the data sent by the satellite, and after processing by the data center, the ground station uploads the satellite for forwarding; for example, the multi-level transmission of the provincial ( / ministerial) satellite remote sensing system, the ground station receives the data sent by the satellite, and after processing by the data center, it is forwarded to the department-level, prefecture-level, county-level and other departments through the ground dedicated line step by step. Before data application, data not only needs to be distributed at multiple levels, but also needs to be calculated for multiple tasks. For example, satellite remote sensing data often needs to undergo a series of data preprocessing, including various data corrections in the time domain, spatial domain, frequency domain, and various ground object inversions, to continuously provide various parameter data for the earth simulation system. Earlier, multi-level data transmission was transmitted by independent stations, and after receiving the files from the previous station, it was independently transmitted to the next station. In order to improve the timeliness of data transmission, multi-level data transmission mostly adopts inter-level memory synchronous forwarding to complete data transmission in real time. The previous station has data calculation tasks (such as data preprocessing), and can use memory to save received data files. In order to avoid wasting time on low-speed external memory, high-speed memory, CPU or GPU can be used to complete the corresponding data calculation, and then forwarded to the next station. Currently, there are data calculation tasks in multi-level data transmission, which requires waiting for the data file to be received. It is still an independent transmission mode between two stations. The waiting time for the forwarding process and the receiving process is relatively long. The data transmission time is not used to perform data calculation at the same time, and the timeliness and comprehensive benefits of the multi-level station data transmission and data calculation links are not considered.

[0004] Although data acquisition and processing are getting faster and faster, the current timeliness of multi-level data transmission and data calculation is still far from people's needs for industry applications. Big data business systems need to cope with the resource pressure of data transmission and data calculation caused by centralized processing of business data from various places and congestion during peak hours. For big data in the fields of military, meteorology, land, ocean, environment, etc., facing the poor timeliness of local and global simulation systems, there is an urgent need for technical innovation and upgrading in a large number of data processing matters such as data transmission in different places, data time domain, spatial domain and frequency domain correction, data inversion of various parameter data, and system simulation operations. For example, military target monitoring, earth resource monitoring, and disaster monitoring and prediction, for local or even global data collection, transmission and calculation, there will be extremely high expectations for the acquisition and processing speed of collected data, and even the corresponding processing capabilities will be improved at any cost. Based on previous related research, it is more urgent to build an efficient multi-level data transmission and data calculation parallel processing application mode, which is different from the big data parallel transmission mode and the big data parallel calculation mode, but improves the transmission and calculation efficiency of big data together, conforms to the national strategy, and has an immeasurable role in promoting the national economy and social benefits. Summary of the invention

[0005] The present invention aims at data sources that require multi-address data transmission and multi-task data calculation, establishes an interactive fusion mechanism for data transmission and data calculation to overcome the closed nature of the data transmission process, builds a super application mode of data transmission and data calculation, improves the transaction efficiency of data transmission and data calculation, and provides an efficient multi-level data transmission and data calculation parallel processing method and device.

[0006] The present invention summarizes the parallel processing method and device for multi-level data transmission and data calculation developed from the perspective of system structure, parallel mechanism, system function, and system module as follows:

[0007] The system structure includes: it is composed of multiple levels of computer sites ( / nodes); inter-station communication adopts long-range or short-range computer communication networks, or long-range or short-range non-computer communication networks; sites and inter-station long-range or short-range communications constitute a multi-level site long-range or short-range data communication system and an independently distributed supercomputer cluster; the multiple levels of sites and their data flows are a multi-level tree-branch static structure; site data processing is divided into modules such as data transmission and data calculation; and a parallel processing dynamic mode of data transmission and calculation is performed in an interleaved manner at each level.

[0008] The parallel mechanism includes: data block provides an opportunity for concurrent transmission and calculation of large data; the minimum data processing block is divided into two basic forms: a group of main data, no or no auxiliary data needs to be transmitted; a group of auxiliary data and its corresponding group of main data; data block transmission and calculation station by station, forming an interleaved transmission calculation mechanism to complete the whole data transmission and calculation task. The mathematical model of parallel transmission calculation is also stated.

[0009] The system functions include: each station is responsible for data transmission and data calculation services; multi-level station data transmission and total task calculation are completed by staggered data transmission and calculation at each level; the inter-station communication method determines the operation, function and benefit of the system, showing different structural model functions; and the long-range or short-range communication between stations determines the combination of multi-level station long-range data communication, short-range data communication and independently distributed supercomputing.

[0010] The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules.

[0011] The system presets include: predetermining the program process form, communication transceiver mode, signal transceiver matching and data exchange mode, pre-compilation of calculation task algorithms, and implementation of standardized file name naming.

[0012] The transmission module is divided into a low-level version, an intermediate version and an advanced version, and the version can be selected as needed. The low-level transmission module includes a sending submodule and a receiving submodule. The sending submodule includes: formulating a data sending start mechanism, reading the sending data according to the minimum data processing block, framing or multi-frame data packaging for frame transmission according to the minimum data processing block, continuously sending subsequent data of the data task, cleaning up the main and auxiliary data (or only the main data) that have been sent, etc. The receiving submodule includes: completely receiving each frame of data, unloading the frame and unpacking according to the flags of the frame header and the packet header, saving the received main and auxiliary data (or only the main data) by block according to the flag, and continuously receiving the subsequent data of the data task. The intermediate version transmission module includes a sending submodule and a receiving submodule. The sending submodule includes: formulating a data sending start mechanism, reading the sending data according to the minimum data processing block, providing exchange data to the interface according to the frame plan, continuously sending subsequent data of the data task, cleaning up the main and auxiliary data (or only the main data) that have been sent, etc. The receiving submodule includes: reading data according to the interface specification, saving the received primary and secondary data (or only the main data) by block according to the flag, and continuously receiving the subsequent data of the data task. The advanced version transmission module includes: formulating the data sending start mechanism, reading the sending data from the current station by block, writing it to the storage place of the "receive" exchange data of the lower station, continuously forwarding the subsequent data of the data task, and cleaning up the sent primary and secondary data (or only the main data).

[0013] The calculation module includes: adjusting the auxiliary data needed, formulating a data calculation startup mechanism, calling the collected parameter data and the "received" auxiliary data, calculating the unprocessed main data in blocks according to the calculation task processing algorithm, continuously calculating the subsequent data of the data task, adjusting the sent auxiliary data, and saving the main and auxiliary data (or only the main data) that have been sent (or read), received (or written), and calculated.

[0014] Compared with the prior art, the invention has the following innovations and benefits: ⑴ It establishes a parallel mechanism for data transmission and data calculation, breaks the closedness of the data transmission process, establishes a block forwarding mechanism for data transmission, provides a convenient door for parallel processing of data transmission and data calculation, provides intensive forwarding opportunities for concurrent transmission of multi-level station data, and provides the time gap required for data processing for data calculation. ⑵ By utilizing the parallel mechanism for data transmission and data calculation, it takes relatively little time to complete the concurrent transmission of big data at multi-level stations, and gathers a large amount of computing power to complete big data computing tasks, which has very high data transmission and data computing benefits. ⑶ By utilizing transmission time and computing resources, three implementation and application modes are constructed using different communication methods: ① Two-in-one mode, multi-level site remote data communication system and supercomputer cluster with independent distributed computing resources, providing high-efficiency data communication and data computing with super computing power at the same time; ② Strong computing mode, short-range ultra-high-efficiency data communication system, supercomputer cluster with independent distributed computing resources, and data computing with special super computing power; ③ Comprehensive mode, multi-level site local remote and local short-range data communication system, supercomputer cluster with independent distributed computing resources, providing high-efficiency data communication and data computing with high-quality super computing power at the same time. (4) The present invention provides a parallel mechanism and its application mode for big data multi-access transmission and multi-task computing, which efficiently solves the different types of requirements for super computing power of big data multi-access data communication and big data multi-task computing.

[0015] Other features and advantages of the present invention will become more apparent after detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Overall principle block diagram;

[0017] Figure 2 System structure diagram;

[0018] Figure 3 Parallel mechanism block diagram;

[0019] Figure 4 System functional block diagram;

[0020] Figure 5 System module block diagram;

[0021] Figure 6 Flowchart of low-level transmission module;

[0022] Figure 7 Intermediate version of the transmission module flow chart;

[0023] Figure 8 Advanced version transmission module flow chart;

[0024] Fig. 9 Flowchart of the computational module. DETAILED DESCRIPTION

[0025] The following is a further description from the overall to the specific implementation methods in conjunction with the accompanying drawings. The data is first explained. On the basis of summarizing the overall implementation plan, 9 related statements are explained, and the detailed scheme of the present invention is gradually explained in three application modes determined by the inter-station communication method. It should be noted that the description of these implementation methods is helpful to help understand the present invention, but some specific descriptions do not constitute a limitation of the present invention.

[0026] 1 Overall method and device

[0027] The data includes auxiliary data and main data, or only main data, which depends on the specific data source and the data processing tasks of transmission and calculation. Auxiliary data is parameter data for data calculation of main data. The present invention mainly takes main data as big data, which requires multi-address data transmission and multi-task data calculation and other processing. Data is divided into real-time data and non-real-time data (or historical data) based on timeliness. It is mainly various ground-based, air-based and space-based collection data, business data generated by technical platforms, and manually collected and statistical information data, such as satellite and radar detection data, or production, operation and other industry data. The main data of satellite detection is remote sensing data, and auxiliary data is telemetry data; there are many types of satellite detectors, including CCD (charge coupled device), WFV (wide field of view), MERSI (medium resolution spectral imager) and other remote sensing data.

[0028] The overall method and device of the present invention includes four aspects: system structure, parallel mechanism, system function, and system module. Figure 1 .

[0029] 1.1 System Structure

[0030] See also Figure 2 , the system structure includes:

[0031] ① Computer site composition: A multi-level site ( / node) consists of a first-level site, a second-level site, ..., an N-th-level site (N is a positive integer, and N ≥ 2), and each site consists of one or more computers.

[0032] ②Inter-station communication technology system: Inter-station communication uses a long-distance computer communication network (wide area network), or a long-distance non-computer communication network (non-network version of long-distance data communication system), a short-distance computer communication network (local area network or metropolitan area network), or a short-distance non-computer communication network (non-network version of short-distance data communication system). Communication lines are wireless and wired, and lines of various specifications can be selected and used as needed.

[0033] ③ Overall structural functional system: long-distance communication between sites and stations constitutes a multi-level site long-distance data communication system and a supercomputer cluster with independent distributed computing resources; or short-distance communication between sites and stations constitutes a multi-level site short-distance data communication system and a supercomputer cluster with independent distributed computing resources; or long-distance and short-distance communication between sites and stations constitutes a multi-level site local long-distance and local short-distance data communication system and a supercomputer cluster with independent distributed computing resources.

[0034] ④Tree-branch static structure: Multi-level sites and their data flows are expressed as a multi-level tree-branch structure, and the multi-level single-branch tree structure is a special form.

[0035] ⑤ Site data processing module: The site system module is divided into data sending (or reading), receiving (or writing), calculation and other modules.

[0036] ⑥ Parallel processing dynamic mode: Data is transmitted concurrently from the 1st-level station to the Nth-level station, and the data processing tasks assigned to each station are calculated in an interleaved manner.

[0037] 1.2 Parallel Mechanism

[0038] See also Figure 3 , the parallel mechanism includes:

[0039] ⑴ Minimum data processing block unit. Design data exchange and data sending (or reading), receiving (or writing), calculation and other data processing services at all levels of stations, all of which are processed in the specified format with the minimum data processing block as the unit. The entire scene [ / frame / track] data is divided into blocks using the minimum data processing block as the processing unit, and the main and auxiliary data are transmitted synchronously or closely, providing opportunities for concurrent data transmission and calculation, relying on the parallel processing of data transmission and data calculation to achieve high efficiency of data transmission and data calculation.

[0040] ⑵ Minimum data processing block setting. The minimum data processing block includes auxiliary data and main data, or only main data. ① If there is no corresponding relationship between the auxiliary data and the main data, the determination of the minimum data processing block only considers the factors affecting the data calculation efficiency, and the minimum data processing block is only a group of main data; ② If there is a corresponding relationship between the auxiliary data and the main data, the auxiliary data does not need to be transmitted, and the minimum data processing block is a group of main data corresponding to the auxiliary data; ③ If the auxiliary data needs to be transmitted, the minimum data processing block is a group of auxiliary data and its corresponding group of main data. The minimum data processing block has a relatively small amount of data, which is more conducive to reducing latency and improving the timeliness of overall data transmission.

[0041] ⑶ Data transmission and calculation interaction mechanism. Taking the smallest data processing block as the unit, the data of the entire scene [ / frame / track] is staggered and continuously transmitted and calculated, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of stations. ① Data block transmission: Each scene [ / frame / track] data is divided into M blocks and transmitted block by block, and the transmission task within the station is completed block by block; in the N-station transmission, each block of data is transmitted station by station. ② Data block calculation: The data calculation task of each scene [ / frame / track] is assigned to each station according to the plan, and the assigned calculation task is processed by block, and the data calculation task is completed station by station. ③ Interlaced transmission calculation: Each scene [ / frame / track] data is transmitted in blocks to each station, and each station calculates the data by block. The transmission and calculation tasks of the main data and auxiliary data (or only the main data) are completed in an interlaced transmission and calculation manner at each station.

[0042] ⑷ Parallel transmission calculation mathematical model. ① Establish a mathematical model. The data is divided into M blocks, and N stations are used for parallel transmission calculation. According to the time and space relationship between the transmission calculation between stations and blocks, the time required for parallel transmission calculation of the whole data is obtained.

[0043]

[0044] Where: N, M are integers greater than or equal to 2, 1≤i≤N, 1≤j≤M; T Ci T is the time consumed by the block data station channel; Di,j T is the time taken to send and receive block data. Di,j =T DSi,j +T DRi,j , T DSi,j T is the time taken to send block data. DRi,j T is the time taken to receive block data; Oi,j Calculate the timeout for the block data, T Oi,j =T SCi,j –T Di,j , T SCi,j =T C1i,j +T C2i,j , T SCi,j Calculate the cache time for block data, T C1i,j is the time taken to calculate the block data, T C2i,j is the block data cache time, and T is specified SCi,j -T Di,j When ≤0, T Oi,j =0; the right item 1 of formula (1) is the total time consumption of the channel between stations at all levels, the right item 2 is the total time consumption of sending and receiving the same block data at all levels, the right item 3 is the total timeout of the calculation of the same block data at all levels, and the right item 4 is the total delay of the remaining block data except the first block; combine the same items to get

[0045]

[0046] The amount of data in each block is the same. If the same-level stations use the same computer to process each block of data, then

[0047]

[0048] The speed is related to the size of the block data. This means that reducing the amount of block data will directly reduce the time it takes to send and receive data at each level. Although the size of each block of data is the same, the equipment used at each level of the station is different, and the data transmission speed may be different. is the total timeout for calculating the same block of data at each station, which is related to the calculation cache speed and data transmission speed of the data task at each station. Oi,1 ≤0, indicating that the data task calculation cache does not delay data transmission. Oi,1 >0, indicating that the data task calculation cache causes delays in the data transmission link. This indicates that improving computer resources will directly reduce data calculation timeouts. is the total delay of the remaining block data except the first block, which is related to the average data transmission speed of each station and the amount of block data. The amount of each block data is the same, and the data transmission speed of the same station is the same. In the simplified case, T NM It can be summarized as the sum of the total channel time consumed by stations at all levels for one block of data, the total time spent on sending and receiving one block of data at all levels, the total calculation timeout of one block of data at all levels, and the total delay in sending and receiving each block of data except the first block at one station.

[0049] ③ Rich data transmission and calculation benefits. The entire data transmission and calculation process is a parallel state of cross-concurrent processing, and the data transmission and calculation benefits are evaluated according to the entire process. This mathematical model reflects the timeliness of data transmission and data calculation in the entire process. From the mathematical model, the overall data transmission and calculation benefits are the same for each station channel delay, which has not been reduced. The total (N×M) blocks only use (N+M-1) blocks of data for sending and receiving, and only use N blocks of data for timeout. The results show that using data transmission time and data calculation at the same time can not only efficiently increase the data calculation benefits, but also efficiently increase the data transmission benefits of multi-level stations; reducing the amount of block data will correspondingly improve the data transmission benefits of multi-level stations.

[0050] 1.3 System Functions

[0051] See also Figure 4 , the system functions include:

[0052] ① In-station business classification function: stations at all levels are responsible for sending (or reading), receiving (or writing), calculating and saving primary and secondary data (or only main data).

[0053] ② Parallel processing dynamic function: Data flows from the 1st level station to the Nth level station. Each level of the station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation of each station, and then completes the multi-level station data transmission and the computing power to complete the total task calculation.

[0054] ③ Communication mode construction function: The inter-station communication mode determines the application mode under the communication technology framework, which is related to the operation of each part of the system, and plays a role and generates benefits; the long-distance or short-distance or both long-distance and short-distance communication between stations and stations respectively demonstrates the functions of data transmission and data calculation parallel processing of computer resources independently distributed in multi-level sites, combined mode, strong computing mode and comprehensive mode.

[0055] ④ Overall structural mode function: two-in-one mode, each station concurrently executes remote data transmission, and processes big data with concurrent computing dispatch tasks and super computing power of superimposed distributed computing resources; or strong computing mode, each station concurrently executes short-range data distribution, and processes big data with concurrent computing dispatch tasks and super computing power of superimposed distributed computing resources; or comprehensive mode, each station concurrently executes part of remote data transmission and part of short-range data distribution, and processes big data with concurrent computing dispatch tasks and super computing power of superimposed distributed computing resources.

[0056] 1.4 System Module

[0057] See also Figure 5 The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules.

[0058] 1.4.1 System Presets

[0059] The system presets include:

[0060] ① Determine the program process form: Data processing must first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading.

[0061] ② Determine the communication sending and receiving methods: Data communication must first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods used between stations, and require that the data sending and receiving communication methods are matched.

[0062] ③ Determine the signal transmission and reception matching: Signal transmission and reception must first determine the signal quality and matching of the specific communication line, and properly arrange the signal low-frequency amplifier, medium-frequency amplifier and power amplifier.

[0063] ④ Determine the data exchange method: Data exchange requires first determining the data exchange method between data sending (or reading), receiving (or writing) and calculation. You can choose arrays, files, variables, or memory, etc. as the storage location for exchanged data.

[0064] ⑤ Prepare computing task algorithms: Different computing tasks are assigned to sites at each level, and the computing task processing algorithms for sites at each level must be prepared first.

[0065] ⑥ Standardized file naming: Data file names involve data call, storage, exchange, etc., and should be named according to rules or regulations.

[0066] 1.4.2 Transmission Module

[0067] The transmission module is divided into low-level, medium-level and high-level versions according to the degree of transparency of the data transmission process to the user. You can choose the version as needed.

[0068] (I) Low-level transmission module

[0069] See also Figure 6 , design the data frame structure, including the entire process of data sending and receiving including data reading, frame packing, data transmission, frame unloading and data saving.

[0070] 1) Sending submodule

[0071] According to the system preset conditions, the main and auxiliary data (or only the main data) are sent in a non-network communication environment.

[0072] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0073] ② Read the sending data in blocks: Pack the main and auxiliary data (or only the main data) into frames according to the minimum data processing block, read the main data in the "send" storage location where the data calculation has been completed according to the format, and if the auxiliary data needs to be sent, also read the auxiliary data in the corresponding "send" storage location.

[0074] ③ Data packaging frame transmission: The packaging framer or sending software processes data synchronously or immediately after the transmission according to the minimum data processing block. When the main data is transmitted alone, the main data is independently packaged into one or more data packets according to the packet format; or when the main and auxiliary data are transmitted in the same frame, the main and auxiliary data are formed into one data packet according to the packet format; or when the main and auxiliary data are transmitted in separate frames, the auxiliary and main data are formed into one or several independent data packets according to the packet format; or the minimum data processing block is only about a fraction of the frame data area, and the block main and auxiliary data are used as data sub-packets, and the data packet is composed of several sub-packets. Frame the data packet and channel coding according to the frame format, and transmit the minimum data processing block data synchronously or immediately.

[0075] ④Continuously send subsequent data: Continue to read and package the subsequent data of the main and auxiliary data (or only the main data) to complete the sending of the entire scene [ / frame / track] data.

[0076] ⑤ Clean up sent data: promptly delete the auxiliary data and main data (or only the main data) that have been sent, calculated, and saved in the exchange data storage location to release computer resources.

[0077] 2) Receiving submodule

[0078] According to the system preset conditions, the main and auxiliary data (or only the main data) are received in a non-network communication environment.

[0079] ① Receive full frame data: Completely receive each frame of data sent by the non-network system.

[0080] ② Receive data and unpack it: The unpacker or receiving software unpacks each frame of data received according to the flags of the frame header and the packet header, distinguishes the main and auxiliary data according to the data packet flag, and memorizes the flag of the smallest data processing block that belongs to the same block.

[0081] ③ Save received data in blocks: store primary and auxiliary data (or only main data) in minimum data processing blocks according to the flag, and save auxiliary data and main data (or only main data) in corresponding data storage locations.

[0082] ④Continue to receive subsequent data: Continue to receive, unpack and save the subsequent data of the main and auxiliary data (or only the main data) to complete the reception of the entire scene [ / frame / track] data.

[0083] (II) Intermediate version transmission module

[0084] See also Figure 7 , the user end uses the network communication protocol in the network environment, or the general data communication system, to exchange the main and auxiliary data (or only the main data) according to the specifications through the hardware or software interface to complete the data sending and receiving.

[0085] 1) Sending submodule

[0086] According to the system preset conditions, the main and auxiliary data (or only the main data) are sent in a network or non-network communication environment.

[0087] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0088] ② Read the sending data in blocks: Pack the main and auxiliary data (or only the main data) into frames according to the minimum data processing block, read the main data in the "send" storage location where the data calculation has been completed according to the format, and if the auxiliary data needs to be sent, also read the auxiliary data in the corresponding "send" storage location.

[0089] ③ Provide exchange data to the interface: The user end writes the exchange data to the hardware or software interface specification, and the network communication protocol or general data communication machine is responsible for data packaging, framing and transmission.

[0090] ④Continuously send subsequent data: Continuously read the subsequent data of the main and auxiliary data (or only the main data), provide an interface to exchange data, and complete the sending of the entire scene [ / frame / track] data.

[0091] ⑤ Clean up sent data: promptly delete the auxiliary data and main data (or only the main data) that have been sent, calculated, and saved in the exchange data storage location to release computer resources.

[0092] 2) Receiving submodule

[0093] According to the system preset conditions, the main and auxiliary data (or only the main data) are received in a network or non-network communication environment.

[0094] ① Standardized reading of interface data: The user end reads the frame-unpacking data transmitted to the interface by the network communication protocol or the general data communication machine.

[0095] ② Save received data in blocks: store primary and auxiliary data (or only main data) in minimum data processing blocks according to the flag, and save auxiliary data and main data (or only main data) in corresponding data storage locations.

[0096] ③Continue to receive subsequent data: Continue to receive and save subsequent data of the main and auxiliary data (or only the main data) to complete the reception of the entire scene [ / frame / track] data.

[0097] (III) Advanced transmission module

[0098] See also Figure 8 The advanced platform can build transparent access to different addresses in a network environment, and directly read and write data like a local machine in an advanced platform environment.

[0099] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0100] ② Reading of receiving and sending data: The receiving and sending software uses the smallest data processing block as the processing unit, and reads the main data in the "send" storage location where the data calculation has been completed at this level of station according to the format. If auxiliary data needs to be sent, the auxiliary data corresponding to the "send" storage location is also read.

[0101] ③ Writing of receiving and sending data: Auxiliary data and main data are written separately in the format to the corresponding storage location of "receiving" of the lower-level station, or only the main data is written in the format to the storage location of "receiving".

[0102] ④Continuously forward subsequent data: Continuously forward the subsequent blocks of auxiliary data and main data (or only main data) of the scene [ / frame / track] data to the lower-level station through the network with the smallest data processing block as the processing unit.

[0103] ⑤ Clean up sent data: promptly delete the auxiliary data and main data (or only the main data) in the storage location that has been sent, calculated, and saved to release computer resources.

[0104] 1.4.3 Computation Module

[0105] See also Fig. 9 , the calculation module includes:

[0106] ① Adjust auxiliary data as needed: stations at all levels appropriately adjust the required auxiliary data according to the progress of data calculation at each level.

[0107] ② Data calculation start-up mechanism: When it is determined that there is data that needs to be calculated in the exchange data storage location, the site at this level will immediately execute the calculation transaction of this level.

[0108] ③ Calculate data according to assigned tasks: Call this level station to collect various required parameter data and auxiliary data in the "receive" storage according to the format, and calculate the corresponding main data that has not been processed in the "receive" storage block by block according to the calculation task processing algorithm assigned to this level station, and save the main data after calculation in the "send" storage according to the format.

[0109] ④Continuously calculate subsequent data: Use the smallest data processing block as the processing unit to continuously calculate subsequent blocks of data for the entire scene [ / frame / track].

[0110] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "Send" storage location.

[0111] ⑥ Save the sent, received and calculated data: Each station at each level shall save the received (or written) main data of the whole scene [ / frame / track] and auxiliary data (or only the main data) processed by the station at this level in the form of files in a timely manner according to the format to the file receiving, processing and forwarding directory. The file name shall comply with the file naming rules and mark the corresponding data level.

[0112] 2. Implementation of application mode

[0113] The overall model of the parallel processing method and device is summarized from four aspects: system structure, parallel mechanism, system function, and system module. From the perspective of three high, medium, and low versions of data transmission, the specific implementation forms and technical means show obvious different characteristics. Especially in the long-range and short-range communication mode between stations, the structural and functional tendency of the application system of the present invention is changed, and the system structure function is more specific and characteristic in space. The communication mode between stations determines the application mode under the communication technology framework, which is related to the operation of each part of the system, plays a role, generates benefits, and forms three application modes.

[0114] The first application mode has its own advantages in data transmission and data computing. Inter-station remote communication emphasizes data transmission tasks as the core and increases data computing benefits. In terms of system structure, due to the use of remote communication between stations, site and inter-station remote communication constitute a multi-level site remote data communication system and a supercomputer cluster with independent distributed computing resources. In terms of system functions, site and inter-station remote communication exhibits the dual-mode function of data transmission and data computing parallel processing of multi-level site independent distributed computer resources; in the dual-mode, stations at all levels concurrently execute remote data transmission, and process big data with concurrent computing dispatch tasks and super computing power of superimposed distributed computing resources.

[0115] The second application mode emphasizes data computing in data transmission and data computing. Short-range communication between stations is convenient for reducing channel costs and increasing data transmission speed, and improving the processing speed of data computing tasks. In terms of system structure, due to the use of short-range communication between stations, short-range communication between sites and stations constitutes a multi-level site short-range data communication system and a supercomputer cluster with independently distributed computing resources. In terms of system functions, short-range communication between sites and stations demonstrates the strong computing mode function of data transmission and data computing parallel processing of multi-level site independently distributed computer resources. In the strong computing mode, stations at all levels execute short-range data distribution concurrently, and process big data with concurrent computing dispatch tasks and super computing power of superimposed distributed computing resources.

[0116] The third application mode emphasizes data transmission locally and data calculation locally. Long-distance and short-distance communication between stations can emphasize data transmission and data calculation respectively according to opportunities. In terms of system structure, due to the use of long-distance and short-distance communication between stations, long-distance and short-distance communication is used at the same time between stations to form a multi-level site local long-distance and local short-distance data communication system and a supercomputer cluster with independent distributed computing resources. In terms of system functions, long-distance and short-distance communication is used at the same time between stations, showing the comprehensive mode function of data transmission and data calculation parallel processing of multi-level site independent distributed computer resources; in the comprehensive mode, each station concurrently executes part of the long-distance data transmission and part of the short-distance data distribution, and processes big data with the super computing power of concurrent computing dispatch tasks and superimposed distributed computing resources.

[0117] 3 Related statements

[0118] 3.1 Site Computer

[0119] The site of the present invention is composed of at least one computer, but the number of site computers is not limited. In practical applications, high-performance configuration of the site should be considered, and the computer responsible for data computing should be equipped with sufficient resources such as advanced CPU or GPU, MIC (integrated multi-core) and memory to improve the site computing ability. The site of the present invention includes optimal and non-optimal computer configurations.

[0120] Arranging a certain site not to perform data computing tasks at each level of sites for data transmission and data computing is a foreseeable structural form of the present invention.

[0121] 3.2 Inter-station communication lines

[0122] Inter-station communication lines include wired and wireless communication lines. Wired lines are communication lines that use condensed media with good electrical or optical conductivity to transmit information, such as optical fiber, twisted pair, coaxial cable and other communication lines; wireless lines are communication lines of various frequency bands that use electromagnetic wave spectrum resources to transmit in the air, such as radio waves, microwaves and infrared communication lines. Dedicated lines include optical fiber, DDN (digital data network), ISDN (integrated services digital network), frame relay, packet switching, PSTN (public telephone exchange network) and other types. Communication lines can be rented from operators, such as leased lines, satellite or terrestrial microwave channels, etc.; wireless channels in approved frequency bands or specific data communications in public frequency bands can be used; wired communication lines can be built by oneself. Communication lines include new communication lines in the future.

[0123] 3.3 Data Communication System

[0124] Data communication system refers to network and non-network data transmission systems, which are divided into three types: network, general and special. Network data communication system refers to various forms of communication systems that meet the computer communication network standards, such as various communication systems of the current IEEE (Institute of Electrical and Electronics Engineers) 802 standard. General data communication systems such as CCSDS AOS (International Consultative Committee on Space Data Systems Advanced On-Orbit System) or DTMB (China Terrestrial Digital Multimedia Television Broadcasting), DVB-S2 (European Second Generation Satellite Digital Television Broadcasting), DVB-T2 (European Second Generation Terrestrial Digital Television Broadcasting) and other standard interface data transmission systems. Special data communication system refers to a specially designed data communication system. From the perspective of the degree of transparency of data transmission to users, general data communication machines use intermediate version transmission modules, and low-level version transmission modules belong to special data communication machines.

[0125] 3.4 Minimum data processing block calculation time

[0126] The speed of remote data transmission is very slow compared to the speed of computer calculation. In the business processing module, the data calculation time of the smallest data processing block should be less than its data transmission time, in order to avoid delaying the timeliness of the overall data transmission. In the tasks of performing data transmission and data calculation, although the data transmission task is the primary and critical, the execution of the data calculation task cannot delay the entire data transmission time, otherwise it is necessary to adjust the site data calculation task or select better computer resources to ensure high time efficiency of the data transmission process. The present invention includes that the data calculation time of the smallest data processing block can be less than, equal to, or greater than its data transmission time.

[0127] 3.5 Data exchange method

[0128] Business modules can exchange data using files, such as database table files, text files, binary files, etc., or using arrays, variable symbols, memory, etc. Considering the read and write speed, the storage medium for exchanging data is preferably memory, but other media such as solid state disks can also be used.

[0129] 3.6 File Naming Rules

[0130] The data file name should be named in a standardized manner, such as the full name of the file name includes the file name and the suffix name, the file name includes the data source, data longitude and latitude, data date, data time, data category, data coding, etc., and "-" can be used as a separator. The suffix name indicates the file category and is identified in a conventional manner. Taking satellite remote sensing data as an example, for example, the data source is the satellite name and the detector name, the data longitude and latitude are the longitude and latitude of the monitoring site, the data date is the specific date of remote sensing data collection, the data time is the specific time of remote sensing data collection, the data category is the data level or data processing batch, the data coding is the data product number, and the suffix name indicates the data file category. The file name, array name and variable symbol used to exchange data, and the file name for saving data should comply with standardized naming. The present invention is not limited to specific file naming and naming methods, including the use of non-standard named file names.

[0131] 3.7 Multi-channel data processing

[0132] In practical applications, the technology of the present invention can be extended to an integrated system to process data from multiple different sources in a time-sharing manner. Taking one channel of data as an example, the present invention can be expanded to multiple channels of different data from the same source, such as multiple channels of collected data from different detectors on the same satellite, and can also be expanded to multiple channels of different data from different sources, such as multiple channels of collected data from different detectors on different satellites. The remote sensing data and telemetry data of each satellite are different, and may have different parameters and formats. The present invention can use a processing mechanism for data from multiple sources and its technical solution, as well as a time conflict processing mechanism. In the processing of multi-source data, it is possible to insert a data sending task at an intermediate site, which still belongs to the technical solution of the present invention. In the processing of multi-source data, some transmission data may not be assigned data calculation tasks, which still belongs to the technical solution of the present invention.

[0133] 3.8 Duplex Mode of the Present Invention

[0134] The present invention only describes a parallel processing method and device for multi-level data transmission and data calculation in one direction. If the reverse direction conforms to the relevant statements of the present invention, and the forward direction partially, mostly or completely overlaps with the equipment or system in the forward direction, it belongs to the duplex mode of the present invention and still falls within the protection scope of the present invention.

[0135] 3.9 Relevant Scope of Protection

[0136] The parallel processing method and device of multi-level data transmission and data calculation are described in combination with the accompanying drawings from aspects such as system structure, parallel mechanism, system function and system module, so as to realize the efficient parallel processing function of data transmission and data calculation of multi-level sites. The implementation process of the present invention is complicated and there are many different implementation methods. Some specific methods are used in the description of the embodiments in order to show actual examples, but the present invention is not limited to the described implementation methods. If different implementation forms are designed according to the teachings of the present invention, without departing from the principles and spirit of the present invention, these changes, modifications, substitutions and variations of the implementation methods still fall within the protection scope of the present invention.

[0137] Example description: 2 examples for each mode, a total of 6 implementation examples

[0138] 4 Embodiment 1: Mode 1 Example 1. Taking the real-time data of the YL1 (Sample No. 1) satellite CCD detector as an example, a parallel processing method and device for multi-level data transmission and data calculation of data transmitted by a CCSDS AOS&XTCE (XML Telemetry and Command Exchange) remote general data communication machine.

[0139] In this embodiment, the YL1 satellite is in a sun-synchronous orbit, and the real-time data from the CCD detector when the YL1 satellite passes the ground station includes auxiliary data and main data. Auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time and other recorded data, and monitoring data for attitude control systems, communication systems, power systems, etc. The main data is the remote sensing data collected by the on-board CCD detector. The satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction for daily monitoring by the station department. Taking the real-time data of the YL1 satellite CCD detector as an example, the parallel processing method and device of multi-level data transmission and data calculation using CCSDS AOS&XTCE remote general data communication machine for data transmission are explained, including statements such as system structure, parallel mechanism, system function and system module.

[0140] 4.1 System Structure

[0141] The system structure includes:

[0142] ① Computer site composition: The site consists of 5 levels from level 1 to level 5, and most sites are composed of 3 computers in the network.

[0143] ②Inter-station communication technology system: The stations are connected by a remote data communication system, using CCSDS AOS&XTCE remote universal data communication equipment, renting satellite channels or dedicated lines, or applying for wireless channels.

[0144] ③ Overall structural functional system: 5-level sites and inter-station remote communications constitute a multi-level site remote data communication system and a supercomputer cluster with independent distributed computing resources.

[0145] ④Tree-branch type static structure: 5-level sites form a multi-level one-branch tree structure.

[0146] ⑤ Site data processing module: The site system module is divided into data sending, receiving, calculation and other modules.

[0147] ⑥ Parallel processing dynamic mode: Data is transmitted concurrently from the first-level station to the fifth-level station, and the data processing tasks assigned to each station are calculated in an interleaved manner.

[0148] 4.2 Parallel Mechanism

[0149] The parallel mechanism includes:

[0150] ⑴ Minimum data processing block unit. Design data exchange and data sending, receiving, computing and other data processing services at all levels, all of which are processed in the specified format with the minimum data processing block as the unit. The entire scene data is divided into blocks using the minimum data processing block as the processing unit, and the main and auxiliary data are transmitted synchronously or closely, providing opportunities for concurrent data transmission and computing, relying on the parallel processing of data transmission and data computing to achieve high efficiency of data transmission and data computing.

[0151] (2) Setting of minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. The correspondence and quantity of telemetry data and remote sensing data collected on the satellite in this example are determined according to the actual data collection situation. The telemetry data related to the remote sensing data correction of the CCD detector is integrated into 1 frame transmission. This frame of telemetry data corresponds to 30 frames of remote sensing data. The minimum data processing block is determined to be 1 frame of telemetry data and 30 frames of remote sensing data. The minimum data processing block has a relatively small amount of data, which is more conducive to reducing delays and improving the timeliness of overall data transmission.

[0152] ⑶ Data transmission and calculation interaction mechanism. Taking the smallest data processing block as the unit, the data of the entire scene is continuously transmitted and calculated in an interleaved manner, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of sites. ① Block data transmission: Each scene data is divided into M blocks and transmitted block by block, and the transmission task within the station is completed block by block; in the N-station transmission, each block of data is transmitted station by station. ② Block data calculation: The calculation task of each scene data is assigned to each station according to the plan, and the calculation task is assigned by block processing, and the data calculation task is completed station by station. ③ Interleaved transmission calculation: Each scene data is transmitted in blocks to each station, and each station calculates the data by block. The transmission and calculation tasks of the main data and auxiliary data are completed in an interleaved transmission and calculation manner at each station.

[0153] 4.3 System Functions

[0154] The system functions include:

[0155] ① In-station business classification function: stations at all levels are responsible for the main and auxiliary data sending, receiving, calculation and storage services.

[0156] ②Parallel processing dynamic function: Data flows from the 1st level station to the 5th level station. Each level of the station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation of each station, and then completes the multi-level station data transmission and the computing power to complete the 0-4 level data correction task calculation.

[0157] ③ Communication mode construction function: The inter-station remote communication mode determines the application mode under the communication technology framework, which is related to the operation of each part of the system, and plays a role and generates benefits; the inter-station and inter-station remote communication exhibits the dual-mode function of data transmission and data calculation parallel processing of computer resources independently distributed at multiple levels of sites.

[0158] ④ Overall structural mode function: Two-in-one mode, each station executes remote data transmission concurrently, and processes big data with the super computing power of concurrent computing dispatch tasks and superimposed distributed computing resources.

[0159] 4.4 System Module

[0160] The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules.

[0161] 4.4.1 System Presets

[0162] The system presets include:

[0163] ① Determine the program process form: Data processing must first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading.

[0164] ② Determine the communication sending and receiving methods: Data communication must first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods used between stations, and require that the data sending and receiving communication methods are matched.

[0165] ③ Determine the signal transmission and reception matching: Signal transmission and reception must first determine the signal quality and matching of the specific communication line, and properly arrange the signal low-frequency amplifier, medium-frequency amplifier and power amplifier.

[0166] ④ Determine the data exchange method: Data exchange requires first determining the data exchange method between data sending, receiving and calculation, and selecting an array as the storage location for exchanged data.

[0167] ⑤ Prepare computing task algorithms: Different computing tasks are assigned to sites at each level, and the computing task processing algorithms for sites at each level must be prepared first.

[0168] ⑥ Standardized file naming: Data file names involve data call, storage, exchange, etc., and should be named according to rules or regulations.

[0169] 4.4.2 Transmission Module

[0170] Choose to use the intermediate version transmission module. The user end uses the CCSDS AOS&XTCE universal data communication system to exchange primary and secondary data according to the specifications through the hardware interface to complete data transmission and reception.

[0171] The sending submodule includes:

[0172] Comply with system preset conditions and send primary and secondary data in a non-network communication environment.

[0173] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0174] ② Read the sending data in blocks: According to the minimum data processing block main and auxiliary data packaging frame plan, read the main data of the "send" storage location where the data calculation has been completed according to the format. If auxiliary data needs to be sent, also read the auxiliary data of the corresponding "send" storage location.

[0175] ③ Provide exchange data to the interface: The user end writes the exchange data to the hardware interface specification, and the general data communication machine is responsible for data packaging, framing and transmission.

[0176] ④Continuously send subsequent data: Continuously read the subsequent data of the main and auxiliary data, provide an interface to exchange data, and complete the sending of the entire scene data.

[0177] ⑤ Clean up sent data: Delete the auxiliary data and main data that have been sent, calculated, and saved in the exchange data storage in time to release computer resources.

[0178] The receiving submodule includes:

[0179] Comply with system preset conditions and receive primary and secondary data in network or non-network communication environment.

[0180] ① Standardized reading of interface data: The user end reads the deframed and unpacked data transmitted to the interface by the data deframer.

[0181] ② Save received data by block: store main and auxiliary data in minimum data processing blocks according to the flag, and save auxiliary data and main data in corresponding data storage locations.

[0182] ③Continue to receive subsequent data: Continue to receive and save subsequent data of primary and auxiliary data to complete the reception of the entire scene data.

[0183] 4.4.3 Computation Module

[0184] The calculation module comprises:

[0185] ① Adjust auxiliary data as needed: stations at all levels appropriately adjust the required auxiliary data according to the progress of data calculation at each level.

[0186] ②Data calculation start-up mechanism: When it is determined that there is data that needs to be calculated in the exchange data storage location, the site at this level will immediately execute the calculation transaction of this level.

[0187] ③ Calculate data according to assigned tasks: Call this level station to collect various required parameter data and auxiliary data in the "receive" storage according to the format, and calculate the corresponding main data that has not been processed in the "receive" storage block by block according to the calculation task processing algorithm assigned to this level station, and save the main data after calculation in the "send" storage according to the format.

[0188] ④Continuously calculate subsequent data: Use the smallest data processing block as the processing unit to continuously calculate subsequent blocks of data for the entire scene.

[0189] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "Send" storage location.

[0190] ⑥ Save the sent, received, and calculated data: Each station at each level saves the received and processed whole scene main data and auxiliary data in the file receiving, processing and forwarding directory in a timely manner in the form of files. The file name complies with the file naming rules and marks the corresponding data level. 5 Embodiment 2: Mode 1 Example 2. Take the non-real-time data of the YL2 (Sample No. 2) satellite WFV detector as an example, taking the multi-level data transmission and data calculation parallel processing method and device of computer wide area network DDN dedicated line transmission data.

[0191] The YL2 satellite in this embodiment is in a sun-synchronous orbit, and the non-real-time data from the WFV detector when the YL2 satellite passes the ground station includes auxiliary data and main data. Auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time and other recorded data, and monitoring data for attitude control systems, communication systems, power systems, etc. The main data is the remote sensing data collected by the WFV detector on the satellite. The satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction for daily monitoring by the station department. Taking the non-real-time data of the YL2 satellite WFV detector as an example, the parallel processing method and device of multi-level data transmission and data calculation using the wide area network DDN dedicated line to transmit data are explained, including statements such as system structure, parallel mechanism, system function and system module.

[0192] 5.1 System Structure

[0193] The system structure includes:

[0194] ① Computer site composition: The site consists of 4 levels from level 1 to level 4, and each level of the site consists of 3 computers to form a network.

[0195] ②Inter-station communication technology system: The stations are connected by a remote computer communication network (wide area network). The inter-station remote network version data transmission system consists of two end networks and an intermediate subnet, and rents a DDN dedicated line.

[0196] ③ Overall structural functional system: 4-level sites and inter-site remote communications constitute a multi-level site remote data communication system and a supercomputer cluster with independent distributed computing resources.

[0197] ④Tree-branch type static structure: 4-level sites form a multi-level one-branch tree structure.

[0198] ⑤ Site data processing module: The site system module is divided into data reading, writing, calculation and other modules.

[0199] ⑥ Parallel processing dynamic mode: Data is transmitted concurrently from the first-level station to the fourth-level station step by step, and the data processing tasks assigned to each station are calculated in an interleaved manner step by step.

[0200] 5.2 Parallel Mechanism

[0201] The parallel mechanism includes:

[0202] ⑴ Minimum data processing block unit. Design data exchange and data sending, receiving, computing and other data processing services at all levels, all of which are processed in the specified format with the minimum data processing block as the unit. The entire scene data is divided into blocks using the minimum data processing block as the processing unit, and the main and auxiliary data are transmitted synchronously or closely, providing opportunities for concurrent data transmission and computing, relying on the parallel processing of data transmission and data computing to achieve high efficiency of data transmission and data computing.

[0203] (2) Setting of minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. The correspondence and quantity of telemetry data and remote sensing data collected on the satellite in this example are determined according to the actual data collection situation. The telemetry data related to the correction of the remote sensing data of the WFV detector are integrated into 1 frame transmission. This frame of telemetry data corresponds to 50 frames of remote sensing data. The minimum data processing block is determined to be 1 frame of telemetry data and 50 frames of remote sensing data. The minimum data processing block has a relatively small amount of data, which is more conducive to reducing delays and improving the timeliness of overall data transmission.

[0204] ⑶ Data transmission and calculation interaction mechanism. Taking the smallest data processing block as the unit, the data of the entire scene is continuously transmitted and calculated in an interleaved manner, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of sites. ① Block data transmission: Each scene data is divided into M blocks and transmitted block by block, and the transmission task within the station is completed block by block; in the N-station transmission, each block of data is transmitted station by station. ② Block data calculation: The calculation task of each scene data is assigned to each station according to the plan, and the calculation task is assigned by block processing, and the data calculation task is completed station by station. ③ Interleaved transmission calculation: Each scene data is transmitted in blocks to each station, and each station calculates the data by block. The transmission and calculation tasks of the main data and auxiliary data are completed in an interleaved transmission and calculation manner at each station.

[0205] 5.3 System Functions

[0206] The system functions include:

[0207] ① In-station business classification function: stations at all levels are responsible for the reading, writing, calculation and storage of primary and secondary data.

[0208] ② Parallel processing dynamic function: Data flows from the 1st level station to the 4th level station. Each level of the station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation of each station, and then completes the multi-level station data transmission and the computing power to complete the 0-4 level data correction task calculation.

[0209] ③ Communication mode construction function: The inter-station remote communication mode determines the application mode under the communication technology framework, which is related to the operation of each part of the system, and plays a role and generates benefits; the inter-station and inter-station remote communication exhibits the dual-mode function of data transmission and data calculation parallel processing of computer resources independently distributed at multiple levels of sites.

[0210] ④ Overall structural mode function: Two-in-one mode, each station executes remote data transmission concurrently, and processes big data with the super computing power of concurrent computing dispatch tasks and superimposed distributed computing resources.

[0211] 5.4 System Module

[0212] The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules.

[0213] 5.4.1 System Presets

[0214] The system presets include:

[0215] ① Determine the program process form: Data processing must first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading.

[0216] ② Determine the communication sending and receiving methods: Data communication must first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods used between stations, and require that the data sending and receiving communication methods are matched.

[0217] ③ Determine the signal transmission and reception matching: Signal transmission and reception must first determine the signal quality and matching of the specific communication line, and properly arrange the signal low-frequency amplifier, medium-frequency amplifier and power amplifier.

[0218] ④ Determine the data exchange method: Data exchange requires first determining the data exchange method between data reading, writing and calculation, and selecting files as the storage location for exchanged data.

[0219] ⑤ Prepare computing task algorithms: Different computing tasks are assigned to sites at each level, and the computing task processing algorithms for sites at each level must be prepared first.

[0220] ⑥ Standardized file naming: Data file names involve data call, storage, exchange, etc., and should be named according to rules or regulations.

[0221] 5.4.2 Transmission Module

[0222] Choose to use the advanced version of the transmission module. The advanced platform can build transparent access to different addresses in the network environment, and read and write data directly like a local machine in the advanced platform environment.

[0223] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0224] ② Reading of receiving and sending data: The receiving and sending software uses the smallest data processing block as the processing unit, and reads the main data in the "send" storage location where the data calculation has been completed at this level of station according to the format. If auxiliary data needs to be sent, the auxiliary data corresponding to the "send" storage location is also read.

[0225] ③ Writing of receiving and sending data: write auxiliary data and main data into the corresponding storage place of "receiving" of the lower-level station according to the format.

[0226] ④Continuously forward subsequent data: Continuously forward the subsequent blocks of auxiliary data and main data of the scene data to the lower-level station through the network with the smallest data processing block as the processing unit.

[0227] ⑤ Clean up sent data: Delete the auxiliary data and main data in the storage location that have been sent, calculated, and saved in time to release computer resources.

[0228] 5.4.3 Computation Module

[0229] The calculation module comprises:

[0230] ① Adjust auxiliary data as needed: stations at all levels appropriately adjust the required auxiliary data according to the progress of data calculation at each level.

[0231] ② Data calculation start-up mechanism: When it is determined that there is data that needs to be calculated in the exchange data storage location, the site at this level will immediately execute the calculation transaction of this level.

[0232] ③ Calculate data according to assigned tasks: Call this level station to collect various required parameter data and auxiliary data in the "receive" storage according to the format, and calculate the corresponding main data that has not been processed in the "receive" storage block by block according to the calculation task processing algorithm assigned to this level station, and save the main data after calculation in the "send" storage according to the format.

[0233] ④Continuously calculate subsequent data: Use the smallest data processing block as the processing unit to continuously calculate subsequent blocks of data for the entire scene.

[0234] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "Send" storage location.

[0235] ⑥ Save the sent, received, and calculated data: Each station at each level saves the written and processed whole scene main data and auxiliary data in the file receiving, processing and forwarding directory in a timely manner in the form of files. The file name complies with the file naming rules and marks the corresponding data level. 6 Embodiment 3: Mode 2 Example 1. Take the real-time data of the YL1 (Sample No. 1) satellite CCD detector as an example, and use the computer parallel port to directly connect to send and receive data for multi-level data transmission and data calculation parallel processing method and device.

[0236] In this embodiment, the YL1 satellite is in a sun-synchronous orbit, and the real-time data from the CCD detector when the YL1 satellite passes the ground station includes auxiliary data and main data. Auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time and other recorded data, and monitoring data for attitude control systems, communication systems, power systems, etc. The main data is the remote sensing data collected by the on-board CCD detector. The satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction, to provide data services for users' daily monitoring in a timely manner. Taking the real-time data of the YL1 satellite CCD detector as an example, the parallel processing method and device of multi-level data transmission and data calculation with parallel port direct connection for sending and receiving data are explained, including statements such as system structure, parallel mechanism, system function and system module.

[0237] 6.1 System Structure

[0238] The system structure includes:

[0239] ① Computer site composition: The site ( / node) consists of 5 levels of sites, from level 1 to level 5, and most sites are composed of a network of 3 computers.

[0240] ②Inter-station communication technology system: The stations are connected by a short-range non-computer communication network (non-network version of short-range data communication system). The short-range non-network version of the data transmission system uses parallel direct communication that is not included in the network technology standard.

[0241] ③ Overall structural functional system: 5-level site and inter-site short-range communication, forming a multi-level site short-range data communication system and a supercomputer cluster with independent distributed computing resources.

[0242] ④Tree-branch type static structure: 5-level sites form a multi-level one-branch tree structure.

[0243] ⑤ Site data processing module: The site system module is divided into data sending, receiving, calculation and other modules.

[0244] ⑥ Parallel processing dynamic mode: Data is transmitted concurrently from the first-level station to the fifth-level station, and the data processing tasks assigned to each station are calculated in an interleaved manner.

[0245] 6.2 Parallel Mechanism

[0246] The parallel mechanism includes:

[0247] ⑴ Minimum data processing block unit. Design data exchange and data sending, receiving, computing and other data processing services at all levels, all of which are processed in the specified format with the minimum data processing block as the unit. The entire scene data is divided into blocks using the minimum data processing block as the processing unit, and the main and auxiliary data are transmitted synchronously or closely, providing opportunities for concurrent data transmission and computing, relying on the parallel processing of data transmission and data computing to achieve high efficiency of data transmission and data computing.

[0248] (2) Setting of minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. The correspondence and quantity of telemetry data and remote sensing data collected on the satellite in this example are determined according to the actual data collection situation. The telemetry data related to the remote sensing data correction of the CCD detector is integrated into 1 frame transmission. This frame of telemetry data corresponds to 30 frames of remote sensing data. The minimum data processing block is determined to be 1 frame of telemetry data and 30 frames of remote sensing data. The minimum data processing block has a relatively small amount of data, which is more conducive to reducing delays and improving the timeliness of overall data transmission.

[0249] ⑶ Data transmission and calculation interaction mechanism. Taking the smallest data processing block as the unit, the data of the entire scene is continuously transmitted and calculated in an interleaved manner, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of sites. ① Block data transmission: Each scene data is divided into M blocks and transmitted block by block, and the transmission task within the station is completed block by block; in the N-station transmission, each block of data is transmitted station by station. ② Block data calculation: The calculation task of each scene data is assigned to each station according to the plan, and the calculation task is assigned by block processing, and the data calculation task is completed station by station. ③ Interleaved transmission calculation: Each scene data is transmitted in blocks to each station, and each station calculates the data by block. The transmission and calculation tasks of the main data and auxiliary data are completed in an interleaved transmission and calculation manner at each station.

[0250] 6.3 System Functions

[0251] The system functions include:

[0252] ① In-station business classification function: stations at all levels are responsible for the main and auxiliary data sending, receiving, calculation and storage services.

[0253] ②Parallel processing dynamic function: Data flows from the 1st level station to the 5th level station. Each level of the station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation of each station, and then completes the multi-level station data transmission and the computing power to complete the 0-4 level data correction task calculation.

[0254] ③ Communication mode construction function: The short-range communication mode between stations determines the application mode under the communication technology framework, which is related to the operation of each part of the system, and plays a role and generates benefits; the short-range communication between stations shows the strong-core mode function of data transmission and data calculation parallel processing of computer resources independently distributed in multi-level stations.

[0255] ④ Overall structural mode function: In the strong computing mode, each station executes short-range data distribution concurrently, and processes big data with the super computing power of concurrent computing tasks and superimposed distributed computing resources.

[0256] 6.4 System Module

[0257] The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules.

[0258] 6.4.1 System Presets

[0259] The system presets include:

[0260] ① Determine the program process form: Data processing must first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading.

[0261] ② Determine the communication sending and receiving methods: Data communication must first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods used between stations, and require that the data sending and receiving communication methods are matched.

[0262] ③ Determine the signal transmission and reception matching: Signal transmission and reception must first determine the signal quality and matching of the specific communication line, and properly arrange the signal low-frequency amplifier, medium-frequency amplifier and power amplifier.

[0263] ④ Determine the data exchange method: Data exchange requires first determining the data exchange method between data sending, receiving and calculation, and selecting an array as the storage location for exchanged data.

[0264] ⑤ Prepare computing task algorithms: Different computing tasks are assigned to sites at each level, and the computing task processing algorithms for sites at each level should be prepared first.

[0265] ⑥ Standardized file naming: Data file names involve data call, storage, exchange, etc., and should be named according to rules or regulations.

[0266] 6.4.2 Transmission Module

[0267] Choose to use the low-level transmission module. Design the data frame structure, data reading, frame packing, data transmission, frame unloading, unpacking and data saving, etc.

[0268] The sending submodule includes:

[0269] Comply with system preset conditions and send primary and secondary data in a non-network communication environment.

[0270] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0271] ② Read the sending data in blocks: According to the minimum data processing block main and auxiliary data packaging frame plan, read the main data of the "send" storage location where the data calculation has been completed according to the format. If auxiliary data needs to be sent, also read the auxiliary data of the corresponding "send" storage location.

[0272] ③ Data packaging and framing transmission: Data is processed synchronously or immediately after transmission according to the minimum data processing block. When the main and auxiliary data are transmitted in the same frame, the main and auxiliary data are combined into one data packet according to the packet format; or when the main and auxiliary data are transmitted in separate frames, the auxiliary and main data are formed into one or several independent data packets according to the packet format. The data packets and channel codes are framed according to the frame format, and the minimum data processing block data is transmitted synchronously or immediately.

[0273] ④Continuously send subsequent data: Continuously read and package the frames of the subsequent data of the main and auxiliary data to complete the sending of the entire scene data.

[0274] ⑤ Clean up sent data: Delete the auxiliary data and main data that have been sent, calculated, and saved in the exchange data storage in time to release computer resources.

[0275] The receiving submodule includes:

[0276] Comply with system preset conditions and receive primary and secondary data in a non-network communication environment.

[0277] ① Full-frame data reception: Completely receive each frame of data sent by the non-network system.

[0278] ② Receive data and unpack the frame: Unpack each received frame according to the flags of the frame header and packet header, distinguish the main and auxiliary data according to the data packet flag, and memorize the flag of the minimum data processing block data belonging to the same block.

[0279] ③ Save received data by block: store main and auxiliary data in the minimum data processing block according to the flag, and save auxiliary data and main data in the corresponding data storage place.

[0280] ④Continue to receive subsequent data: Continue to receive, unpack and save the subsequent data of the main and auxiliary data to complete the reception of the entire scene data.

[0281] 6.4.3 Computation Module

[0282] The calculation module comprises:

[0283] ① Adjust auxiliary data as needed: stations at all levels appropriately adjust the required auxiliary data according to the progress of data calculation at each level.

[0284] ②Data calculation start-up mechanism: When it is determined that there is data that needs to be calculated in the exchange data storage location, the site at this level will immediately execute the calculation transaction of this level.

[0285] ③ Calculate data according to assigned tasks: Call this level station to collect various required parameter data and auxiliary data in the "receive" storage according to the format, and calculate the corresponding main data that has not been processed in the "receive" storage block by block according to the calculation task processing algorithm assigned to this level station, and save the main data after calculation in the "send" storage according to the format.

[0286] ④Continuously calculate subsequent data: Use the smallest data processing block as the processing unit to continuously calculate subsequent blocks of data for the entire scene.

[0287] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "Send" storage location.

[0288] ⑥ Save the sent, received, and calculated data: Each station at each level saves the received and processed main scene data and auxiliary data in the form of files in a timely manner in the file receiving, processing and forwarding directory according to the format. The file name complies with the file naming rules and marks the corresponding data level. 7 Embodiment 4: Mode 2 Example 2. Take the non-real-time data of the YL1 (Sample No. 1) satellite CCD detector as an example, taking the multi-level data transmission and data calculation parallel processing method and device of computer metropolitan area network transmission data.

[0289] In this embodiment, the YL1 satellite is in a sun-synchronous orbit, and the real-time data from the CCD detector when the YL1 satellite passes the ground station includes auxiliary data and main data. Auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time and other recorded data, and monitoring data for attitude control systems, communication systems, power systems, etc. The main data is the remote sensing data collected by the on-board CCD detector. The satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction, to provide data services for users' daily monitoring in a timely manner. Taking the non-real-time data of the YL1 satellite CCD detector as an example, the parallel processing method and device of multi-level data transmission and data calculation using the metropolitan area network for data transmission are explained, including statements such as system structure, parallel mechanism, system function and system module.

[0290] 7.1 System Structure

[0291] The system structure includes:

[0292] ① Computer site composition: The site ( / node) consists of 4 levels from level 1 to level 4, and each level of the site consists of 3 computers in the network.

[0293] ②Inter-station communication technology system: The stations are connected by a short-range computer communication network (metropolitan area network). The inter-station short-range network version data transmission system consists of two-end networks and an intermediate subnet, and leases a DDN dedicated line.

[0294] ③ Overall structural functional system: 4-level sites and short-range communications between sites constitute a multi-level site short-range data communication system and a supercomputer cluster with independent distributed computing resources.

[0295] ④Tree-branch type static structure: 4-level sites form a multi-level one-branch tree structure.

[0296] ⑤ Site data processing module: The site system module is divided into data reading, writing, calculation and other modules.

[0297] ⑥ Parallel processing dynamic mode: Data is transmitted concurrently from the first-level station to the fourth-level station step by step, and the data processing tasks assigned to each station are calculated in an interleaved manner step by step.

[0298] 7.2 Parallel Mechanism

[0299] The parallel mechanism includes:

[0300] ⑴ Minimum data processing block unit. Design data exchange and data sending, receiving, computing and other data processing services at all levels, all of which are processed in the specified format with the minimum data processing block as the unit. The entire scene data is divided into blocks using the minimum data processing block as the processing unit, and the main and auxiliary data are transmitted synchronously or closely, providing opportunities for concurrent data transmission and computing, relying on the parallel processing of data transmission and data computing to achieve high efficiency of data transmission and data computing.

[0301] (2) Setting of minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. The correspondence and quantity of telemetry data and remote sensing data collected on the satellite in this example are determined according to the actual data collection situation. The telemetry data related to the remote sensing data correction of the CCD detector is integrated into 1 frame transmission. This frame of telemetry data corresponds to 30 frames of remote sensing data. The minimum data processing block is determined to be 1 frame of telemetry data and 30 frames of remote sensing data. The minimum data processing block has a relatively small amount of data, which is more conducive to reducing delays and improving the timeliness of overall data transmission.

[0302] ⑶ Data transmission and calculation interaction mechanism. Taking the smallest data processing block as the unit, the data of the entire scene is continuously transmitted and calculated in an interleaved manner, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of sites. ① Block data transmission: Each scene data is divided into M blocks and transmitted block by block, and the transmission task within the station is completed block by block; in the N-station transmission, each block of data is transmitted station by station. ② Block data calculation: The calculation task of each scene data is assigned to each station according to the plan, and the calculation task is assigned by block processing, and the data calculation task is completed station by station. ③ Interleaved transmission calculation: Each scene data is transmitted in blocks to each station, and each station calculates the data by block. The transmission and calculation tasks of the main data and auxiliary data are completed in an interleaved transmission and calculation manner at each station.

[0303] 7.3 System Functions

[0304] The system functions include:

[0305] ① In-station business classification function: stations at all levels are responsible for the reading, writing, calculation and storage of primary and secondary data.

[0306] ② Parallel processing dynamic function: Data flows from the 1st level station to the 4th level station. Each level of the station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation of each station, and then completes the multi-level station data transmission and the computing power to complete the 0-4 level data correction task calculation.

[0307] ③ Communication mode construction function: The inter-station communication mode determines the application mode under the communication technology framework, which is related to the operation of each part of the system, and plays a role and generates benefits; the short-range communication between stations and stations shows the strong computing mode function of data transmission and data calculation parallel processing of computer resources independently distributed in multiple levels of stations.

[0308] ④ Overall structural mode function: In the strong computing mode, each station executes short-range data distribution concurrently, and processes big data with the super computing power of concurrent computing tasks and superimposed distributed computing resources.

[0309] 7.4 System Module

[0310] The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules.

[0311] 7.4.1 System Presets

[0312] The system presets include:

[0313] ① Determine the program process form: Data processing must first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading.

[0314] ② Determine the communication sending and receiving methods: Data communication must first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods used between stations, and require that the data sending and receiving communication methods are matched.

[0315] ③ Determine the signal transmission and reception matching: Signal transmission and reception must first determine the signal quality and matching of the specific communication line, and properly arrange the signal low-frequency amplifier, medium-frequency amplifier and power amplifier.

[0316] ④ Determine the data exchange method: Data exchange requires first determining the data exchange method between data reading, writing and calculation, and selecting files as the storage location for exchanged data.

[0317] ⑤ Prepare computing task algorithms: Different computing tasks are assigned to sites at each level, and the computing task processing algorithms for sites at each level should be prepared first.

[0318] ⑥ Standardized file naming: Data file names involve data call, storage, exchange, etc., and should be named according to rules or regulations.

[0319] 7.4.2 Transmission Module

[0320] Choose to use the advanced version of the transmission module. The advanced platform can build transparent access to different addresses in the network environment, and read and write data directly like a local machine in the advanced platform environment.

[0321] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0322] ② Reading of receiving and sending data: The receiving and sending software uses the smallest data processing block as the processing unit, and reads the main data in the "send" storage location where the data calculation has been completed at this level of station according to the format. If auxiliary data needs to be sent, the auxiliary data corresponding to the "send" storage location is also read.

[0323] ③ Writing of receiving and sending data: write auxiliary data and main data into the corresponding storage place of "receiving" of the lower-level station according to the format.

[0324] ④Continuously forward subsequent data: Continuously forward the subsequent blocks of auxiliary data and main data of the scene data to the lower-level station through the network with the smallest data processing block as the processing unit.

[0325] ⑤ Clean up sent data: Delete the auxiliary data and main data in the storage location that have been sent, calculated, and saved in time to release computer resources.

[0326] 7.4.3 Computation Module

[0327] The calculation module comprises:

[0328] ① Adjust auxiliary data as needed: stations at all levels appropriately adjust the required auxiliary data according to the progress of data calculation at each level.

[0329] ②Data calculation start-up mechanism: When it is determined that there is data that needs to be calculated in the exchange data storage location, the site at this level will immediately execute the calculation transaction of this level.

[0330] ③ Calculate data according to assigned tasks: Call this level station to collect various required parameter data and auxiliary data in the "receive" storage according to the format, and calculate the corresponding main data that has not been processed in the "receive" storage block by block according to the calculation task processing algorithm assigned to this level station, and save the main data after calculation in the "send" storage according to the format.

[0331] ④Continuously calculate subsequent data: Use the smallest data processing block as the processing unit to continuously calculate subsequent blocks of data for the entire scene.

[0332] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "Send" storage location.

[0333] ⑥ Save the sent, received and calculated data: Each station at each level shall save the written and processed whole scene main data and auxiliary data in the file receiving, processing and forwarding directory in a timely manner in the form of files. The file name shall comply with the file naming rules and mark the corresponding data level. 8 Embodiment 5: Mode 3 Example 1. Take the real-time data of the YL1 (Sample No. 1) satellite CCD detector as an example. The long-range part between stations is transmitted by CCSDS AOS&XTCE, and the short-range part is transmitted by DTMB. A parallel processing method and device for multi-level data transmission and data calculation.

[0334] In this embodiment, the YL1 satellite is in a sun-synchronous orbit, and the real-time data from the CCD detector when the YL1 satellite passes the ground station includes auxiliary data and main data. Auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time and other recorded data, and monitoring data for attitude control systems, communication systems, power systems, etc. The main data is the remote sensing data collected by the on-board CCD detector. Satellite remote sensing data products are divided into five levels, and data correction is arranged during the transmission process, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction, and the inversion of parameter data such as temperature, humidity, wind, cloud, precipitation, sunshine and surface background is arranged for numerical prediction model system simulation. Taking the real-time data of the YL1 satellite CCD detector as an example, the parallel processing method and device of multi-level data transmission and data calculation between stations, in which the remote part is transmitted by CCSDS AOS&XTCE and the short-range part is transmitted by DTMB, is described, including system structure, parallel mechanism, system function and system module.

[0335] 8.1 System Structure

[0336] The system structure includes:

[0337] ① Computer site composition: The site ( / node) consists of 6 levels of sites, from level 1 to level 6, and most sites are composed of 3 computers in the network.

[0338] ②Inter-station communication technology system: The stations are connected by long-distance or short-distance data communication systems. The first five stations use CCSDS AOS&XTCE universal data communication machines, rent satellite channels or apply for wireless channels, and the last five stations use DTMB universal data communication machines and optical fiber communication.

[0339] ③ Overall structural functional system: The first 5 levels of sites and inter-station long-distance communication constitute a multi-level site long-distance data communication system and a supercomputer cluster with independent distributed computing resources; the last level of sites and inter-station short-range communication constitute a multi-site short-range data communication system and a supercomputer cluster with independent distributed computing resources.

[0340] ④Tree-branch type static structure: The 1st to 5th level sites form a multi-level one-branch tree structure, and the 5th and 6th level sites form a 5-branch type.

[0341] ⑤ Site data processing module: The site system module is divided into data sending, receiving, calculation and other modules.

[0342] ⑥ Parallel processing dynamic mode: Data is transmitted concurrently from the 1st level station to the 6th level station, and the data processing tasks assigned to each station are calculated in an interleaved manner.

[0343] 8.2 Parallel Mechanism

[0344] The parallel mechanism includes:

[0345] ⑴ Minimum data processing block unit. Design data exchange and data sending, receiving, computing and other data processing services at all levels, all of which are processed in the specified format with the minimum data processing block as the unit. The entire scene data is divided into blocks using the minimum data processing block as the processing unit, and the main and auxiliary data are transmitted synchronously or closely, providing opportunities for concurrent data transmission and computing, relying on the parallel processing of data transmission and data computing to achieve high efficiency of data transmission and data computing.

[0346] (2) Setting of minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. The correspondence and quantity of telemetry data and remote sensing data collected on the satellite in this example are determined according to the actual data collection situation. The telemetry data related to the remote sensing data correction of the CCD detector is integrated into 1 frame transmission. This frame of telemetry data corresponds to 30 frames of remote sensing data. The minimum data processing block is determined to be 1 frame of telemetry data and 30 frames of remote sensing data. The minimum data processing block has a relatively small amount of data, which is more conducive to reducing delays and improving the timeliness of overall data transmission.

[0347] ⑶ Data transmission and calculation interaction mechanism. Taking the smallest data processing block as the unit, the data of the entire scene is continuously transmitted and calculated in an interleaved manner, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of sites. ① Block data transmission: Each scene data is divided into M blocks and transmitted block by block, and the transmission task within the station is completed block by block; in the N-station transmission, each block of data is transmitted station by station. ② Block data calculation: The calculation task of each scene data is assigned to each station according to the plan, and the calculation task is assigned by block processing, and the data calculation task is completed station by station. ③ Interleaved transmission calculation: Each scene data is transmitted in blocks to each station, and each station calculates the data by block. The transmission and calculation tasks of the main data and auxiliary data are completed in an interleaved transmission and calculation manner at each station.

[0348] 8.3 System Functions

[0349] The system functions include:

[0350] ① In-station business classification function: stations at all levels are responsible for the main and auxiliary data sending, receiving, calculation and storage services.

[0351] ②Parallel processing dynamic function: Data flows from the 1st level station to the 6th level station. Each level of the station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation of each station, and then completes the multi-level station data transmission and the computing power to complete the 0-5 level data correction task calculation.

[0352] ③ Communication mode construction function: The short-range and long-range communication modes between stations determine the application mode under the communication technology framework, which is related to the operation of each part of the system, and plays a role and generates benefits; the stations partially adopt long-range communication and partially adopt short-range communication, which shows the comprehensive mode function of data transmission and data calculation for parallel processing of computer resources independently distributed in multi-level sites.

[0353] ④ Overall structural mode function: comprehensive mode, each station concurrently executes part of the long-distance data transmission and part of the short-distance data distribution, and processes big data with the super computing power of concurrent computing assignment tasks and superimposed distributed computing resources.

[0354] 8.4 System Modules

[0355] The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules.

[0356] 8.4.1 System Presets

[0357] The system presets include:

[0358] ① Determine the program process form: Data processing must first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading.

[0359] ② Determine the communication sending and receiving methods: Data communication must first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods used between stations, and require that the data sending and receiving communication methods are matched.

[0360] ③ Determine the signal transmission and reception matching: Signal transmission and reception must first determine the signal quality and matching of the specific communication line, and properly arrange the signal low-frequency amplifier, medium-frequency amplifier and power amplifier.

[0361] ④ Determine the data exchange method: Data exchange requires first determining the data exchange method between data sending, receiving and calculation, and selecting an array as the storage location for exchanged data.

[0362] ⑤ Prepare computing task algorithms: Different computing tasks are assigned to sites at each level, and the computing task processing algorithms for sites at each level should be prepared first.

[0363] ⑥ Standardized file naming: Data file names involve data call, storage, exchange, etc., and should be named according to rules or regulations.

[0364] 8.4.2 Transmission Module

[0365] Choose to use the intermediate version transmission module. The user end uses CCSDS AOS&XTCE, DTMB data communication system to exchange primary and secondary data through the hardware interface according to the specification to complete data transmission and reception.

[0366] The sending submodule includes:

[0367] Comply with system preset conditions and send primary and secondary data in a non-network communication environment.

[0368] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0369] ② Read the sending data in blocks: According to the minimum data processing block main and auxiliary data packaging frame plan, read the main data of the "send" storage location where the data calculation has been completed according to the format. If auxiliary data needs to be sent, also read the auxiliary data of the corresponding "send" storage location.

[0370] ③ Provide exchange data to the interface: The user end writes the exchange data to the hardware interface specification, and the general data communication machine is responsible for data packaging, framing and transmission.

[0371] ④Continuously send subsequent data: Continuously read the subsequent data of the main and auxiliary data, provide an interface to exchange data, and complete the sending of the entire scene data.

[0372] ⑤ Clean up sent data: Delete the auxiliary data and main data that have been sent, calculated, and saved in the exchange data storage in time to release computer resources.

[0373] The receiving submodule includes:

[0374] Comply with system preset conditions and receive primary and secondary data in network or non-network communication environment.

[0375] ① Standardized reading of interface data: The user end reads the deframed and unpacked data transmitted to the interface by the data deframer.

[0376] ② Save received data by block: store main and auxiliary data in minimum data processing blocks according to the flag, and save auxiliary data and main data in corresponding data storage locations.

[0377] ③Continue to receive subsequent data: Continue to receive and save subsequent data of primary and auxiliary data to complete the reception of the entire scene data.

[0378] 8.4.3 Computation Module

[0379] The calculation module comprises:

[0380] ① Adjust auxiliary data as needed: stations at all levels appropriately adjust the required auxiliary data according to the progress of data calculation at each level.

[0381] ② Data calculation start-up mechanism: When it is determined that there is data that needs to be calculated in the exchange data storage location, the site at this level will immediately execute the calculation transaction of this level.

[0382] ③ Calculate data according to assigned tasks: Call this level station to collect various required parameter data and auxiliary data in the "receive" storage according to the format, and calculate the corresponding main data that has not been processed in the "receive" storage block by block according to the calculation task processing algorithm assigned to this level station, and save the main data after calculation in the "send" storage according to the format.

[0383] ④Continuously calculate subsequent data: Use the smallest data processing block as the processing unit to continuously calculate subsequent blocks of data for the entire scene.

[0384] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "Send" storage location.

[0385] ⑥ Save the sent, received, and calculated data: Each station at each level shall save the received and processed whole scene main data and auxiliary data in the file receiving, processing and forwarding directory in a timely manner in the form of files. The file name shall comply with the file naming rules and mark the corresponding data level. 9 Embodiment 6: Mode 3 Example 2. Take the non-real-time data of the WFV detector of the YL2 (Sample No. 2) satellite as an example, and take the multi-level data transmission and data calculation parallel processing method and device of the long-range and short-range computer communication network transmission data as an example.

[0386] In this embodiment, the YL2 satellite is in a sun-synchronous orbit, and the non-real-time data from the WFV detector when the YL2 satellite passes the ground station includes auxiliary data and main data. Auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time and other recorded data, and monitoring data for attitude control systems, communication systems, power systems, etc. The main data is the remote sensing data collected by the WFV detector on the satellite. Satellite remote sensing data products are divided into five levels, and data correction is arranged during the transmission process, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction. Inversion of parameter data such as temperature, humidity, wind, cloud, precipitation, sunshine and surface background is arranged for use in numerical forecast model system simulation. Taking the non-real-time data of the YL2 satellite WFV detector as an example, the parallel processing method and device of multi-level data transmission and data calculation using long-range and short-range computer networks to transmit data are explained, including system structure, parallel mechanism, system function and system module

[0387] The system structure includes:

[0388] ① Computer site composition: The site ( / node) consists of 6 levels from level 1 to level 6, and each level of the site consists of 3 computers in the network.

[0389] ②Inter-station communication technology system: The stations are connected by long-distance or short-distance computer communication networks. The first 5 levels of inter-station wide area networks are composed of two-end networks and intermediate subnets, renting optical fiber or DDN dedicated lines and other communication lines; the last 1 level of 5 inter-station LAN communications (such as IEEE 802 standard LAN various communication technologies) use wireless or wired (such as optical fiber, twisted pair, coaxial cable, etc.) communication lines.

[0390] ③ Overall structural functional system: The first 5 levels of sites and inter-station long-distance communication constitute a multi-level site long-distance data communication system and a supercomputer cluster with independent distributed computing resources; the last level of sites and inter-station short-range communication constitute a multi-site short-range data communication system and a supercomputer cluster with independent distributed computing resources.

[0391] ④Tree-branch type static structure: The 1st to 5th level sites form a multi-level one-branch tree structure, and the 5th and 6th level sites form a 5-branch type.

[0392] ⑤ Site data processing module: The site system module is divided into data reading, writing, calculation and other modules.

[0393] ⑥ Parallel processing dynamic mode: Data is transmitted concurrently from the 1st level station to the 6th level station, and the data processing tasks assigned to each station are calculated in an interleaved manner.

[0394] 9.2 Parallel Mechanism

[0395] The parallel mechanism includes:

[0396] ⑴ Minimum data processing block unit. Design data exchange and data sending, receiving, computing and other data processing services at all levels, all of which are processed in the specified format with the minimum data processing block as the unit. The entire scene data is divided into blocks using the minimum data processing block as the processing unit, and the main and auxiliary data are transmitted synchronously or closely, providing opportunities for concurrent data transmission and computing, relying on the parallel processing of data transmission and data computing to achieve high efficiency of data transmission and data computing.

[0397] (2) Setting of minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. The correspondence and quantity of telemetry data and remote sensing data collected on the satellite in this example are determined according to the actual data collection situation. The telemetry data related to the correction of the remote sensing data of the WFV detector are integrated into 1 frame transmission. This frame of telemetry data corresponds to 50 frames of remote sensing data. The minimum data processing block is determined to be 1 frame of telemetry data and 50 frames of remote sensing data. The minimum data processing block has a relatively small amount of data, which is more conducive to reducing delays and improving the timeliness of overall data transmission.

[0398] ⑶ Data transmission and calculation interaction mechanism. Taking the smallest data processing block as the unit, the data of the entire scene is continuously transmitted and calculated in an interleaved manner, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of sites. ① Block data transmission: Each scene data is divided into M blocks and transmitted block by block, and the transmission task within the station is completed block by block; in the N-station transmission, each block of data is transmitted station by station. ② Block data calculation: The calculation task of each scene data is assigned to each station according to the plan, and the calculation task is assigned by block processing, and the data calculation task is completed station by station. ③ Interleaved transmission calculation: Each scene data is transmitted in blocks to each station, and each station calculates the data by block. The transmission and calculation tasks of the main data and auxiliary data are completed in an interleaved transmission and calculation manner at each station.

[0399] 9.3 System Functions

[0400] The system functions include:

[0401] ① In-station business classification function: stations at all levels are responsible for the reading, writing, calculation and storage of primary and secondary data.

[0402] ②Parallel processing dynamic function: Data flows from the 1st level station to the 6th level station. Each level of the station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation of each station, and then completes the multi-level station data transmission and the computing power to complete the 0-5 level data correction task calculation.

[0403] ③ Communication mode construction function: The long-distance and short-distance communication modes between stations determine the application mode under the communication technology framework, which is related to the operation of each part of the system, and plays a role and generates benefits; the stations partially adopt long-distance communication and partially adopt short-distance communication, which shows the comprehensive mode function of data transmission and data calculation for parallel processing of computer resources independently distributed in multi-level sites.

[0404] ④ Overall structural mode function: comprehensive mode, each station concurrently executes part of the long-distance data transmission and part of the short-distance data distribution, and processes big data with the super computing power of concurrent computing assignment tasks and superimposed distributed computing resources.

[0405] 9.4 System Modules

[0406] The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules.

[0407] 9.4.1 System Presets

[0408] ① Determine the program process form: Data processing must first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading.

[0409] ② Determine the communication sending and receiving methods: Data communication must first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods used between stations, and require that the data sending and receiving communication methods are matched.

[0410] ③ Determine the signal transmission and reception matching: Signal transmission and reception must first determine the signal quality and matching of the specific communication line, and properly arrange the signal low-frequency amplifier, medium-frequency amplifier and power amplifier.

[0411] ④ Determine the data exchange method: Data exchange requires first determining the data exchange method between data reading, writing and calculation, and selecting files as the storage location for exchanged data.

[0412] ⑤ Prepare computing task algorithms: Different computing tasks are assigned to sites at each level, and the computing task processing algorithms for sites at each level should be prepared first.

[0413] ⑥ Standardized file naming: Data file names involve data call, storage, exchange, etc., and should be named according to rules or regulations.

[0414] 9.4.2 Transmission Module

[0415] Choose to use the advanced version of the transmission module. The advanced platform can build transparent access to different addresses in the network environment, and read and write data directly like a local machine in the advanced platform environment.

[0416] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0417] ② Reading of receiving and sending data: The receiving and sending software uses the smallest data processing block as the processing unit, and reads the main data in the "send" storage location where the data calculation has been completed at this level of station according to the format. If auxiliary data needs to be sent, the auxiliary data corresponding to the "send" storage location is also read.

[0418] ③ Writing of receiving and sending data: write auxiliary data and main data into the corresponding storage place of "receiving" of the lower-level station according to the format.

[0419] ④Continuously forward subsequent data: Continuously forward the subsequent blocks of auxiliary data and main data of the scene data to the lower-level station through the network with the smallest data processing block as the processing unit.

[0420] ⑤ Clean up sent data: Delete the auxiliary data and main data in the storage location that have been sent, calculated, and saved in time to release computer resources.

[0421] 9.4.3 Computation Module

[0422] The calculation module comprises:

[0423] ① Adjust auxiliary data as needed: stations at all levels appropriately adjust the required auxiliary data according to the progress of data calculation at each level.

[0424] ② Data calculation start-up mechanism: When it is determined that there is data that needs to be calculated in the exchange data storage location, the site at this level will immediately execute the calculation transaction of this level.

[0425] ③ Calculate data according to assigned tasks: Call this level station to collect various required parameter data and auxiliary data in the "receive" storage according to the format, and calculate the corresponding main data that has not been processed in the "receive" storage block by block according to the calculation task processing algorithm assigned to this level station, and save the main data after calculation in the "send" storage according to the format.

[0426] ④Continuously calculate subsequent data: Use the smallest data processing block as the processing unit to continuously calculate subsequent blocks of data for the entire scene.

[0427] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "Send" storage location.

[0428] ⑥ Save the sent, received and calculated data: Each station at each level will save the written main data and auxiliary data of the whole scene and processed by the station at this level in the form of files in a timely manner according to the format to the file receiving, processing and forwarding directory. The file name follows the file naming rules and marks the corresponding data level.

Claims

1. An interactive concurrent method for multi-level data transmission and data calculation, It is characterized in that It includes three parts: minimum data processing block unit, minimum data processing block setting, and data transmission and calculation interaction mechanism; The minimum data processing block unit, that is, the data exchange and data sending, receiving, calculation and other data processing services of each level of stations are designed to be processed in the prescribed format with the minimum data processing block as the unit; The whole scene [ / frame / track] data is divided into blocks with the smallest data processing block as the processing unit, and the main and auxiliary data are transmitted synchronously or closely, providing opportunities for concurrent data transmission and calculation, relying on the parallel processing of data transmission and data calculation to obtain high efficiency of data transmission and data calculation; The minimum data processing block setting is as follows: the minimum data processing block includes auxiliary data and main data, or only includes main data; ① if there is no corresponding relationship between the auxiliary data and the main data, the minimum data processing block is determined by considering only the factors affecting the data calculation efficiency, and the minimum data processing block is only a group of main data; ② if there is a corresponding relationship between the auxiliary data and the main data, the auxiliary data does not need to be transmitted, and the minimum data processing block is a group of main data corresponding to the auxiliary data; ③ if the auxiliary data needs to be transmitted, the minimum data processing block is a group of auxiliary data and a corresponding group of main data; the minimum data processing block has a relatively small data volume, which is more conducive to reducing delays and improving the timeliness of overall data transmission; The data transmission and calculation interactive mechanism is: taking the smallest data processing block as the unit, staggered continuous data transmission and data calculation of the entire scene [ / frame / track] are carried out, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of stations; ① Data block transmission: each scene [ / frame / track] data is divided into M blocks and transmitted block by block, completing the transmission task within the station block by block; in N-station transmission, each block of data is transmitted station by station; ② Data block calculation: The data calculation task of each scene [ / frame / track] is assigned to each station according to the plan, and the assigned calculation task is processed in blocks, and the data calculation task is completed station by station; ③ Interlaced transmission calculation: The data of each scene [ / frame / track] is transmitted in blocks to each station, and the data is calculated by blocks at each station. The transmission and calculation tasks of the main data and auxiliary data (or only the main data) are completed by interlaced transmission calculation at each station.

2. The interactive concurrent method according to claim 1, It is characterized in that Also includes: Parallel transmission calculation mathematical model, that is, data is divided into M blocks, N stations are parallelly transmitted and calculated, and the time required for parallel transmission calculation of the whole data is obtained according to the time and space relationship between stations and blocks. Where: N, M are integers greater than or equal to 2, 1≤i≤N, 1≤j≤M; T Ci T is the time consumed by the block data station channel; Di,j T is the time taken to send and receive block data. Di,j =T DSi,j +T DRi,j , T DSi,j T is the time taken to send block data. DRi,j The time taken to receive block data; T Oi,j Calculate the timeout for the block data, T Oi,j =T SCi,j –T Di,j , T SCi,j =T C1i,j +T C2i,j , T SCi,j Calculate the cache time for block data, T C1i,j is the time taken to calculate the block data, T C2i,j is the block data cache time, and T is specified SCi,j -T Di,j When ≤0, T Oi,j =0; the right item 1 of formula (1) is the total time consumption of the channel between stations at all levels, the right item 2 is the total time consumption of sending and receiving the same block data at all levels, the right item 3 is the total timeout of the calculation of the same block data at all levels, and the right item 4 is the total delay of the remaining block data except the first block; combine the same items to get The size of each block of data is the same. If the same-level stations use the same computer to process each block of data, then In the simplified case, T NM It can be summarized as the sum of the total channel time consumed by stations at all levels for one block of data, the total time spent on sending and receiving one block of data at all levels, the total calculation timeout of one block of data at all levels, and the total delay in sending and receiving each block of data except the first block at one station.

3. A parallel processing method for multi-level data transmission and data calculation, It is characterized in that include: 4 parts: system structure, parallel mechanism, system function and system module; The system structure includes: it is composed of multi-level computer sites; inter-station communication adopts long-range or short-range computer communication network, or long-range or short-range non-computer communication network; the long-range or short-range communication between sites and stations constitutes a multi-level site long-range or short-range data communication system and an independently distributed supercomputer cluster; the multi-level sites and their data flows are a multi-level tree-branch static structure; the site data processing is divided into modules such as data transmission and data calculation; the data is transmitted and calculated in a step-by-step staggered parallel processing dynamic mode; The parallel mechanism includes: data block provides an opportunity for concurrent transmission and calculation of large data; the minimum data processing block is divided into three forms: ① a group of main data, ② a group of main data corresponding to auxiliary data, ③ a group of auxiliary data and its corresponding group of main data; data block transmission and calculation station by station, forming an interleaved transmission and calculation mechanism, completing the whole data transmission and calculation task; The system functions include: each station undertakes data transmission and data calculation business; multi-level station data transmission and total task calculation are completed by staggered data transmission calculation mode; the communication mode between stations determines the operation, function and benefit of the system, showing different structural mode functions; the long-distance or short-distance communication between stations determines the combination of multi-level station long-distance data communication, short-distance data communication and independently distributed supercomputing; The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules; The system preset includes: pre-determining the program process form, communication transceiver mode, signal transceiver matching and data exchange mode, pre-compilation of calculation task algorithm, and implementation of file name standardization; The transmission module is divided into a low-level version, an intermediate version and an advanced version, and the version can be selected as needed; the low-level transmission module includes a sending submodule and a receiving submodule; the sending submodule includes: formulating a data sending start mechanism, reading the sending data according to the minimum data processing block, framing or multi-frame data packaging for frame transmission according to the minimum data processing block, continuously sending subsequent data of the data task, cleaning up the sent main and auxiliary data (or only the main data), etc.; the receiving submodule includes: completely receiving each frame of data, unloading the frame and unpacking according to the flags of the frame header and the packet header, saving the received main and auxiliary data (or only the main data) by block according to the flags, and continuously receiving subsequent data of the data task; the intermediate version transmission module includes a sending submodule Block and receiving submodules; the sending submodule includes: formulating a data sending start mechanism, reading the sending data according to the minimum data processing block, providing exchange data to the interface according to the frame plan, continuously sending the subsequent data of the data task, cleaning up the main and auxiliary data (or only the main data) that have been sent, etc.; the receiving submodule includes: reading data according to the interface specification, saving the received main and auxiliary data (or only the main data) by block according to the flag, and continuously receiving the subsequent data of the data task; the advanced version transmission module includes: formulating a data sending start mechanism, reading the sending data from the current station by block, writing it to the storage place of the "receiving" exchange data of the lower station, continuously forwarding the subsequent data of the data task, and cleaning up the main and auxiliary data (or only the main data) that have been sent; The calculation module includes: adjusting the auxiliary data needed, formulating a data calculation startup mechanism, calling the collected parameter data and the "received" auxiliary data, calculating the unprocessed main data in blocks according to the calculation task processing algorithm, continuously calculating the subsequent data of the data task, adjusting the sent auxiliary data, and saving the main and auxiliary data (or only the main data) that have been sent, received, and calculated.

4. The parallel processing method according to claim 3, It is characterized in that Also includes: Remote communication is used between stations, forming a multi-level remote data communication system and a supercomputer cluster with independent distributed computing resources.

5. The parallel processing method according to claim 3, It is characterized in that Also includes: Short-range communication is adopted between stations, forming a multi-level site short-range data communication system and a supercomputer cluster with independent distributed computing resources with each site.

6. The parallel processing method according to claim 3, It is characterized in that Also includes: The stations use both long-range and short-range communications to form a multi-level site local long-range and local short-range data communication system and a supercomputer cluster with independent distributed computing resources.

7. A parallel processing device for multi-level data transmission and data calculation, It is characterized in that include: 4 parts: system structure, parallel mechanism, system function and system module; The system structure includes: it is composed of multi-level computer sites; inter-station communication adopts long-range or short-range computer communication network, or long-range or short-range non-computer communication network; the long-range or short-range communication between sites and stations constitutes a multi-level site long-range or short-range data communication system and an independently distributed supercomputer cluster; the multi-level sites and their data flows are a multi-level tree-branch static structure; the site data processing is divided into modules such as data transmission and data calculation; the data is transmitted and calculated in a step-by-step staggered parallel processing dynamic mode; The parallel mechanism includes: data block provides an opportunity for concurrent transmission and calculation of large data; the minimum data processing block is divided into three forms: ① a group of main data, ② a group of main data corresponding to auxiliary data, ③ a group of auxiliary data and its corresponding group of main data; data block transmission and calculation station by station, forming an interleaved transmission and calculation mechanism, completing the whole data transmission and calculation task; The system functions include: each station undertakes data transmission and data calculation business; multi-level station data transmission and total task calculation are completed by staggered data transmission calculation mode; the communication mode between stations determines the operation, function and benefit of the system, showing different structural mode functions; the long-distance or short-distance communication between stations determines the combination of multi-level station long-distance data communication, short-distance data communication and independently distributed supercomputing; The system modules are system modules of sites at all levels, including system presets, transmission modules and calculation modules; The system preset includes: pre-determining the program process form, communication transceiver mode, signal transceiver matching and data exchange mode, pre-compilation of calculation task algorithm, and implementation of file name standardization; The transmission module is divided into a low-level version, an intermediate version and an advanced version, and the version can be selected as needed; the low-level transmission module includes a sending submodule and a receiving submodule; the sending submodule includes: formulating a data sending start mechanism, reading the sending data according to the minimum data processing block, framing or multi-frame data packaging for frame transmission according to the minimum data processing block, continuously sending subsequent data of the data task, cleaning up the sent main and auxiliary data (or only the main data), etc.; the receiving submodule includes: completely receiving each frame of data, unloading the frame and unpacking according to the flags of the frame header and the packet header, saving the received main and auxiliary data (or only the main data) by block according to the flags, and continuously receiving subsequent data of the data task; the intermediate version transmission module includes a sending submodule Block and receiving submodules; the sending submodule includes: formulating a data sending start mechanism, reading the sending data according to the minimum data processing block, providing exchange data to the interface according to the frame plan, continuously sending the subsequent data of the data task, cleaning up the main and auxiliary data (or only the main data) that have been sent, etc.; the receiving submodule includes: reading data according to the interface specification, saving the received main and auxiliary data (or only the main data) by block according to the flag, and continuously receiving the subsequent data of the data task; the advanced version transmission module includes: formulating a data sending start mechanism, reading the sending data from the current station by block, writing it to the storage place of the "receiving" exchange data of the lower station, continuously forwarding the subsequent data of the data task, and cleaning up the main and auxiliary data (or only the main data) that have been sent; The calculation module includes: adjusting the auxiliary data needed, formulating a data calculation startup mechanism, calling the collected parameter data and the "received" auxiliary data, calculating the unprocessed main data in blocks according to the calculation task processing algorithm, continuously calculating the subsequent data of the data task, adjusting the sent auxiliary data, and saving the main and auxiliary data (or only the main data) that have been sent, received, and calculated.

8. The parallel processing device according to claim 7, It is characterized in that Also includes: Remote communication is used between stations, forming a multi-level remote data communication system and a supercomputer cluster with independent distributed computing resources.

9. The parallel processing device according to claim 7, It is characterized in that Also includes: Short-range communication is adopted between stations, forming a multi-level site short-range data communication system and a supercomputer cluster with independent distributed computing resources with each site.

10. The parallel processing device according to claim 7, It is characterized in that Also includes: The stations use both long-range and short-range communications to form a multi-level site local long-range and local short-range data communication system and a supercomputer cluster with independent distributed computing resources.

Images