Parallel Processing Method and Apparatus for Multi-Level Data Transmission and Data Calculation

Abstract

The present invention relates to a parallel processing method and apparatus for multi-level data transmission and data calculation of big data, including an interactive concurrency method for multi-level data transmission and data calculation. The method and apparatus include four parts: system structure, parallel mechanism, system function, and system module. To solve the timeliness problem of multi-address data transmission and multi-task calculation of big data, three application modes of data communication systems with multi-level sites for remote or short-range or local remote and local short-range and supercomputer clusters with independent distributed computing resources are established. The interactive concurrency method establishes a data block forwarding calculation mechanism, which can complete the concurrent transmission of big data at multi-level sites in relatively little time, and pools the computing power of each site to complete the big data calculation task, with very high data transmission and data calculation efficiency. It provides a parallel mechanism and application mode for multi-address transmission and multi-task calculation of big data, and efficiently solves the different types of requirements for multi-address data communication of big data and the super computing power for multi-task calculation of big data.

Description

Technical Field

[0001] The present invention relates to a parallel processing method and device for multi-level data transmission and data calculation, and belongs to the technical fields of big data, computer networks, data communication, and data calculation. Technical Background

[0002] Current communication technologies and computing technologies are striving to meet and promote the human need for rapid information processing. Information data collected manually, business data generated by technical platforms, and information data obtained by collectors are all growing explosively. For example, mobile phones collect voice, cameras collect images, video recorders and cameras collect audio and video, radars detect targets, satellites detect the earth, and spacecraft detect the universe. All kinds of collectors are collecting information all the time, and the distribution and sharing of information data require processing such as transmission and calculation. According to Shannon's principle, the information transmission speed is related to the channel spectrum bandwidth. To improve the timeliness of data transmission, whether wired or wireless channels must rely on rich spectrum resources. Currently, for data transmission on the ground, optical fibers are mostly used, and for satellite communication, microwave channels are mostly leased. To improve the data processing speed, a computer with strong data processing capabilities needs to be equipped. To improve the data calculation speed, a faster CPU (Central Processing Unit) or GPU (Graphics Processing Unit) and memory components are required. Increasing the memory capacity is beneficial for big data processing. To improve the data reading and saving speed of peripherals, an external memory with a faster read-write access and its interface circuit are required, such as a solid-state drive and its supporting interface. Currently, parallel transmission and parallel computing systems have achieved great success. To further improve the data transmission and computing efficiency, more expensive parallel transmission and parallel computing systems can be used.

[0003] There are multi-level data transmission and data calculation requirements for business data multitasking and hierarchical setting of business organizations. There are multiple factors determining whether multi-level data transmission is needed. For example, the front end needs to process the received data and then forward it; the cost of the ground dedicated line is related to the distance, or a sub-center is set up for forwarding; the cost of satellite receiving equipment is high. After receiving the data sent by the satellite, it is transferred to the ground dedicated line for continuous transmission; and so on. For multi-level data transmission modes, such as the multi-level transmission of satellite remote sensing data, the ground station receives the data sent by the satellite. After being processed by the data center, it is uploaded by the ground station and forwarded by the satellite; for example, for the multi-level transmission of provincial ( / ministry-level) satellite remote sensing systems, the ground station receives the data sent by the satellite. After being processed by the data center, it is gradually forwarded to its affiliated department at the department level, prefecture level, county level, etc. via the ground dedicated line. Before data application, the data not only needs to be distributed at multiple levels, but also needs to perform multi-task data calculation. For example, satellite remote sensing data often needs to undergo a series of data preprocessing, including various data corrections in the time domain, space domain, frequency domain, etc., as well as various land cover inversions, continuously providing various parameter data for the earth simulation system. Previously, multi-level data transmission was carried out by independent stations for continuous transmission. After receiving the file from the previous-level station, it was independently transmitted to the next-level station. In order to improve the timeliness of data transmission, multi-level data transmission mostly adopts in-level memory synchronous forwarding to complete data transmission in real time. If the previous-level station has data calculation tasks (such as data preprocessing), it can use the memory to save the received data file. To avoid wasting time on low-speed external storage, high-speed memory, CPU, or GPU can be used to complete the corresponding data calculation and then forward it to the next-level station. In current multi-level data transmission, when there are data calculation tasks, it is necessary to wait for the data file to be received completely, or it is still in the independent transmission mode between two stations. The waiting time in the forwarding process and the receiving process is relatively long. The data calculation is not carried out while taking advantage of the data transmission time, and the timeliness and comprehensive benefits of the data transmission and data calculation links at multiple levels are not considered.

[0004] Although aspects such as data acquisition and processing are getting faster and faster, currently, the timeliness of multi-level data transmission and data calculation is still far from meeting people's requirements for industry applications. Big data business systems need to cope with the resource pressure of data transmission and data calculation brought about by the centralized processing of business data from various places and the congestion during peak periods. For big data in fields such as military, meteorology, national land, ocean, and environment, in the face of the poor timeliness of local and global simulation systems, in a large number of data processing tasks such as data remote transmission, data time-domain, spatial-domain, and frequency-domain correction, data inversion of various parameter data, and system simulation operations, there is an urgent need for innovation and upgrading in technology. For example, for military target monitoring, earth resource monitoring, and disaster monitoring and prediction, etc., for local and even global data collection, transmission, and calculation, extremely high hopes are placed on the acquisition and processing speed of the collected data, and even the cost is not spared to improve the corresponding processing ability. Based on the relevant research of predecessors, it is even more urgent to construct an efficient parallel processing application mode for multi-level data transmission and data calculation. Different from the big data parallel transmission mode and the big data parallel calculation mode, it can together improve the transmission and calculation efficiency of big data, which conforms to the national strategy and has an inestimable promoting effect on the national economy and social benefits. Summary of the Invention

[0005] The present invention aims at data sources that require multi-address data transmission and multi-task data calculation. To overcome the closedness in the data transmission process, an interactive fusion mechanism for data transmission and data calculation is established, a super application mode for data transmission and data calculation is constructed, the transaction efficiency of data transmission and data calculation is improved, and an efficient parallel processing method and device for multi-level data transmission and data calculation are provided.

[0006] The present invention includes an interactive concurrency method, a parallel processing method, and a device for multi-level data transmission and data calculation. The interactive concurrency method is the parallel mechanism of the system. The parallel processing method includes four aspects: constructing the system structure, adopting the parallel mechanism, realizing the system function, and establishing the system module. The parallel processing device includes four aspects: the system structure of the device, the parallel mechanism, the system function, and the system module.

[0007] Constructing the system structure includes: being composed of multi-level computer sites (nodes); the communication between sites adopts remote and short-range computer communication networks, or remote and short-range non-computer communication networks; the sites and the remote or short-range communication between sites form a multi-level site remote or short-range data communication system and an independently distributed supercomputer cluster; the multi-level sites and their data flow directions are a multi-level tree-branch type static structure; the site data processing is divided into modules such as data transmission and data calculation; a parallel processing dynamic mode of data interleaved transmission and calculation level by level.

[0008] The adoption of the parallel mechanism includes: data chunking provides an opportunity for concurrent transmission and calculation of big data; the minimum data processing chunks are divided into two basic forms: a group of main data with no or no need to transmit auxiliary data; a group of auxiliary data and its corresponding group of main data; the data chunks are transmitted station by station and calculated station by station, forming an interleaved transmission and calculation mechanism to complete the whole-process data transmission and calculation task. The mathematical model of parallel transmission and calculation is also described.

[0009] The realization of system functions includes: each level of stations undertakes data transmission and data calculation services; the multi-level station data transmission and total amount task calculation are completed in the way of data interleaved transmission and calculation step by step; the inter-station communication mode determines the operation, function and benefit of the system, showing different structural mode functions; the remote or short-range communication between stations and among stations determines the combination of remote data communication, short-range data communication and independently distributed supercomputing of multi-level stations.

[0010] The establishment of system modules, namely system modules for each level of stations, includes system preset, transmission module and calculation module.

[0011] The system preset includes: pre-determining the program process form, communication sending and receiving mode, signal sending and receiving matching, data exchange mode, etc., pre-compiling the calculation task algorithm, and implementing the standard naming of file names.

[0012] The transmission module is divided into a low-level version, a middle-level version and a high-level version, and the version is selected according to needs. The low-level version transmission module includes a sending sub-module and a receiving sub-module. The sending sub-module includes: formulating a data sending start mechanism, reading and sending data according to the minimum data processing chunk, packing and framing the data in frames or multiple frames according to the minimum data processing chunk for transmission, continuously sending the subsequent data of the data sending task, and clearing the already sent main and auxiliary data (or only the main data). The receiving sub-module includes: completely receiving each frame of data, unpacking the frame according to the flags of the frame header and packet header, saving the received main and auxiliary data (or only the main data) in chunks according to the flags, and continuously receiving the subsequent data of the data receiving task. The middle-level version transmission module includes a sending sub-module and a receiving sub-module. The sending sub-module includes: formulating a data sending start mechanism, reading and sending data according to the minimum data processing chunk, providing exchange data to the interface according to the frame plan, continuously sending the subsequent data of the data sending task, and clearing the already sent main and auxiliary data (or only the main data). The receiving sub-module includes: reading data according to the interface specification, saving the received main and auxiliary data (or only the main data) in chunks according to the flags, and continuously receiving the subsequent data of the data receiving task. The high-level version transmission module includes: formulating a data sending start mechanism, reading and sending data from the current-level station in chunks, writing it into the "receiving" exchange data storage location of the lower-level station, continuously forwarding the subsequent data of the data forwarding task, and clearing the already sent main and auxiliary data (or only the main data).

[0013] The computing module includes: adjusting the auxiliary data to be used, formulating a data calculation startup mechanism, calling the collected parameter data and the "received" auxiliary data, calculating the unprocessed main data in blocks according to the calculation task processing algorithm, continuously calculating the subsequent data of the data task, adjusting the sent auxiliary data, and saving the main and auxiliary data (or only the main data) that have been sent (or read), received (or written), and calculated.

[0014] Compared with the prior art, the innovation and benefits of the present invention are as follows: ⑴ A parallel mechanism for data transmission and data calculation is established, breaking the closedness of the data transmission process, establishing a block forwarding mechanism for data transmission, providing a convenient way for parallel processing of data transmission and data calculation, providing intensive forwarding opportunities for multi-level station data concurrent transmission, and providing time slots required for data processing for data calculation. ⑵ Utilizing the parallel mechanism of data transmission and data calculation, it is possible to complete the concurrent transmission of big data at multi-level stations in relatively little time, and pool the computing power of each station to complete the big data calculation task, with very high data transmission and data calculation efficiency. ⑶ Using the transmission time and computing resources, three implementation application modes are constructed using different communication methods: ① The two-in-one mode, a multi-level station remote data communication system and a supercomputer cluster with independently distributed computing resources, providing high-efficiency data communication and supercomputing power data calculation at the same time; ② The strong computing mode, a short-range and extremely high-efficiency data communication system, a supercomputer cluster with independently distributed computing resources, and data calculation that exploits the special supercomputing power; ③ The comprehensive mode, a multi-level station local remote and local short-range data communication system, a supercomputer cluster with independently distributed computing resources, providing high-efficiency data communication and high-quality supercomputing power data calculation at the same time. ⑷ The present invention provides a parallel mechanism and its application modes for big data multi-address transmission and multi-task calculation, efficiently solving different types of requirements for big data multi-address data communication and the supercomputing power of big data multi-task calculation.

[0015] After the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, other features and advantages of the present invention will become clearer. Description of the Drawings

[0016] Figure 1 General principle block diagram;

[0017] Figure 2 System structure block diagram;

[0018] Figure 3 Parallel mechanism block diagram;

[0019] Figure 4 System function block diagram;

[0020] Figure 5 System module block diagram;

[0021] Figure 6 Flowchart of the low-level version transmission module

[0022] Figure 7 Flow chart of the intermediate version transmission module;

[0023] Figure 8 Flow chart of the advanced version transmission module;

[0024] Figure 9 Flow chart of the calculation module. Specific implementation manners

[0025] The following further describes from the overall to the specific implementation manners in combination with the accompanying drawings. First, the data is described. On the basis of summarizing the overall implementation scheme, 9 related statements are presented, and the detailed scheme of the present invention is gradually presented in 3 application modes determined by the inter-station communication method. It should be noted that these descriptions of the implementation manners are helpful for understanding the present invention. However, some specific descriptions do not constitute a limitation to the present invention.

[0026] 1 Overall method and device

[0027] The data includes auxiliary data and main data, or only includes main data, which is determined by the specific data source and the data processing tasks of transmission and calculation. Auxiliary data is the parameter data used for data calculation of the main data. In the present invention, the main data is big data, which requires multi-address data transmission and multi-task data calculation and other processing. The data is classified into real-time data and non-real-time data (or historical data) according to timeliness. It is mainly various data collected by ground-based, air-based and space-based, business data generated by the technical platform, and information data collected and statistically analyzed manually, such as satellite and radar detection data, or data in industries such as production and operation. The main data detected by the satellite is remote sensing data, and the auxiliary data is telemetry data; there are many types of satellite detectors, including remote sensing data such as CCD (Charge Coupled Device), WFV (Wide Field of View), and MERSI (Medium Resolution Spectral Imager).

[0028] The overall method and device of the present invention includes four aspects: system structure, parallel mechanism, system function, and system module. See Figure 1 .

[0029] 1.1 System structure

[0030] See Figure 2 , the system structure includes:

[0031] ① Composition of computer sites: Multiple levels of sites ( / nodes) are composed of the first-level station, the second-level station,..., the Nth-level station (N is a positive integer and N≥2), and each site is composed of one or more computers, etc.

[0032] ② Inter-station communication technology system: Remote computer communication network (WAN), or remote non-computer communication network (non-network version remote data communication system), short-range computer communication network (LAN or MAN), and short-range non-computer communication network (non-network version short-range data communication system) are used for inter-station communication. Communication lines include wireless and wired ones, and various specifications of lines can be selected according to needs.

[0033] ③ Overall structure and function system: The sites and inter-station remote communication form a multi-level site remote data communication system and a supercomputer cluster with independent distributed computing resources; or the sites and inter-station short-range communication form a multi-level site short-range data communication system and a supercomputer cluster with independent distributed computing resources; or the sites and inter-station use both remote and short-range communication to form a multi-level site local remote and local short-range data communication system and a supercomputer cluster with independent distributed computing resources.

[0034] ④ Fork-shaped static structure: The multi-level sites and their data flow directions are shown as a multi-level fork-shaped structure, and the multi-level single-fork tree structure is a special form.

[0035] ⑤ Site data processing module: The site system module is divided into modules such as data sending (or reading), receiving (or writing), and computing.

[0036] ⑥ Parallel processing dynamic mode: Data is transmitted concurrency level by level from the first-level station to the Nth-level station, and the data processing tasks assigned to each station are calculated interleavingly level by level.

[0037] 1.2 Parallel mechanism

[0038] See Figure 3 , the parallel mechanism includes:

[0039] ⑴ Minimum data processing block unit. Design the data exchange, data sending (or reading), receiving (or writing), computing and other data processing operations at each level of the station to be processed in units of the minimum data processing block according to the specified format. The whole scene [ / image / track] data is divided into blocks with the minimum data processing block as the processing unit, and the main and auxiliary data are synchronized or transmitted closely following each other, providing an opportunity for concurrent data transmission and calculation, and relying on to achieve parallel processing of data transmission and data calculation, and obtaining high efficiency in data transmission and data calculation.

[0040] ⑵ Setting of the minimum data processing block. The minimum data processing block includes auxiliary data and main data, or only includes main data. ① If there is no corresponding relationship between the auxiliary data and the main data, when determining the minimum data processing block, only consider the influencing factors of its own data calculation efficiency, and the minimum data processing block is just a set of main data; ② If there is a corresponding relationship between the auxiliary data and the main data and the auxiliary data does not need to be transmitted, the minimum data processing block is the main data corresponding to a set of auxiliary data; ③ If the auxiliary data needs to be transmitted, the minimum data processing block is a set of auxiliary data and the corresponding set of main data. The data volume of the minimum data processing block is relatively small, which is more conducive to reducing latency and improving the timeliness of overall data transmission.

[0041] ⑶ Data transmission calculation interaction mechanism. Taking the minimum data processing block as a unit, perform interleaved continuous data transmission and data calculation on the entire scene [ / frame / track] of data, forming a parallel processing linkage mechanism for data transmission and data calculation at each level of the station. ① Data block transmission: Each scene [ / frame / track] of data is divided into M blocks and transmitted block by block, and the in-station transmission task is completed block by block; during the transmission at N stations, each block of data is transmitted station by station. ② Data block calculation: The data calculation tasks for each scene [ / frame / track] of data are assigned to each station according to the plan, and the assigned calculation tasks are processed block by block, and the data calculation tasks are jointly completed by each station. ③ Interleaved transmission calculation: Each scene [ / frame / track] of data is transmitted block by block to each station, and each station calculates the data block by block, and completes the transmission and calculation tasks of the main data and the auxiliary data (or only the main data) in the way of interleaved transmission calculation at each station.

[0042] ⑷ Parallel transmission calculation mathematical model. ① Establish a mathematical model. The data is divided into M blocks and N stations perform parallel transmission calculation. According to the spatio-temporal relationship of transmission and calculation between stations and between blocks, the time taken for the whole-process data parallel transmission calculation is obtained

[0043]

[0044] In the formula: N and M are integers greater than or equal to 2, 1 ≤ i ≤ N, 1 ≤ j ≤ M; T Ci is the channel time consumption between stations for block data; T Di,j is the time for sending and receiving block data, T Di,j = T DSi,j + T DRi,j , T DSi,j is the time for sending block data, T DRi,j is the time for receiving block data; T Oi,j is the calculation timeout for block data, T Oi,j = T SCi,j – T Di,j , T SCi,j = T C1i,j + T C2i,j , T SCi,j is the cache time for block data calculation, T C1i,j is the calculation time for block dataC2i,j When it is for block data caching, and it is stipulated that T SCi,j -T Di,j ≤0, T Oi,j = 0; The right item 1 of formula (1) is the total time-consuming of the channels between stations at all levels, the right item 2 is the total time for receiving and sending the same block data by stations at all levels, the right item 3 is the total timeout for calculating the same block data by stations at all levels, and the right item 4 is the total delay of the sub-block data other than the first block of station 1 for each station on average; Combining like terms, we get

[0045]

[0046] The size of each block of data is the same. If stations at the same level use the same computer to process each block of data, then there is

[0047]

[0048]

[0049] The speed is related to the size of the block data. It shows that reducing the amount of sub-block data will directly reduce the time for receiving and sending data by stations at all levels. Although the size of each block of data is the same, the equipment used by stations at all levels is different, and the data transmission speed may be different. is the total timeout for calculating the same block data by stations at all levels, and is related to the calculation cache speed and data transmission speed of the data tasks of stations at all levels. When T Oi,1 ≤0, it means that the calculation cache of the data task does not delay the data transmission. When T Oi,1 > 0, it means that the calculation cache of the data task causes a delay in the data transmission link. It shows that improving computer resources will directly reduce the data calculation timeout. is the total delay of the sub-block data other than the first block of station 1, and is related to the average data transmission speed of stations at all levels and the amount of block data. The size of each block of data is the same, and the data transmission speed of the same station is the same. In a simplified case, T NM can be summarized as the sum of the total time-consuming of the channels between stations at all levels for 1 block of data, the total time for receiving and sending by stations at all levels for 1 block of data, the total timeout for calculating 1 block of data by stations at all levels, and the total time for receiving and sending of each block of data other than the first block of station 1.

[0050] ③ Substantial benefits of data transmission and calculation. In the whole process of data transmission and calculation, it is in a parallel state of cross-concurrent processing. Evaluate the data transmission and calculation benefits according to the whole process. This mathematical model reflects the data transmission and data calculation timeliness of the whole process. Looking at the overall data transmission and calculation benefits from the mathematical model, the channel delay of each station is the same and not reduced, while only (N + M - 1) block data receiving and sending time is used for a total of (N × M) blocks, and only N block data calculation timeouts are used. The results show that using the data transmission time and calculating data at the same time can not only efficiently increase the data calculation benefits, but also efficiently increase the data transmission benefits of multi-level stations; reducing the amount of sub-block data will correspondingly improve the data transmission benefits of multi-level stations.

[0051] 1.3 System Functions

[0052] See Figure 4 , the system functions include:

[0053] ① In-station service classification function: Each level of station undertakes services such as sending (or reading), receiving (or writing), calculating, and saving main and auxiliary data (or only main data).

[0054] ② Parallel processing dynamic function: Data flows from the first-level station to the Nth-level station. Each level of station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation between stations, then continues to complete the data transmission of multi-level stations, and finally continues to complete the total task calculation with computing power.

[0055] ③ Communication method construction function: The inter-station communication method determines the application mode under the communication technology framework, which is related to the operation, function, and benefit of each part of the system; the communication between stations and between remote or short-range or combined remote and short-range stations respectively shows functions such as a two-in-one mode, a strong computing mode, and a comprehensive mode of parallel processing of data transmission and data calculation of independent distributed computer resources of multi-level stations.

[0056] ④ Overall structure mode function: Two-in-one mode, where each level of station concurrently executes remote data transmission, and uses the super computing power of concurrent computing to dispatch tasks and superimpose distributed computing resources to process big data; or strong computing mode, where each level of station concurrently executes short-range data distribution, and uses the super computing power of concurrent computing to dispatch tasks and superimpose distributed computing resources to process big data; or comprehensive mode, where each level of station concurrently executes part of the remote data transmission and part of the short-range data distribution, and uses the super computing power of concurrent computing to dispatch tasks and superimpose distributed computing resources to process big data.

[0057] 1.4 System Modules

[0058] See Figure 5 , the system modules are the system modules of each level of station, including system preset, transmission module, and calculation module.

[0059] 1.4.1 System Preset

[0060] The system preset includes:

[0061] ① Determine the program process form: It is necessary to first determine the programming route and implementation of multi-task, multi-process, and multi-thread for data processing.

[0062] ② Determine the communication sending and receiving method: It is necessary to first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods between stations for data communication, and require the matching of data sending and receiving communication methods.

[0063] ③ Determine the signal sending and receiving matching: It is necessary to first determine the signal quality and matching of the specific communication line for signal sending and receiving, and appropriately arrange signal low-frequency amplification, intermediate-frequency amplification, and power amplification.

[0064] ④ Determine the data exchange method: Data exchange requires first determining the data exchange method between data sending (or reading), receiving (or writing) and calculation. You can choose arrays, files, variables, or memory, etc. as the storage location for exchanged data.

[0065] ⑤ Prepare computing task algorithms: Different computing tasks are assigned to sites at each level, and the computing task processing algorithms for sites at each level must be prepared first.

[0066] ⑥ Standardized file naming: Data file names involve data call, storage, exchange, etc., and should be named according to rules or regulations.

[0067] 1.4.2 Transmission Module

[0068] The transmission module is divided into low-level, medium-level and high-level versions according to the degree of transparency of the data transmission process to the user. You can choose the version as needed.

[0069] (I) Low-level transmission module

[0070] See also Figure 6 , design the data frame structure, including the entire process of data sending and receiving including data reading, frame packing, data transmission, frame unloading and data saving.

[0071] 1) Sending submodule

[0072] According to the system preset conditions, the main and auxiliary data (or only the main data) are sent in a non-network communication environment.

[0073] ① Data sending start mechanism: When it is determined that there is data that needs to be sent at the exchange data storage location, the site at this level will immediately execute the data sending transaction of this level.

[0074] ② Read the sending data in blocks: Pack the main and auxiliary data (or only the main data) into frames according to the minimum data processing block, read the main data in the "send" storage location where the data calculation has been completed according to the format, and if the auxiliary data needs to be sent, also read the auxiliary data in the corresponding "send" storage location.

[0075] ③ Data packaging frame transmission: The packaging framer or sending software processes data synchronously or immediately after the transmission according to the minimum data processing block. When the main data is transmitted alone, the main data is independently packaged into one or more data packets according to the packet format; or when the main and auxiliary data are transmitted in the same frame, the main and auxiliary data are formed into one data packet according to the packet format; or when the main and auxiliary data are transmitted in separate frames, the auxiliary and main data are formed into one or several independent data packets according to the packet format; or the minimum data processing block is only about a fraction of the frame data area, and the block main and auxiliary data are used as data sub-packets, and the data packet is composed of several sub-packets. Frame the data packet and channel coding according to the frame format, and transmit the minimum data processing block data synchronously or immediately.

[0076] ④ Continuously send subsequent data: Continuously read, package, and frame the subsequent data of the primary and secondary data (or only the primary data), and complete the transmission of the entire scene / frame / track data.

[0077] ⑤ Clean up the sent data: Timely delete the sent, calculated, and saved secondary data and primary data (or only the primary data) in the exchange data storage location to release computer resources.

[0078] 2) Receiving sub-module

[0079] Comply with the system preset conditions and receive the primary and secondary data (or only the primary data) in a non-network communication environment.

[0080] ① Receive full-frame data: Completely receive each frame of data sent by the non-network system.

[0081] ② Receive data frame unloading and unpacking: The frame unloading and unpacking device or receiving software unloads and unpacks each frame of received data according to the flags of the frame header and packet header, distinguishes the primary and secondary data based on the data packet flags, and memorizes the flags for the smallest data processing block data belonging to the same block.

[0082] ③ Save the received data by block: Store the primary and secondary data (or only the primary data) in the smallest data processing block according to the flags, and save the secondary data and primary data (or only the primary data) in the corresponding data storage location.

[0083] ④ Continuously receive subsequent data: Continue to receive, unload, unpack, and save the subsequent data of the primary and secondary data (or only the primary data), and complete the reception of the entire scene / frame / track data.

[0084] (2) Intermediate version transmission module

[0085] See Figure 7 , the user terminal uses the network communication protocol in the network environment or a general data communication system to exchange the primary and secondary data (or only the primary data) according to the specification through the hardware or software interface, and complete the data sending and receiving.

[0086] 1) Sending sub-module

[0087] Comply with the system preset conditions and send the primary and secondary data (or only the primary data) in the network or non-network communication environment.

[0088] ① Data sending startup mechanism: When it is determined that there is data to be sent in the exchange data storage location, this level of site immediately executes the data sending transaction of this level of site.

[0089] ②Read and send data in blocks: According to the master-slave data (or only the main data) packaging and framing plan for the minimum data processing block, read the main data at the "send" storage location where the data calculation has been completed in the specified format. If auxiliary data needs to be sent, also read the auxiliary data at the corresponding "send" storage location.

[0090] ③Provide exchange data to the interface: The user end writes the exchange data to the hardware or software interface specification, and then the network communication protocol or the general data communication machine is responsible for data packaging, framing, and transmission.

[0091] ④Continuously send subsequent data: Continuously read the subsequent data of the master-slave data (or only the main data), provide interface exchange data, and complete the transmission of the entire scene / frame / track data.

[0092] ⑤Clean up the sent data: Timely delete the sent, calculated, and saved auxiliary data and main data (or only the main data) in the exchange data storage location to release computer resources.

[0093] 2) Receiving sub-module

[0094] Comply with the system preset conditions and receive the master-slave data (or only the main data) in a network or non-network communication environment.

[0095] ①Standardize the reading of interface data: The user end reads the unpacked data transmitted to the interface by the network communication protocol or the general data communication machine.

[0096] ②Save the received data in blocks: Store the master-slave data (or only the main data) according to the flag in the minimum data processing block, and save the auxiliary data and the main data (or only the main data) in the corresponding data storage locations.

[0097] ③Continuously receive subsequent data: Continue to receive and save the subsequent data of the master-slave data (or only the main data) to complete the reception of the entire scene / frame / track data.

[0098] (3) Advanced version transmission module

[0099] See Figure 8 , the advanced platform can build transparent access to different addresses in a network environment and directly read and write data as if it were a local machine in the advanced platform environment.

[0100] ①Data sending startup mechanism: When it is determined that there is data to be sent in the exchange data storage location, this level of site immediately executes the data sending transaction of this level of site.

[0101] ②Reading of sending and receiving data: The sending and receiving software takes the minimum data processing block as the processing unit and reads the main data at the "send" storage location where the data calculation has been completed at this level of site in the specified format. If auxiliary data needs to be sent, also read the auxiliary data at the corresponding "send" storage location.

[0102] ③ Writing of received and sent data: Write the auxiliary data and the main data separately in the corresponding storage location of "receiving" at the lower-level station according to the format, or only write the main data in the "receiving" storage location according to the format.

[0103] ④ Continuously forwarding subsequent data: Continuously forward the subsequent blocks of auxiliary data and main data (or only the main data) of this scene / frame / track data to the lower-level station through the network with the smallest data processing block as the processing unit.

[0104] ⑤ Cleaning the sent data: Timely delete the auxiliary data and main data (or only the main data) in the storage locations that have been sent, calculated, and saved to release computer resources.

[0105] 1.4.3 Calculation module

[0106] See Figure 9 The calculation module includes:

[0107] ① Adjusting the auxiliary data as needed: Each level of station appropriately adjusts the required auxiliary data according to the progress of data calculation level by level.

[0108] ② Data calculation start mechanism: When it is judged that there is data to be calculated in the data exchange storage location, this level of station immediately executes the calculation transaction of this level of station.

[0109] ③ Calculating data according to the assigned tasks: Call the various required parameter data collected at this level of station and the auxiliary data in the "receiving" storage location according to the format, and calculate each block of the corresponding main data that has not been processed and is stored in the "receiving" storage location one by one according to the calculation task processing algorithm assigned to this level of station, and save the calculated main data in the "sending" storage location according to the format.

[0110] ④ Continuously calculating subsequent data: Continuously calculate the subsequent blocks of data of the entire scene / frame / track with the smallest data processing block as the processing unit.

[0111] ⑤ Adjusting the sent auxiliary data: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "sending" storage location.

[0112] ⑥ Saving the sent, received, and calculated data: Each level of station timely saves the main data and auxiliary data (or only the main data) of the entire scene / frame / track received (or written) and processed by this level of station in the form of files to the file receiving, processing, and forwarding directory according to the format. The file name complies with the file naming rule and marks the corresponding data level.

[0113] 2 Implementation application mode

[0114] The overall mode of the parallel processing method and device is outlined from four aspects: system structure, parallel mechanism, system function, and system modules. From the perspective of the three high, medium, and low versions of data transmission, obvious differences are shown in the specific implementation forms and technical means. Especially in the inter-station remote and short-range communication methods, it promotes the change in the structural function tendency of the application system of the present invention, and the system structure and function are more specific and prominent in space. The inter-station communication method determines the application mode under the communication technology framework, which is related to the operation, function, and benefit of each part of the system, and forms three application modes.

[0115] In the first application mode, data transmission and data calculation are equally matched. Inter-station remote communication emphasizes taking data transmission tasks as the core and increasing the benefits of data calculation. In terms of system structure, since remote communication is adopted between stations, the stations and the inter-station remote communication form a multi-level station remote data communication system and a supercomputer cluster with independent distributed computing resources. In terms of system function, the stations and the inter-station remote communication show the dual-mode function of parallel processing of data transmission and data calculation of multi-level stations with independent distributed computer resources; in the dual mode, each level of station concurrently executes remote data transmission, and uses concurrent computing to dispatch tasks and superimpose distributed computing resources to process big data with super computing power.

[0116] In the second application mode, data calculation is more prominent in data transmission and data calculation. Inter-station short-range communication is convenient for reducing channel costs and increasing data transmission speed, thereby improving the processing speed of data calculation tasks. In terms of system structure, since short-range communication is adopted between stations, the stations and the inter-station short-range communication form a multi-level station short-range data communication system and a supercomputer cluster with independent distributed computing resources. In terms of system function, the stations and the inter-station short-range communication show the strong calculation mode function of parallel processing of data transmission and data calculation of multi-level stations with independent distributed computer resources. In the strong calculation mode, each level of station concurrently executes short-range data distribution, and uses concurrent computing to dispatch tasks and superimpose distributed computing resources to process big data with super computing power.

[0117] In the third application mode, data transmission is locally prominent and data calculation is locally prominent. Inter-station remote and short-range communication can respectively highlight data transmission and data calculation according to opportunities. In terms of system structure, since remote and short-range communication is adopted between stations, the stations and the same use of remote and short-range communication between stations form a multi-level station local remote and local short-range data communication system, and a supercomputer cluster with independent distributed computing resources. In terms of system function, the stations and the same use of remote and short-range communication between stations show the comprehensive mode function of parallel processing of data transmission and data calculation of multi-level stations with independent distributed computer resources; in the comprehensive mode, each level of station concurrently executes part of the remote data transmission and part of the short-range data distribution, and uses concurrent computing to dispatch tasks and superimpose distributed computing resources to process big data with super computing power.

[0118] 3 Related statements

[0119] 3.1 Site Computers

[0120] The relevant sites of the present invention consist of at least one or more computers. However, the number of site computers is not limited. In practical applications, high-performance configurations of sites should be considered. Computers responsible for data calculation tasks should be equipped with sufficient resources such as high-level CPUs, GPUs, MICs (integrated multi-cores), and memory. Even high-performance parallel computers can be used to improve the computing power of sites. The sites related to the present invention include optimal and non-optimal computer configurations.

[0121] In all levels of sites for data transmission and data calculation, arranging for a certain site not to perform data calculation tasks is a foreseeable structural form of the present invention.

[0122] 3.2 Inter-site Communication Lines

[0123] Inter-site communication lines include wired and wireless communication lines. Wired lines are communication lines that use condensed media with good electrical conductivity or optical conductivity to transmit information, such as optical fibers, twisted pairs, coaxial cables, etc. Wireless lines are communication lines in various frequency bands that use electromagnetic wave spectrum resources to transmit in the air, such as radio waves, microwaves, and infrared rays, etc. Special lines include types such as optical fibers, DDN (Digital Data Network), ISDN (Integrated Services Digital Network), frame relay, packet switching, PSTN (Public Switched Telephone Network), etc. Communication lines can be leased from operators, such as leased dedicated lines, leased satellite or terrestrial microwave channels, etc.; wireless channels in approved frequency bands or specific data communications in public frequency bands can be used; wired communication lines can be built by oneself. Communication lines include future new types of communication lines.

[0124] 3.3 Data Communication Systems

[0125] Data communication systems refer to networked and non-networked data transmission systems, which are divided into three types: network, general, and special. Network data communication systems are various forms of communication systems that conform to computer communication network standards, such as various communication systems of the current IEEE (Institute of Electrical and Electronics Engineers) 802 standards. General data communication systems such as CCSDS AOS (Consultative Committee for Space Data Systems Advanced Orbiting System) or DTMB (Digital Terrestrial Multimedia Broadcast in China), DVB-S2 (Second Generation Satellite Digital TV Broadcasting in Europe), DVB-T2 (Second Generation Terrestrial Digital TV Broadcasting in Europe), etc. with standard interface data transmission systems. Special data communication systems refer to data communication systems designed specifically. From the perspective of the transparency of data transmission for users, general data communication machines use intermediate version transmission modules, and low-level version transmission modules belong to special data communication machines. In order to obtain higher data communication speeds, data parallel transmission systems can also be used in data communication systems.

[0126] 3.4 Calculation Time of the Minimum Data Processing Block

[0127] The remote data transmission speed is relatively very slow compared to the computer's computing speed. In the service processing module, the data calculation time of the smallest data processing block should be less than its data transmission time, aiming to avoid delaying the timeliness of the overall data transmission. In the tasks of data transmission and data calculation, although the data transmission task is primary and crucial, the execution of the data calculation task cannot delay the overall data transmission time. Otherwise, it is necessary to adjust the site data calculation task or select better computer resources to ensure high time efficiency during the data transmission process. In the present invention, the data calculation time of the smallest data processing block can be less than or equal to, or greater than its data transmission time.

[0128] 3.5 Data exchange method

[0129] Data can be exchanged between service modules using file types such as database table files, text files, and binary files, or by using arrays, variable symbols, memory, etc. Considering the read and write speeds, the storage medium for data exchange is preferably memory, and other media such as solid-state disks can also be used.

[0130] 3.6 File naming rules

[0131] Data file names should be named in a standardized manner. For example, the full file name includes the file name and the file extension. The file name includes data source, data longitude and latitude, data date, data time, data category, data encoding, etc., and "-" can be used as a separator. The file extension indicates the file category and is identified conventionally. Taking satellite remote sensing data as an example, for instance, the data source is the satellite name and the detector name, the data longitude and latitude are the longitude and latitude of the monitoring location, the data date is the specific date of remote sensing data acquisition, the data time is the specific time of remote sensing data acquisition, the data category is the data level or data processing batch, the data encoding is the data product number, and the file extension indicates the data file category. The file names, array names, and variable symbols used for data exchange, as well as the file names for storing data, should comply with the standardized naming. The present invention is not limited to specific file naming and naming methods, including using non-standardized file names.

[0132] 3.7 Multi-channel data processing

[0133] In practical applications, the technology of the present invention can be extended to an integrated system to process multiple different sources of data in a time-sharing manner. Taking one path of data as an example, the present invention can be extended to multiple paths of different data from the same source, such as multiple acquisition data from different detectors on the same satellite, or to multiple paths of different data from different sources, such as multiple acquisition data from different detectors on different satellites. The remote sensing data and telemetry data of each satellite are different, and the parameters and formats may be different. The present invention can use the processing mechanisms and technical solutions for multiple sources of data, as well as the time conflict processing mechanism. In the processing of multiple sources of data, a data sending task may be inserted at an intermediate site, which still belongs to the technical solution of the present invention. In the processing of multiple sources of data, some transmitted data may not be arranged for data calculation tasks, which still belongs to the technical solution of the present invention.

[0134] 3.8 Duplex Mode of the Present Invention

[0135] The present invention only describes the parallel processing method and device for multi-level data transmission and data calculation in one direction. If the reverse direction conforms to the relevant description of the present invention, and the forward direction overlaps with part or most or all of the devices or systems in the reverse direction, it belongs to the duplex mode of the present invention and still falls within the protection scope of the present invention.

[0136] 3.9 Related Protection Scope

[0137] From aspects such as system structure, parallel mechanism, system function, and system modules, in combination with the accompanying drawings, the parallel processing method and device for multi-level data transmission and data calculation are described, realizing the efficient parallel processing function of data transmission and data calculation at multi-level sites. The implementation process of the present invention is complex and has many different implementation manners. Some specific manners are used in the description of the embodiments to show actual examples, but the present invention is not limited to the described implementation manners. If different implementation forms are designed according to the teachings of the present invention, without departing from the principles and spirit of the present invention, these changes, modifications, substitutions, and variations to the implementation manners still fall within the protection scope of the present invention.

[0138] Example statement: 2 examples are given for each mode, with a total of 6 embodiments

[0139] 4 Example 1: Example 1 of Mode 1. Taking the real-time data of the CCD detector of the YL1 (Example No. 1) satellite as an example, a parallel processing method and device for multi-level data transmission and data calculation of a CCSDS AOS&XTCE (XML Telemetry and Command Exchange) remote general data communication machine for transmitting data.

[0140] In this embodiment, the YL1 satellite is in a sun-synchronous orbit. The real-time data comes from the CCD detector when the YL1 satellite passes over the ground station, including auxiliary data and main data. The auxiliary data is telemetry data, which includes recorded data such as satellite orbit parameters, satellite positioning, and on-board time, as well as monitoring data for the attitude control system, communication system, power supply system, etc. The main data is the remote sensing data collected by the on-board CCD detector. During the transmission process of satellite remote sensing data, data correction is arranged, including system positioning, system projection, radiometric calibration and correction, conversion to apparent reflectance, and geometric precision correction, for daily monitoring use by the site department. Taking the real-time data of the YL1 satellite CCD detector as an example, this paper elaborates on the parallel processing method and device for multi-level data transmission and data calculation using a CCSDS AOS&XTCE remote general data communication machine, including system structure, parallel mechanism, system functions, and system modules, etc.

[0141] 4.1 System Structure

[0142] The system structure includes:

[0143] ① Composition of computer sites: The site consists of a total of 5 levels of stations from the 1st level station to the 5th level station. Most sites are composed of 3 computers to form a network.

[0144] ② Inter-station communication technology system: The stations are connected by a remote data communication system. A CCSDS AOS&XTCE remote general data communication machine is used, and a satellite channel, special line, or wireless channel is leased or applied for.

[0145] ③ Overall structure and function system: The 5-level stations and inter-station remote communication form a multi-level station remote data communication system and a supercomputer cluster with independent distributed computing resources.

[0146] ④ Fork-shaped static structure: The 5-level stations form a multi-level one-fork tree structure.

[0147] ⑤ Site data processing module: The site system module is divided into modules such as data sending, receiving, and calculation.

[0148] ⑥ Parallel processing dynamic mode: Data is transmitted concurrency level by level from the 1st level station to the 5th level station, and the data processing tasks assigned to each station are calculated alternately level by level.

[0149] 4.2 Parallel Mechanism

[0150] The parallel mechanism includes:

[0151] ⑴ The minimum data processing block unit. For the data processing services such as data exchange, data sending, receiving, and calculation at each level of the station, they are all processed in units of the minimum data processing block in accordance with the specified format. The entire scene data is segmented with the minimum data processing block as the processing unit, and the primary and auxiliary data are synchronized or transmitted closely following each other, providing an opportunity for concurrent data transmission calculation. Depending on this, the parallel processing of data transmission and data calculation is realized, and high efficiency of data transmission and data calculation is obtained.

[0152] ⑵ The setting of the minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. According to the actual data collection situation, determine the corresponding relationship and quantity between the telemetry data and remote sensing data collected on the satellite in this instance. Integrate the telemetry data related to the correction of remote sensing data of the CCD detector into 1 frame for transmission. This frame of telemetry data corresponds to 30 frames of remote sensing data, and determine that the minimum data processing block is 1 frame of telemetry data and 30 frames of remote sensing data. The data volume of the minimum data processing block is relatively small, which is more conducive to reducing latency and improving the timeliness of overall data transmission.

[0153] ⑶ The data transmission calculation interaction mechanism. Taking the minimum data processing block as the unit, perform interleaved continuous data transmission and data calculation on the entire scene data, forming a parallel processing linkage mechanism for data transmission and data calculation at each level of the station. ① Data block transmission: Each scene of data is divided into M blocks and transmitted block by block, and the in-station transmission task is completed block by block; during the transmission at N stations, each block of data is transmitted station by station. ② Data block calculation: The calculation tasks of each scene of data are assigned to each station according to the plan, and the calculation tasks are processed by block, and each station jointly completes the data calculation task. ③ Interleaved transmission calculation: Each scene of data is divided into blocks and transmitted station by station, and each station calculates the data by block, and completes the transmission and calculation tasks of the main data and auxiliary data in the way of interleaved transmission calculation at each station.

[0154] 4.3 System functions

[0155] The system functions include:

[0156] ① In-station service classification function: Each level of the station undertakes services such as sending, receiving, calculating, and storing primary and auxiliary data.

[0157] ② Parallel processing dynamic function: Data flows from the first-level station to the fifth-level station. Each level of the station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation at each station, successively complete the data transmission of multiple levels of stations, and successively complete the calculation tasks of data correction from level 0 to level 4 with computing power.

[0158] ③ Communication mode construction function: The inter-station remote communication mode determines the application mode under the communication technology framework, which is related to the operation, function, and benefit of each part of the system; the site and inter-station remote communication show the dual-mode function of parallel processing of data transmission and data calculation of computer resources independently distributed at multiple levels of sites.

[0159] ④ Overall structure mode function: two-in-one mode. Each level of station concurrently executes remote data transmission, and uses super computing power that distributes tasks through concurrent computing and superimposes distributed computing resources to process big data.

[0160] 4.4 System modules

[0161] The system modules mentioned above refer to the system modules of each level of station, including system preset, transmission module, and computing module.

[0162] 4.4.1 System preset

[0163] The system preset includes:

[0164] ① Determine the program process form: For data processing, it is necessary to first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading.

[0165] ② Determine the communication transceiver method: For data communication, it is necessary to first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods between stations. It is required that the data sending and receiving communication methods be matched.

[0166] ③ Determine the signal transceiver matching: For signal transceiver, it is necessary to first determine the signal quality and matching of the specific communication line, and appropriately arrange signal low-frequency amplification, intermediate-frequency amplification, and power amplification.

[0167] ④ Determine the data exchange method: For data exchange, it is necessary to first determine the data exchange method between data sending, receiving, and computing, and select an array as the storage location for the exchanged data.

[0168] ⑤ Compile the calculation task algorithm: Each level of station is assigned different calculation tasks, and it is necessary to first compile the calculation task processing algorithm for each level of station.

[0169] ⑥ Standardize the naming of file names: Data file names are involved in data calling, saving, exchanging, etc., and should be named according to rules or regulations.

[0170] 4.4.2 Transmission module

[0171] Select to use the intermediate version of the transmission module. The user side uses the CCSDS AOS&XTCE general data communication system to exchange primary and secondary data according to the specification through the hardware interface to complete data sending and receiving.

[0172] The sending sub-module includes:

[0173] Comply with the system preset conditions and send primary and secondary data in a non-network communication environment.

[0174] ① Data sending start mechanism: When it is judged that there is data to be sent in the storage location of the exchanged data, this level of station immediately executes the data sending transaction of this level of station.

[0175] ②Read and send data in chunks: According to the master-slave data packaging and framing plan for the minimum data processing block, read the main data stored in the "send" location where the data calculation has been completed in the specified format. If auxiliary data needs to be sent, also read the corresponding auxiliary data in the "send" location.

[0176] ③Provide exchange data to the interface: The user end writes the exchange data into the hardware interface specification, and then the general data communication machine is responsible for data packaging, framing, and transmission.

[0177] ④Continuously send subsequent data: Continuously read the subsequent data of the master-slave data, provide interface exchange data, and complete the transmission of the entire scene data.

[0178] ⑤Clean up the sent data: Timely delete the sent, calculated, and saved auxiliary data and main data in the exchange data storage location to release computer resources.

[0179] The receiving sub-module includes:

[0180] Comply with the system preset conditions and receive the master-slave data in a network or non-network communication environment.

[0181] ①Standardize the reading of interface data: The user end reads the frame-unpacking data transmitted from the data deframer to the interface.

[0182] ②Save the received data in chunks: Store the master-slave data in the minimum data processing block according to the flag, and save the auxiliary data and main data in the corresponding data storage locations.

[0183] ③Continuously receive subsequent data: Continue to receive and save the subsequent data of the master-slave data to complete the reception of the entire scene data.

[0184] 4.4.3 Calculation Module

[0185] The calculation module includes:

[0186] ①Adjust the auxiliary data as needed: Each level of station appropriately adjusts the required auxiliary data according to the progress of data calculation step by step.

[0187] ②Data calculation startup mechanism: When it is determined that there is data to be calculated in the exchange data storage location, the station at this level immediately executes the calculation tasks at this level.

[0188] ③Calculate the data according to the assigned tasks: Call the various required parameter data collected at this level of station and the auxiliary data in the "receive" storage location in the specified format, and process the corresponding main data that has not been processed and stored in the "receive" storage location block by block according to the calculation task processing algorithm assigned to this level of station, and save the calculated main data in the "send" storage location in the specified format.

[0189] ④ Continuously calculate subsequent data: Continuously calculate the data of each subsequent block of the entire scene with the minimum data processing block as the processing unit.

[0190] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "send" storage location.

[0191] ⑥ Save the sent, received, and calculated data: Each level of station timely saves the received and processed data of the entire scene main data and auxiliary data at the current level in the file receiving, processing, and forwarding directory in the form of files. The file name complies with the file naming rules and marks the corresponding data level. 5 Example two: Example of mode 1. Taking the non-real-time data of the WFV detector of the YL2 (sample No. 2) satellite as an example, a multi-level data transmission and parallel processing method and device for data calculation with data transmitted through a computer wide area network DDN dedicated line.

[0192] In this example, the YL2 satellite is in a sun-synchronous orbit, and the non-real-time data comes from the WFV detector of the YL2 satellite when it passes over the ground station, including auxiliary data and main data. The auxiliary data is telemetry data, which includes recorded data such as satellite orbit parameters, satellite positioning, and on-board time, and monitoring data for the attitude control system, communication system, power supply system, etc. The main data is the remote sensing data collected by the on-board WFV detector. In the satellite remote sensing data transmission process, data correction is arranged, including system positioning, system projection, radiometric calibration, conversion to apparent reflectance, and geometric precision correction, for daily monitoring by the site department. Taking the non-real-time data of the WFV detector of the YL2 satellite as an example, a multi-level data transmission and parallel processing method and device for data calculation with data transmitted through a wide area network DDN dedicated line are described, including system structure, parallel mechanism, system functions, and system modules, etc.

[0193] 5.1 System structure

[0194] The system structure includes:

[0195] ① Composition of computer sites: The sites are composed of a total of 4 levels of stations from the 1st level station to the 4th level station, and each level of station consists of 3 computers forming a network.

[0196] ② Inter-station communication technology system: The stations are connected by a remote computer communication network (wide area network). The inter-station remote network version data transmission system consists of two end networks and an intermediate subnet, and leases a DDN dedicated line.

[0197] ③ Overall structure function system: The 4-level stations and the inter-station remote communication form a multi-level station remote data communication system and a supercomputer cluster with independent distributed computing resources.

[0198] ④ Fork-shaped static structure: The 4-level stations form a multi-level one-fork tree structure.

[0199] ⑤ Site data processing module: The site system module is divided into data reading, writing, calculation and other modules.

[0200] ⑥ Parallel processing dynamic mode: Data is transmitted concurrently from the first-level station to the fourth-level station, and the data processing tasks assigned to each station are calculated in a staggered manner at each level.

[0201] 5.2 Parallel mechanism

[0202] The parallel mechanism includes:

[0203] ⑴ Minimum data processing block unit. Design the data processing operations such as data exchange, data sending, receiving, and calculation at each level of the station to be processed in units of the minimum data processing block in accordance with the specified format. Use the minimum data processing block as the processing unit to divide the entire scene data into blocks, synchronize or transmit the primary and auxiliary data closely, provide an opportunity for concurrent data transmission and calculation, and rely on realizing the parallel processing of data transmission and data calculation to obtain high efficiency in data transmission and data calculation.

[0204] ⑵ Setting of the minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. Determine the corresponding relationship and quantity between the telemetry data and remote sensing data collected on the satellite in this instance according to the actual data collection situation. Integrate the telemetry data related to the calibration of the remote sensing data of the WFV detector into 1 frame for transmission. This frame of telemetry data corresponds to 50 frames of remote sensing data, and determine that the minimum data processing block is 1 frame of telemetry data and 50 frames of remote sensing data. The data volume of the minimum data processing block is relatively small, which is more conducive to reducing latency and improving the timeliness of overall data transmission.

[0205] ⑶ Data transmission and calculation interaction mechanism. Using the minimum data processing block as the unit, continuously transmit and calculate the data of the entire scene data in a staggered manner to form a parallel processing linkage mechanism for data transmission and data calculation at each level of the station. ① Data block transmission: Each scene of data is divided into M blocks and transmitted block by block, and the in-station transmission tasks are completed block by block; during the transmission at station N, each block of data is transmitted station by station. ② Data block calculation: The calculation tasks of each scene of data are assigned to each station according to the plan, and the calculation tasks are processed by block, and each station jointly completes the data calculation tasks. ③ Staggered transmission and calculation: Each scene of data is divided into blocks and transmitted station by station, and each station calculates the data by block, and completes the transmission and calculation tasks of the main data and auxiliary data in the way of staggered transmission and calculation at each station.

[0206] 5.3 System functions

[0207] The system functions include:

[0208] ① In-station service classification function: Each level of station undertakes services such as reading, writing, calculating, and storing primary and auxiliary data.

[0209] ② Parallel processing dynamic function: Data flows from the first-level station to the fourth-level station. Each level of station relies on a multi-level tree-branch structure to complete the interleaved data transmission and data calculation at each station, then continues to complete the multi-level station data transmission, and finally completes the data correction task calculation from level 0 to level 4 with computing power.

[0210] ③ Communication mode construction function: The inter-station remote communication mode determines the application mode under the communication technology framework, which is related to the operation, function, and benefit of each part of the system; the site and inter-station remote communication show the dual-mode function of parallel processing of data transmission and data calculation of independent distributed computer resources at multiple levels of stations.

[0211] ④ Overall structure mode function: Dual-mode. Each level of station concurrently executes remote data transmission, and uses super computing power of concurrently computing to dispatch tasks and superimpose distributed computing resources to process big data.

[0212] 5.4 System module

[0213] The system module mentioned above refers to the system modules of each level of station, including system preset, transmission module, and calculation module.

[0214] 5.4.1 System preset

[0215] The system preset includes:

[0216] ① Determine the program process form: Data processing needs to first determine the programming route and implementation of multi-task, multi-process, and multi-thread.

[0217] ② Determine the communication transceiver mode: Data communication needs to first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods between stations, and requires the matching of data sending and receiving communication methods.

[0218] ③ Determine the signal transceiver matching: Signal transceiver needs to first determine the signal quality and matching of the specific communication line, and appropriately arrange signal low-frequency amplification, intermediate-frequency amplification, and power amplification.

[0219] ④ Determine the data exchange mode: Data exchange needs to first determine the data exchange mode between data reading, writing, and calculation, and select a file as the storage location for exchanged data.

[0220] ⑤ Compile the calculation task algorithm: Different calculation tasks are assigned to each level of station, and the calculation task processing algorithms of each level of station need to be compiled first.

[0221] ⑥ Standardize the naming of file names: Data file names are involved in data calling, saving, exchanging, etc., and should be named according to rules or regulations.

[0222] 5.4.2 Transmission module

[0223] Select to use the advanced transmission module. The advanced platform can build transparent access to different addresses in a network environment and directly read and write data as if it were a local machine in the advanced platform environment.

[0224] ① Data sending startup mechanism: When it is determined that there is data to be sent in the data exchange storage location, this level of site immediately executes the data sending transaction of this level of site.

[0225] ② Reading of sending and receiving data: The sending and receiving software takes the smallest data processing block as the processing unit and reads the main data in the "sending" storage location where the data calculation of this level of site has been completed according to the format. If auxiliary data needs to be sent, the auxiliary data in the corresponding "sending" storage location is also read.

[0226] ③ Writing of sending and receiving data: Write the auxiliary data and main data to the corresponding "receiving" storage location of the lower-level site according to the format respectively.

[0227] ④ Continuously forwarding subsequent data: Continuously forward the subsequent blocks of auxiliary data and main data of this scene data to the lower-level site through the network with the smallest data processing block as the processing unit.

[0228] ⑤ Clearing the sent data: Immediately delete the auxiliary data and main data in the storage location that has been sent, calculated, and saved, and release computer resources.

[0229] 5.4.3 Calculation module

[0230] The calculation module includes:

[0231] ① Adjusting auxiliary data as needed: Each level of site appropriately adjusts the required auxiliary data according to the progress of data calculation level by level.

[0232] ② Data calculation startup mechanism: When it is determined that there is data to be calculated in the data exchange storage location, this level of site immediately executes the calculation transaction of this level of site.

[0233] ③ Calculating data according to the assigned tasks: Call the various required parameter data collected by this level of site and the auxiliary data in the "receiving" storage location according to the format, and process the corresponding main data that has not been processed and is stored in the "receiving" storage location block by block according to the calculation task processing algorithm assigned to this level of site, and save the calculated main data in the "sending" storage location according to the format.

[0234] ④ Continuously calculating subsequent data: Continuously calculate the subsequent blocks of the entire scene data with the smallest data processing block as the processing unit.

[0235] ⑤ Adjusting the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "sending" storage location.

[0236] ⑥ Save the sent, received, and calculated data: Each level of station saves the whole-scene main data and auxiliary data written to and processed by this level of station in a timely manner in the file receiving, processing, and forwarding directory in the form of files. The file name complies with the file naming rules and marks the corresponding data level. Example 3: Example of Mode 2. Taking the real-time data of the CCD detector of the YL1 (Sample No. 1) satellite as an example, a parallel processing method and device for multi-level data transmission and data calculation with direct serial data transmission and reception through the computer parallel port.

[0237] In this embodiment, the YL1 satellite is in a sun-synchronous orbit, and the real-time data comes from the CCD detector of the YL1 satellite when it passes over the ground station, including auxiliary data and main data. The auxiliary data is telemetry data, which includes recorded data such as satellite orbit parameters, satellite positioning, and on-board time, and monitoring data for attitude control systems, communication systems, power systems, etc. The main data is the remote sensing data collected by the on-board CCD detector. The satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiometric calibration and correction, conversion to apparent reflectance, and geometric precision correction, to provide data services for users' daily monitoring in a timely manner. Taking the real-time data of the CCD detector of the YL1 satellite as an example, a parallel processing method and device for multi-level data transmission and data calculation with direct serial data transmission and reception through the computer parallel port are described, including system structure, parallel mechanism, system functions, and system modules.

[0238] 6.1 System Structure

[0239] The system structure includes:

[0240] ① Composition of computer sites: The site ( / node) consists of a total of 5 levels of stations from the 1st level to the 5th level, and most sites are composed of 3 computers to form a network.

[0241] ② Inter-station communication technology system: The stations are connected by a short-range non-computer communication network (non-network version short-range data communication system). The short-range non-network version data transmission system uses parallel port direct connection communication that is not included in the network technology standard.

[0242] ③ Overall structure and function system: The 5-level stations and the short-range inter-station communication constitute a multi-level station short-range data communication system and a supercomputer cluster with independent distributed computing resources.

[0243] ④ Fork-shaped static structure: The 5-level stations form a multi-level one-fork tree structure.

[0244] ⑤ Site data processing module: The site system module is divided into modules such as data sending, receiving, and calculation.

[0245] ⑥ Parallel processing dynamic mode: The data is transmitted concurrency from the 1st level station to the 5th level station, and the data processing tasks assigned to each station are calculated alternately level by level.

[0246] 6.2 Parallel Mechanism

[0247] The parallel mechanism includes:

[0248] (1) Minimum data processing block unit. For data processing operations such as data exchange, data sending, receiving, and calculation at each level of the station, they are all processed in units of the minimum data processing block in a specified format. The entire scene of data is divided into blocks with the minimum data processing block as the processing unit, and the primary and auxiliary data are synchronized or transmitted closely following each other, providing an opportunity for concurrent data transmission and calculation. Relying on this, parallel processing of data transmission and data calculation is achieved, and high efficiency of data transmission and data calculation is obtained.

[0249] (2) Setting of the minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is the auxiliary data, and remote sensing data is the main data. According to the actual data collection situation, determine the corresponding relationship and quantity between the telemetry data and remote sensing data collected on the satellite in this example. Integrate the telemetry data related to the correction of remote sensing data of the CCD detector into 1 frame for transmission. This frame of telemetry data corresponds to 30 frames of remote sensing data, and determine that the minimum data processing block is 1 frame of telemetry data and 30 frames of remote sensing data. The data volume of the minimum data processing block is relatively small, which is more conducive to reducing latency and improving the timeliness of overall data transmission.

[0250] (3) Data transmission calculation interaction mechanism. Taking the minimum data processing block as the unit, perform interleaved and continuous data transmission and data calculation on the entire scene of data, forming a parallel processing linkage mechanism for data transmission and data calculation at each level of the station. ① Data block transmission: Each scene of data is divided into M blocks and transmitted block by block, and the in-station transmission task is completed block by block; during the transmission at N stations, each block of data is transmitted station by station. ② Data block calculation: The calculation tasks of each scene of data are assigned to each station according to the plan, and the calculation tasks are processed by block, and each station jointly completes the data calculation task. ③ Interleaved transmission and calculation: Each scene of data is divided into blocks and transmitted station by station, and each station calculates the data by block, and completes the transmission and calculation tasks of the main data and auxiliary data in the way of interleaved transmission and calculation at each station.

[0251] 6.3 System functions

[0252] The system functions include:

[0253] ① In-station service classification function: Each level of the station undertakes services such as sending, receiving, calculating, and storing primary and auxiliary data.

[0254] ② Parallel processing dynamic function: Data flows from the first-level station to the fifth-level station. Each level of the station relies on a multi-level tree-branch structure to complete interleaved data transmission and data calculation at each station, then continues to complete multi-level station data transmission, and then uses the computing power to complete the calculation tasks of data correction at levels 0-4.

[0255] ③ Communication mode structure function: The short-range communication mode between stations determines the application mode under the communication technology framework, which is related to the operation, function, and benefit of each part of the system; the stations and the short-range communication between stations show the strong computing mode function of parallel processing of data transmission and data calculation of independent distributed computer resources at multiple levels of stations.

[0256] ④ Overall structure mode function: Strong computing mode, where each level of station concurrently executes short-range data distribution, and uses the super computing power of concurrent computing to dispatch tasks and superimpose distributed computing resources to process big data.

[0257] 6.4 System modules

[0258] The system modules mentioned above are the system modules of each level of station, including system preset, transmission module, and calculation module.

[0259] 6.4.1 System preset

[0260] The system preset includes:

[0261] ① Determine the program process form: For data processing, it is necessary to first determine the programming routes and implementations of multi-tasking, multi-process, and multi-threading.

[0262] ② Determine the communication sending and receiving methods: For data communication, it is necessary to first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods between stations, and require the matching of data sending and receiving communication methods.

[0263] ③ Determine the signal sending and receiving matching: For signal sending and receiving, it is necessary to first determine the signal quality and matching of the specific communication line, and appropriately arrange signal low-frequency amplification, intermediate-frequency amplification, and power amplification.

[0264] ④ Determine the data exchange method: For data exchange, it is necessary to first determine the data exchange method between data sending, receiving, and calculation, and select an array as the storage location for the exchanged data.

[0265] ⑤ Compile the calculation task algorithm: Different calculation tasks are assigned to each level of station, and the calculation task processing algorithms for each level of station need to be compiled first.

[0266] ⑥ Standardize the naming of file names: The data file names are related to data calling, saving, exchanging, etc., and should be named according to rules or regulations.

[0267] 6.4.2 Transmission module

[0268] Select and use the low-level version of the transmission module. Design the data frame structure for the entire process of data sending and receiving, including data reading, packaging and framing, data transmission, frame unloading and unpacking, and data saving.

[0269] The sending sub-module includes:

[0270] Comply with the system preset conditions and send the main and auxiliary data in a non-network communication environment.

[0271] ①Data sending startup mechanism: When it is determined that there is data to be sent in the data exchange storage location, the station at this level immediately executes the data sending transaction of this level station.

[0272] ②Read and send data in blocks: According to the master-slave data packaging and framing plan of the minimum data processing block, read the main body data of the "send" storage location where the data calculation has been completed in the specified format. If auxiliary data needs to be sent, also read the corresponding auxiliary data of the "send" storage location.

[0273] ③Data packaging, framing and transmission: Process data synchronously or following the transmission according to the minimum data processing block. When the master-slave data is transmitted in the same frame, form 1 data packet for the master-slave data together according to the packet format; or when the master-slave data is transmitted in separate frames, form 1 or several independent data packets for the slave-master data according to the packet format. Package and frame the data packets and channel coding according to the frame format, and synchronously or following the transmission, transmit the minimum data processing block data.

[0274] ④Continuously send subsequent data: Continuously read, package and frame the subsequent data of the master-slave data to complete the transmission of the entire scene data.

[0275] ⑤Clean up the sent data: Timely delete the sent, calculated and saved auxiliary data and main body data in the data exchange storage location to release computer resources.

[0276] The receiving sub-module includes:

[0277] Comply with the system preset conditions and receive the master-slave data in a non-network communication environment.

[0278] ①Full-frame data reception: Completely receive each frame of data sent by the non-network system.

[0279] ②Unframe and unpack the received data: Unframe and unpack each frame of received data according to the flags of the frame header and packet header, distinguish the master-slave data according to the data packet flag, and remember the flag for the data of the minimum data processing block belonging to the same block.

[0280] ③Save the received data in blocks: Store the master-slave data according to the minimum data processing block according to the flag, and save the auxiliary data and main body data in the corresponding data storage location.

[0281] ④Continuously receive subsequent data: Continue to receive, unframe, unpack and save the subsequent data of the master-slave data to complete the reception of the entire scene data.

[0282] 6.4.3 Calculation module

[0283] The said calculation module includes:

[0284] ①Adjust the auxiliary data as needed: Each level station appropriately adjusts the required auxiliary data according to the progress of data calculation level by level.

[0285] ②Data calculation startup mechanism: When it is determined that there is data to be calculated at the data exchange storage location, the station at this level immediately executes the calculation transaction of this level station.

[0286] ③Calculate data according to the assigned tasks: Call various required parameter data collected by this level station and the auxiliary data in the "received" storage location according to the format, and process the corresponding main data that has not been processed and is stored in the "received" storage location block by block according to the calculation task processing algorithm assigned to this level station, and save the calculated main data in the "sent" storage location according to the format.

[0287] ④Continuously calculate subsequent data: Continuously calculate all subsequent blocks of data for the entire scene with the smallest data processing block as the processing unit.

[0288] ⑤Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "sent" storage location.

[0289] ⑥Save the sent, received, and calculated data: Each level station saves the entire scene of main data and auxiliary data received and processed by this level station in the file reception, processing, and forwarding directory in the form of files in a timely manner according to the format. The file name complies with the file naming rules and marks the corresponding data level. 7 Example 4: Example 2 of Mode 2. Taking the non-real-time data of the CCD detector of the YL1 (Sample No. 1) satellite as an example, a multi-level data transmission and parallel processing method and device for data calculation of data transmitted through a computer metropolitan area network.

[0290] In this embodiment, the YL1 satellite is in a sun-synchronous orbit, and the data comes from the real-time data of the CCD detector when the YL1 satellite passes over the ground station, including auxiliary data and main data. The auxiliary data is telemetry data, which includes recorded data such as satellite orbit parameters, satellite positioning, and on-board time, and monitoring data for the attitude control system, communication system, power supply system, etc. The main data is the remote sensing data collected by the on-board CCD detector. The satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiometric calibration, conversion to apparent reflectance, and geometric precision correction, to provide data services for users' daily monitoring in a timely manner. Taking the non-real-time data of the CCD detector of the YL1 satellite as an example, this paper elaborates on a multi-level data transmission and parallel processing method and device for data calculation of data transmitted through a metropolitan area network, including system structure, parallel mechanism, system functions, and system modules.

[0291] 7.1 System Structure

[0292] The system structure includes:

[0293] ①Composition of computer sites: The sites ( / nodes) are composed of a total of 4 levels of stations from the 1st level station to the 4th level station, and each level station consists of 3 computers forming a network.

[0294] ② Inter-station communication technology system: The stations are connected by a short-range computer communication network (metropolitan area network). The inter-station short-range network version data transmission system consists of two end networks and an intermediate subnet, and leases DDN dedicated lines.

[0295] ③ Overall structure and function system: Four-level stations and inter-station short-range communication constitute a multi-level station short-range data communication system and a supercomputer cluster with independent distributed computing resources.

[0296] ④ Fork-shaped static structure: The four-level stations form a multi-level one-fork tree structure.

[0297] ⑤ Station data processing module: The station system module is divided into modules such as data reading, writing, and calculation.

[0298] ⑥ Parallel processing dynamic mode: Data is transmitted concurrency level by level from the first-level station to the fourth-level station, and the data processing tasks assigned to each station are calculated staggeredly level by level.

[0299] 7.2 Parallel mechanism

[0300] The parallel mechanism includes:

[0301] ⑴ Minimum data processing block unit. Design the data processing services such as data exchange, data sending, receiving, and calculation at each level of the station to be processed in units of the minimum data processing block in accordance with the specified format. Use the minimum data processing block as the processing unit to divide the entire scene data into blocks, synchronize or transmit the main and auxiliary data closely, provide an opportunity for concurrent data transmission and calculation, and rely on it to achieve parallel processing of data transmission and data calculation, obtaining high efficiency in data transmission and data calculation.

[0302] ⑵ Setting of the minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. Determine the corresponding relationship and quantity between the telemetry data and remote sensing data collected on the satellite in this instance according to the actual data collection situation. Integrate the telemetry data related to the correction of remote sensing data by the CCD detector into one frame for transmission. This frame of telemetry data corresponds to 30 frames of remote sensing data, and determine the minimum data processing block as one frame of telemetry data and 30 frames of remote sensing data. The data volume of the minimum data processing block is relatively small, which is more conducive to reducing latency and improving the timeliness of overall data transmission.

[0303] ⑶ Data transmission calculation interaction mechanism. Taking the minimum data processing block as a unit, staggered and continuous data transmission and data calculation are performed on the entire scene data, forming a parallel processing linkage mechanism for data transmission and data calculation at each level of the site. ① Data block transmission: Each scene of data is divided into M blocks and transmitted block by block, and the in-site transmission tasks are completed block by block; during the transmission at N stations, each block of data is transmitted station by station. ② Data block calculation: The calculation tasks of each scene of data are assigned to each station according to the plan, and the calculation tasks are processed block by block, and each station jointly completes the data calculation tasks. ③ Staggered transmission calculation: Each scene of data is transmitted block by block to each station, and each station calculates the data block by block, and the transmission and calculation tasks of the main data and auxiliary data are completed in the way of staggered transmission and calculation at each station.

[0304] 7.3 System functions

[0305] The system functions include:

[0306] ① In-site service classification function: Each level of station undertakes services such as reading, writing, calculating, and saving main and auxiliary data.

[0307] ② Parallel processing dynamic function: Data flows from the first-level station to the fourth-level station, and each level of site depends on the multi-level tree-branch structure to complete the staggered data transmission and data calculation at each station, continuously completes the data transmission at multiple levels of stations, and continuously completes the calculation of the data correction tasks from level 0 to level 4 with computing power.

[0308] ③ Communication mode construction function: The inter-station communication mode determines the application mode under the communication technology framework, which is related to the operation, function, and benefit of each part of the system; the short-range communication between stations and among stations shows the strong computing mode function of parallel processing of data transmission and data calculation of independent distributed computer resources at multiple levels of stations.

[0309] ④ Overall structure mode function: Strong computing mode, each level of station concurrently executes short-range data distribution, and uses the super computing power of concurrently calculating and distributing tasks and superimposing distributed computing resources to process big data.

[0310] 7.4 System modules

[0311] The system modules are the system modules of each level of site, including system preset, transmission module, and calculation module.

[0312] 7.4.1 System preset

[0313] The system preset includes:

[0314] ① Determine the program process form: It is necessary to first determine the programming route and implementation of multi-task, multi-process, and multi-thread for data processing.

[0315] ② Determine the communication transceiver mode: For data communication, it is necessary to first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods between stations, and require the matching of data sending and receiving communication modes.

[0316] ③ Determine signal transceiver matching: Before signal transceiver, it is necessary to determine the signal quality and matching of the specific communication line, and appropriately arrange the signal low-frequency amplification, intermediate-frequency amplification, and power amplification.

[0317] ④ Determine the data exchange method: Before data exchange, it is necessary to determine the data exchange method among data reading, writing, and calculation, and select a file as the storage location for the exchanged data.

[0318] ⑤ Compile the calculation task algorithm: Different calculation tasks are assigned to each level of site, and first, compile the calculation task processing algorithms for each level of site.

[0319] ⑥ Standardize the naming of file names: The data file names are involved in data calling, saving, exchanging, etc., and should be named according to rules or regulations.

[0320] 7.4.2 Transmission module

[0321] Select to use the advanced version of the transmission module. The advanced platform can build transparent access to different addresses in the network environment and directly read and write data as if it were a local machine in the advanced platform environment.

[0322] ① Data sending startup mechanism: When it is judged that there is data to be sent in the storage location for exchanged data, the site at this level immediately executes the data sending transaction of this level of site.

[0323] ② Reading of transceiver data: The transceiver software takes the smallest data processing block as the processing unit and reads the main data in the "sending" storage location where the data calculation of this level of site has been completed according to the format. If auxiliary data needs to be sent, the auxiliary data in the corresponding "sending" storage location is also read.

[0324] ③ Writing of transceiver data: Write the auxiliary data and main data into the corresponding "receiving" storage location of the lower-level site according to the format respectively.

[0325] ④ Continuously forward subsequent data: Continuously forward the subsequent blocks of auxiliary data and main data of this scene data to the lower-level site through the network with the smallest data processing block as the processing unit.

[0326] ⑤ Clean up the sent data: Timely delete the auxiliary data and main data in the storage location that has been sent, calculated, and saved to release computer resources.

[0327] 7.4.3 Calculation module

[0328] The calculation module includes:

[0329] ① Adjust auxiliary data as needed: Each level of site appropriately adjusts the required auxiliary data according to the progress of data calculation level by level.

[0330] ② Data calculation startup mechanism: When it is determined that there is data to be calculated at the data exchange storage location, the station at this level immediately executes the calculation transaction of this level station.

[0331] ③ Calculate data according to the assigned tasks: Call various required parameter data collected by this level station and the auxiliary data in the "reception" storage location according to the format, and for the corresponding main body data that has not been processed and is stored in the "reception" storage location, calculate it block by block according to the calculation task processing algorithm assigned to this level station, and save the calculated main body data in the "transmission" storage location according to the format.

[0332] ④ Continuously calculate subsequent data: Continuously calculate all subsequent blocks of data for the entire scene with the minimum data processing block as the processing unit.

[0333] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "transmission" storage location.

[0334] ⑥ Save the sent, received, and calculated data: Each level station timely saves the entire scene of main body data and auxiliary data processed by this level station in the file reception, processing, and forwarding directory in the form of a file according to the format. The file name follows the file naming rule, indicating the corresponding data level. 8 Example Five: Example 1 of Mode 3. Taking the real-time data of the CCD detector of the YL1 (Sample No. 1) satellite as an example, a multi-level data transmission and parallel processing method and device for data calculation in which the remote part between stations transmits data by CCSDS AOS&XTCE and the short-range part transmits data by DTMB.

[0335] In this embodiment, the YL1 satellite is in a sun-synchronous orbit, and the real-time data from the CCD detector of the YL1 satellite when it passes over the ground station includes auxiliary data and main body data. The auxiliary data is the telemetry data, which includes recorded data such as satellite orbit parameters, satellite positioning, and on-board time, and monitoring data for the attitude control system, communication system, power system, etc. The main body data is the remote sensing data collected by the on-board CCD detector. The satellite remote sensing data products are divided into five levels, and data correction is arranged during the transmission process, including system positioning, system projection, radiometric calibration, conversion to apparent reflectance, and geometric precision correction, and the inversion of parameter data such as temperature, humidity, wind, cloud, precipitation, sunshine, and surface background is arranged for use in the numerical prediction model system simulation. Taking the real-time data of the CCD detector of the YL1 satellite as an example, a multi-level data transmission and parallel processing method and device for data calculation in which the remote part between stations transmits data by CCSDS AOS&XTCE and the short-range part transmits data by DTMB are described, including system structure, parallel mechanism, system functions, and system modules, etc.

[0336] 8.1 System Structure

[0337] The system structure includes:

[0338] ① Composition of computer sites: The site (or node) consists of six levels of stations from the first-level station to the sixth-level station. Most sites are composed of three computers to form a network.

[0339] ② Inter-station communication technology system: The stations are connected by a remote or short-range data communication system. For the first five levels of stations, a CCSDS AOS&XTCE general data communication machine is used, and a satellite channel is leased or a wireless channel is applied for. For the last level of five stations, a DTMB general data communication machine is used, and fiber optic communication is adopted.

[0340] ③ Overall structure and function system: For the first five levels of stations and inter-station remote communication, a multi-level station remote data communication system and a supercomputer cluster with independent distributed computing resources are formed; for the last level of stations and inter-station short-range communication, a multi-site short-range data communication system and a supercomputer cluster with independent distributed computing resources are formed.

[0341] ④ Fork-shaped static structure: The first to fifth levels of stations form a multi-level one-fork tree structure, and the fifth and sixth levels of stations form a five-fork type.

[0342] ⑤ Site data processing module: The site system module is divided into modules such as data sending, receiving, and calculation.

[0343] ⑥ Parallel processing dynamic mode: Data is transmitted concurrency level by level from the first-level station to the sixth-level station, and the data processing tasks assigned to each station are calculated alternately level by level.

[0344] 8.2 Parallel mechanism

[0345] The described parallel mechanism includes:

[0346] ⑴ Minimum data processing block unit. Design the data exchange, data sending, receiving, calculation and other data processing services at each level of the station to be processed in units of the minimum data processing block according to the specified format. Use the minimum data processing block as the processing unit to divide the entire scene data into blocks, synchronize or transmit the main and auxiliary data closely, provide an opportunity for concurrent data transmission and calculation, and rely on it to achieve parallel processing of data transmission and data calculation, so as to obtain high efficiency in data transmission and data calculation.

[0347] ⑵ Setting of the minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. Determine the corresponding relationship and quantity between the telemetry data and remote sensing data collected on the satellite in this example according to the actual data collection situation. Integrate the telemetry data related to the correction of remote sensing data of the CCD detector into one frame for transmission. This frame of telemetry data corresponds to 30 frames of remote sensing data, and determine that the minimum data processing block is one frame of telemetry data and 30 frames of remote sensing data. The data volume of the minimum data processing block is relatively small, which is more conducive to reducing latency and improving the timeliness of overall data transmission.

[0348] ⑶ Data transmission and calculation interaction mechanism. Taking the smallest data processing block as a unit, the whole scene data is transmitted and calculated alternately and continuously, forming a parallel processing linkage mechanism for data transmission and data calculation at each level of the site. ① Data block transmission: Each scene of data is divided into M blocks and transmitted block by block, and the in-site transmission task is completed block by block; during the transmission at N stations, each block of data is transmitted station by station. ② Data block calculation: The calculation tasks of each scene of data are assigned to each station according to the plan, and the calculation tasks are processed block by block, and the data calculation tasks are jointly completed by each station. ③ Interleaved transmission and calculation: Each scene of data is transmitted block by block to each station, and each station calculates the data block by block, and the transmission and calculation tasks of the main data and auxiliary data are completed in the way of interleaved transmission and calculation at each station.

[0349] 8.3 System functions

[0350] The system functions include:

[0351] ① In-site service classification function: Each level of the site undertakes services such as sending, receiving, calculating, and storing main and auxiliary data.

[0352] ② Parallel processing dynamic function: Data flows from the first-level site to the sixth-level site. Each level of the site relies on a multi-level tree-like structure to complete the interleaved data transmission and data calculation at each site, then continues to complete the data transmission at multiple levels of the site, and continues to calculate the data correction tasks at levels 0-5 with the computing power.

[0353] ③ Communication mode construction function: The short-range and long-range communication modes between stations determine the application mode under the communication technology framework, which is related to the operation, function, and benefit of each part of the system; part of the communication between stations uses long-range communication, and part uses short-range communication, showing the comprehensive mode function of data transmission and data calculation with parallel processing of computer resources independently distributed at multiple levels of sites.

[0354] ④ Overall structure mode function: In the comprehensive mode, each level of the site concurrently executes part of the long-range data transmission and part of the short-range data distribution, and uses the super computing power of concurrent computing to assign tasks and superimpose distributed computing resources to process big data.

[0355] 8.4 System modules

[0356] The system modules are the system modules of each level of the site, including system preset, transmission module, and calculation module.

[0357] 8.4.1 System preset

[0358] The system preset includes:

[0359] ① Determine the program process form: It is necessary to first determine the programming route and implementation of multi-tasking, multi-process, and multi-threading for data processing.

[0360] ② Determine the communication transceiver method: For data communication, it is necessary to first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods between stations. It is required that the data sending and receiving communication methods be matched.

[0361] ③ Determine the signal transceiver matching: For signal transceiver, it is necessary to first determine the signal quality and matching of the specific communication line, and appropriately arrange signal low amplification, intermediate amplification, and power amplification.

[0362] ④ Determine the data exchange method: For data exchange, it is necessary to first determine the data exchange method between data sending, receiving, and calculation, and select an array as the storage location for the exchanged data.

[0363] ⑤ Compile the calculation task algorithm: Different calculation tasks are assigned to stations at all levels. First, compile the calculation task processing algorithms for stations at all levels.

[0364] ⑥ Standardize the naming of file names: Data file names are involved in data calling, saving, exchanging, etc., and should be named according to rules or regulations.

[0365] 8.4.2 Transmission module

[0366] Select to use the intermediate version of the transmission module. The user side uses the CCSDS AOS&XTCE and DTMB data communication systems to exchange primary and secondary data according to the specification through the hardware interface to complete data sending and receiving.

[0367] The sending sub-module includes:

[0368] Comply with the system preset conditions and send primary and secondary data in a non-network communication environment.

[0369] ① Data sending start mechanism: When it is judged that there is data to be sent in the storage location of the exchanged data, the station at this level immediately executes the data sending transaction of the station at this level.

[0370] ② Read the sending data in blocks: According to the packaging and framing plan of the primary and secondary data in the minimum data processing block, read the main body data in the "sending" storage location where the data calculation has been completed in the format. If auxiliary data needs to be sent, also read the auxiliary data in the corresponding "sending" storage location.

[0371] ③ Provide the exchanged data to the interface: The user side writes the exchanged data to the hardware interface specification, and then the general data communication machine is responsible for data packaging, framing, and transmission.

[0372] ④ Continuously send the subsequent data: Continuously read the subsequent data of the primary and secondary data, provide the interface with the exchanged data, and complete the sending of the entire scene data.

[0373] ⑤ Clean up the sent data: Immediately delete the sent, calculated, and saved auxiliary data and main body data in the storage location of the exchanged data to release computer resources.

[0374] The receiving sub-module includes:

[0375] Comply with the system preset conditions and receive primary and secondary data in a network or non-network communication environment.

[0376] ① Standardize the reading of interface data: The user end reads the unpacked data transmitted to the interface by the data deframer.

[0377] ② Save the received data in blocks: Store the primary and secondary data in the smallest data processing blocks according to the flag, and save the auxiliary data and the main body data in the corresponding data storage locations.

[0378] ③ Continuously receive subsequent data: Continuously receive and save the subsequent data of the primary and secondary data to complete the reception of the entire scene data.

[0379] 8.4.3 Calculation Module

[0380] The said calculation module includes:

[0381] ① Adjust the auxiliary data as needed: Each level of station appropriately adjusts the required auxiliary data according to the progress of data calculation level by level.

[0382] ② Data calculation start mechanism: When it is determined that there is data to be calculated in the exchange data storage location, this level of station immediately executes the calculation transaction of this level of station.

[0383] ③ Calculate the data according to the assigned tasks: Call the various required parameter data collected by this level of station and the auxiliary data in the "receiving" storage location in a format, and calculate the corresponding main body data that has not been processed and is stored in the "receiving" storage location block by block according to the calculation task processing algorithm assigned to this level of station, and save the calculated main body data in the "sending" storage location in a format.

[0384] ④ Continuously calculate the subsequent data: Continuously calculate the subsequent blocks of the entire scene data with the smallest data processing block as the processing unit.

[0385] ⑤ Adjust the auxiliary data to be sent: If the auxiliary data needs to be sent, save the original or adjusted auxiliary data in the "sending" storage location.

[0386] ⑥ Save the sent, received, and calculated data: Each level of station saves the entire scene main body data and auxiliary data received and processed by this level of station in a file in a timely manner in the file receiving, processing, and forwarding directory according to the format. The file name complies with the file naming rules and marks the corresponding data level. Example 6: Example of Mode 3. Taking the non-real-time data of the WFV detector of the YL2 (Sample No. 2) satellite as an example, a parallel processing method and device for multi-level data transmission and data calculation for data transmission between a remote and a local computer communication network.

[0387] In this embodiment, the YL2 satellite is in a sun-synchronous orbit. The non-real-time data comes from the WFV detector when the YL2 satellite passes over the ground station, including auxiliary data and main data. The auxiliary data is telemetry data, which includes recorded data such as satellite orbit parameters, satellite positioning, and on-board time, as well as monitoring data for the attitude control system, communication system, power supply system, etc. The main data is remote sensing data collected by the on-board WFV detector. The satellite remote sensing data products are divided into five levels. During the transmission process, data correction is arranged, including system positioning, system projection, radiometric calibration and correction, conversion to apparent reflectance, and geometric precision correction. Inversion of parameter data such as air temperature, humidity, wind, cloud, precipitation, sunshine, and surface background is also arranged for use in the numerical prediction model system simulation. Taking the non-real-time data of the WFV detector on the YL2 satellite as an example, a parallel processing method and device for multi-level data transmission and data calculation using remote and short-range computer networks are described, including system structure, parallel mechanism, system functions, and system modules, etc.

[0388] 9.1 System Structure

[0389] The system structure includes:

[0390] ① Composition of computer sites: The sites (nodes) are composed of a total of 6 levels of stations from the 1st level station to the 6th level station, and each level of station consists of 3 computers forming a network.

[0391] ② Inter-station communication technology system: The inter-station is connected by a remote or short-range computer communication network. The first 5 levels of stations are connected by a wide area network, which consists of two end networks and an intermediate subnet, and leased communication lines such as optical fibers or DDN dedicated lines are used; the last 1 level of 5 stations communicate via a local area network (such as various communication technologies of the IEEE 802 standard local area network), and wireless or wired (such as optical fibers, twisted pairs, coaxial cables, etc.) communication lines are used.

[0392] ③ Overall structure function system: The first 5 levels of stations and inter-station remote communication form a multi-level station remote data communication system and a supercomputer cluster with independent distributed computing resources; the last 1 level of stations and inter-station short-range communication form a multi-station short-range data communication system and a supercomputer cluster with independent distributed computing resources.

[0393] ④ Fork-shaped static structure: The 1st to 5th level stations form a multi-level one-fork tree structure, and the 5th and 6th level stations form a 5-fork type.

[0394] ⑤ Site data processing module: The site system module is divided into modules such as data reading, writing, and calculation.

[0395] ⑥ Parallel processing dynamic mode: Data is transmitted concurrency level by level from the 1st level station to the 6th level station, and the data processing tasks assigned to each station are calculated alternately level by level.

[0396] 9.2 Parallel Mechanism

[0397] The parallel mechanism includes:

[0398] ⑴ The minimum data processing block unit. Design the data processing services such as data exchange, data sending, receiving, and calculation at each level station to be processed in units of the minimum data processing block in accordance with the specified format. Use the minimum data processing block as the processing unit to divide the entire scene data into blocks, synchronize or transmit the main and auxiliary data immediately, provide an opportunity for concurrent data transmission and calculation, and rely on it to achieve parallel processing of data transmission and data calculation to obtain high efficiency in data transmission and data calculation.

[0399] ⑵ The setting of the minimum data processing block. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data, and remote sensing data is main data. Determine the corresponding relationship and quantity between the telemetry data and remote sensing data collected on the satellite in this instance according to the actual data collection situation. Integrate the telemetry data related to the correction of the remote sensing data of the WFV detector into 1 frame for transmission. This frame of telemetry data corresponds to 50 frames of remote sensing data, and determine that the minimum data processing block is 1 frame of telemetry data and 50 frames of remote sensing data. The data volume of the minimum data processing block is relatively small, which is more conducive to reducing latency and improving the timeliness of overall data transmission.

[0400] ⑶ The data transmission and calculation interaction mechanism. Use the minimum data processing block as the unit to continuously and alternately transmit and calculate the entire scene data, forming a parallel processing linkage mechanism for data transmission and data calculation at each level station. ① Data block transmission: Each scene of data is divided into M blocks and transmitted block by block, and the in-station transmission task is completed block by block; during the transmission at N stations, each block of data is transmitted station by station. ② Data block calculation: The calculation tasks of each scene of data are assigned to each station according to the plan, and the calculation tasks are allocated and processed block by block, and each station jointly completes the data calculation task. ③ Alternate transmission and calculation: Each scene of data is divided into blocks and transmitted station by station, and each station calculates the data block by block, and completes the transmission and calculation tasks of the main data and auxiliary data in the way of alternate transmission and calculation at each station.

[0401] 9.3 System functions

[0402] The system functions include:

[0403] ① In-station service classification function: Each level station undertakes services such as reading, writing, calculating, and saving main and auxiliary data.

[0404] ② Parallel processing dynamic function: The data flows from the first-level station to the sixth-level station. Each level station relies on a multi-level tree-branch structure to complete the alternate data transmission and data calculation at each station, then completes the data transmission at multiple levels of stations, and then completes the calculation tasks of data correction from level 0 to level 5 with the computing power.

[0405] ③ Communication mode structure function: The inter-station remote and short-range communication modes determine the application mode under the communication technology framework, which is related to the operation, function, and benefit of each part of the system; part of the inter-station communication uses remote communication, and part uses short-range communication, showing a comprehensive mode function of data transmission and data calculation with parallel processing of computer resources independently distributed among multiple levels of sites.

[0406] ④ Overall structure mode function: Comprehensive mode, where each level of station concurrently executes part of the remote data transmission and part of the short-range data distribution, and uses super computing power with concurrent computing to dispatch tasks and superimposed distributed computing resources to process big data.

[0407] 9.4 System modules

[0408] The system modules mentioned above are the system modules of each level of site, including system preset, transmission module, and calculation module.

[0409] 9.4.1 System preset

[0410] ① Determine the program process form: For data processing, it is necessary to first determine the programming routes and implementations of multi-tasking, multi-processing, and multi-threading.

[0411] ② Determine the communication sending and receiving methods: For data communication, it is necessary to first determine the specific communication lines, signal modulation and demodulation, and channel error correction coding and decoding methods between stations, and require the communication methods of data sending and receiving to be matched.

[0412] ③ Determine the signal sending and receiving matching: For signal sending and receiving, it is necessary to first determine the signal quality and matching of the specific communication line, and appropriately arrange signal low-frequency amplification, intermediate-frequency amplification, and power amplification.

[0413] ④ Determine the data exchange method: For data exchange, it is necessary to first determine the data exchange method between data reading, writing, and calculation, and select a file as the storage location for the exchanged data.

[0414] ⑤ Compile the calculation task algorithm: Different calculation tasks are assigned to each level of site, and the calculation task processing algorithms of each level of site need to be compiled first.

[0415] ⑥ Standardize the naming of file names: The names of data files are related to data calling, saving, exchanging, etc., and should be named according to rules or regulations.

[0416] 9.4.2 Transmission module

[0417] Select to use the advanced version of the transmission module. The advanced platform can build transparent access to different addresses in the network environment and directly read and write data as if it were on the local machine in the advanced platform environment.

[0418] ① Data sending startup mechanism: When it is judged that there is data to be sent in the storage location of the exchanged data, the site at this level immediately executes the data sending transaction of this level of site.

[0419] ② Reading of transmitted and received data: The transmission and reception software takes the minimum data processing block as the processing unit and reads the main data in the "transmission" storage location where the data calculation at this level station has been completed according to the format. If auxiliary data needs to be transmitted, the auxiliary data in the corresponding "transmission" storage location is also read.

[0420] ③ Writing of transmitted and received data: Write the auxiliary data and main data to the corresponding "reception" storage location of the lower-level station respectively according to the format.

[0421] ④ Continuously forwarding subsequent data: Continuously forward the subsequent blocks of auxiliary data and main data of this scene data to the lower-level station through the network with the minimum data processing block as the processing unit.

[0422] ⑤ Cleaning the transmitted data: Timely delete the auxiliary data and main data in the storage locations that have been transmitted, calculated, and saved to release computer resources.

[0423] 9.4.3 Calculation module

[0424] The calculation module includes:

[0425] ① Adjusting auxiliary data as needed: Each level station appropriately adjusts the required auxiliary data according to the progress of data calculation level by level.

[0426] ② Data calculation startup mechanism: When it is judged that there is data that needs to be calculated in the exchange data storage location, this level station immediately executes the calculation transaction of this level station.

[0427] ③ Calculating data according to the assigned tasks: Call the various required parameter data collected by this level station and the auxiliary data in the "reception" storage location according to the format, and process the corresponding main data that has not been processed and is stored in the "reception" storage location block by block according to the calculation task processing algorithm assigned to this level station, and save the calculated main data to the "transmission" storage location according to the format.

[0428] ④ Continuously calculating subsequent data: Continuously calculate the subsequent blocks of the entire scene with the minimum data processing block as the processing unit.

[0429] ⑤ Adjusting the transmitted auxiliary data: If the auxiliary data needs to be transmitted, save the original or adjusted auxiliary data in the "transmission" storage location.

[0430] ⑥ Saving the transmitted, received, and calculated data: Each level station timely saves the entire scene of main data and auxiliary data written and processed by this level station to the file reception, processing, and forwarding directory in the form of files. The file name complies with the file naming rules and marks the corresponding data level.

Claims

1. An interactive concurrency method for multi-level data transmission and data calculation, characterized in that, It includes: a minimum data processing block unit, a minimum data processing block setting, and a data transmission calculation interaction mechanism, which are three parts; The minimum data processing block unit means that for the data exchange, data sending, receiving, and calculation data processing services at all levels of stations, they are all processed in units of the minimum data processing block in accordance with the specified format; Using the minimum data processing block as the processing unit, the entire scene / frame / track data is divided into blocks, and the main and auxiliary data are synchronized or transmitted immediately following each other, providing an opportunity for concurrent data transmission calculation. Relying on this, the parallel processing of data transmission and data calculation is realized, and high efficiency of data transmission and data calculation is obtained; The minimum data processing block setting means that the minimum data processing block includes auxiliary data and main data, or only includes main data; ① If there is no corresponding relationship between the auxiliary data and the main data, when determining the minimum data processing block, only consider the influencing factors of its own data calculation efficiency, and the minimum data processing block is just a set of main data; ② If there is a corresponding relationship between the auxiliary data and the main data and the auxiliary data does not need to be transmitted, the minimum data processing block is a set of main data corresponding to a set of auxiliary data; ③ If the auxiliary data needs to be transmitted, the minimum data processing block is a set of auxiliary data and a set of main data corresponding to it; The amount of data in the minimum data processing block is relatively small, which is more conducive to reducing latency and improving the timeliness of overall data transmission; The data transmission calculation interaction mechanism means that with the minimum data processing block as the unit, the entire scene / frame / track data is continuously transmitted and calculated alternately, forming a parallel processing linkage mechanism for data transmission and data calculation at all levels of stations; ① Data block transmission: Each scene / frame / track data is divided into M blocks and transmitted block by block, and the in-station transmission task is completed block by block; During the transmission at N stations, each block of data is transmitted station by station; ② Data block calculation: The calculation tasks of each scene / frame / track data are assigned to each station according to the plan, and the assigned calculation tasks are processed block by block, and the data calculation tasks are jointly completed by each station; ③ Alternate transmission calculation: Each scene / frame / track data is divided into blocks and transmitted station by station, and each station calculates the data block by block, and the transmission and calculation tasks of the main data and the auxiliary data or only the main data are completed in the way of alternate transmission calculation at each station.

2. The interactive concurrency method according to claim 1, wherein It also includes: A parallel transmission calculation mathematical model, that is: the data is divided into M blocks, and N stations perform parallel transmission calculation. According to the spatio-temporal relationship of transmission and calculation between stations and between blocks, the time taken for the whole-process data parallel transmission calculation is obtained Where: N and M are integers greater than or equal to 2, 1 ≤ i ≤ N, 1 ≤ j ≤ M; T Ci is the channel time consumption between block data stations; T Di,j is the time for block data transceiver, T Di,j = T DSi,j + T DRi,j , T DSi,j is the time for block data transmission, T DRi,j is the time for block data reception; T Oi,j is the timeout for block data calculation, T Oi,j = T SCi,j – T Di,j where T SCi,j = T C1i,j + T C2i,j and T SCi,j is the cache time for block data calculation, T C1i,j is the calculation time for block data, T C2i,j is the cache time for block data, and it is stipulated that when T SCi,j - T Di,j ≤ 0, T Oi,j = 0; the right item 1 of formula (1) is the total time consumed by the channels between stations at all levels, the right item 2 is the total time for sending and receiving the same block data at stations at all levels, the right item 3 is the total timeout for calculating the same block data at stations at all levels, and the right item 4 is the total delay of the sub-block data except the first block of station 1 averaged by each station; combining like terms, we get The amount of data in each block is the same. If the same-level stations use the same computer to process each block of data, then there is In a simplified case, T NM can be summarized as the sum of the following four items: the total time consumed by the channels between stations at all levels for one block of data, the total time for sending and receiving at stations at all levels for one block of data, the total timeout for calculations at stations at all levels for one block of data, and the total delay for sending and receiving all blocks of data except the first block at one station.

3. A parallel processing method for multi-level data transmission and data calculation, characterized in that It includes: Four parts: system structure, parallel mechanism, system function, and system module; Building the system structure includes: consisting of multiple levels of computer stations; the communication between stations uses a remote and short-range computer communication network, or a remote and short-range non-computer communication network; the stations and the remote or short-range communication between stations form a multi-level station remote or short-range data communication system and an independently distributed supercomputer cluster; the multi-level stations and their data flow directions are a multi-level tree-branch type static structure; the data processing at the stations is divided into data transmission and data calculation modules; a dynamic mode of parallel processing for data to be transmitted and calculated alternately level by level; The adoption of a parallel mechanism includes: data chunking provides an opportunity for concurrent data transmission and computing of big data; the minimum data processing chunks are divided into three forms: ① a set of main data, ② the main data corresponding to a set of auxiliary data, and ③ a set of auxiliary data and its corresponding set of main data; the data chunks are transmitted and calculated station by station, forming an interleaved transmission and computing mechanism to complete the whole-process data transmission and computing tasks; The realization of system functions includes: each level of station undertakes data transmission and data computing services; the multi-level station data transmission and total amount task computing are completed in the form of data interleaved transmission and computing level by level; the inter-station communication mode determines the operation, function and benefit of the system, showing different structural mode functions; the remote or short-range communication between stations and among stations determines the combination of multi-level station remote data communication, short-range data communication and independently distributed supercomputing; The establishment of system modules, namely system modules for each level of station, includes system preset, transmission module and computing module; The said system preset includes: pre-determining the program process form, communication sending and receiving mode, signal sending and receiving matching and data exchange mode, pre-compiling the computing task algorithm, and implementing the standardized naming of file names; The said transmission module is divided into a low-level version, a middle-level version and a high-level version, and the version is selected as needed; the low-level version transmission module includes a sending sub-module and a receiving sub-module; the sending sub-module includes: formulating a data sending start mechanism, reading and sending data according to the minimum data processing chunk, packing and framing the data into frames or multiple frames according to the minimum data processing chunk for transmission, continuously sending the subsequent data of the data sending task, and clearing the already sent main and auxiliary data or only the main data; the receiving sub-module includes: completely receiving each frame of data, unpacking the frame according to the flags of the frame header and packet header, saving the received main and auxiliary data or only the main data in chunks according to the flags, and continuously receiving the subsequent data of the data receiving task; the middle-level version transmission module includes a sending sub-module and a receiving sub-module; the sending sub-module includes: formulating a data sending start mechanism, reading and sending data according to the minimum data processing chunk, providing exchange data to the interface according to the frame plan, continuously sending the subsequent data of the data sending task, and clearing the already sent main and auxiliary data or only the main data; the receiving sub-module includes: reading data according to the interface specification, saving the received main and auxiliary data or only the main data in chunks according to the flags, and continuously receiving the subsequent data of the data receiving task; the high-level version transmission module includes: formulating a data sending start mechanism, reading and sending data from the local station in chunks, writing it into the "receiving" exchange data storage place of the lower station, continuously forwarding the subsequent data of the data forwarding task, and clearing the already sent main and auxiliary data or only the main data; The said computing module includes: adjusting the auxiliary data to be used, formulating a data computing start mechanism, calling the collected parameter data and the "received" auxiliary data, calculating the unprocessed main data in chunks according to the computing task processing algorithm, continuously calculating the subsequent data of the data computing task, adjusting the sent auxiliary data, and saving the already sent, received and calculated main and auxiliary data or only the main data.

4. The parallel processing method according to claim 3, wherein It also includes: Remote communication is adopted between stations, and together with its respective stations, it constitutes a multi-level station remote data communication system and a supercomputer cluster with independently distributed computing resources.

5. The parallel processing method according to claim 3, wherein It also includes: Short-range communication is adopted between stations, and together with its respective stations, it forms a multi-level station short-range data communication system and a supercomputer cluster with independently distributed computing resources.

6. The parallel processing method according to claim 3, wherein It also includes: Shared remote and short-range communication between stations, which forms a multi-level station local remote and local short-range data communication system and a supercomputer cluster with independently distributed computing resources with its respective stations.

7. A parallel processing device for multi-level data transmission and data calculation, characterized in that, It includes: Four parts: system structure, parallel mechanism, system function, and system module; The device system structure includes: consisting of multi-level computer stations; remote and short-range computer communication networks or remote and short-range non-computer communication networks are used for communication between stations; remote or short-range communication between stations and among stations constitutes a multi-level station remote or short-range data communication system and an independently distributed supercomputer cluster; the multi-level stations and their data flow directions are a multi-level tree-branch type static structure; site data processing is divided into data transmission and data calculation modules; a parallel processing dynamic mode of data interleaved transmission and calculation. The device parallel mechanism includes: data chunking provides an opportunity for large data concurrent transmission and calculation; the smallest data processing chunks are divided into three forms: ① a group of main data, ② the main data corresponding to a group of auxiliary data, ③ a group of auxiliary data and the corresponding group of main data; data chunking is transmitted station by station and calculated station by station, forming an interleaved transmission and calculation mechanism to complete the whole-process data transmission and calculation task. The device system function includes: each level of station undertakes data transmission and data calculation services; the multi-level station data transmission and total task calculation are completed in the way of data interleaved transmission and calculation step by step; the communication method between stations determines the operation, function, and benefit of the system, showing different structural mode functions; remote or short-range communication between stations and among stations determines the combination of multi-level station remote data communication, short-range data communication, and independently distributed supercomputing. The device system module, that is, the system module of each level of station, includes system preset, transmission module, and calculation module; The said system preset includes: pre-determining the program process form, communication sending and receiving method, signal sending and receiving matching, and data exchange method, pre-compiling the calculation task algorithm, and implementing the standardized naming of file names; The transmission module is divided into a low-level version, a medium-level version, and a high-level version, and the version is selected according to requirements; the low-level transmission module includes a sending sub-module and a receiving sub-module; the sending sub-module includes: formulating a data sending start mechanism, reading the sending data in the smallest data processing blocks, packing and framing the data in frames or multiple frames according to the smallest data processing blocks for transmission, continuously sending the subsequent data of the data task, and clearing the sent primary and secondary data or only the primary data; the receiving sub-module includes: completely receiving each frame of data, unpacking the frame according to the flags of the frame header and packet header, saving the received primary and secondary data or only the primary data in blocks according to the flags, and continuously receiving the subsequent data of the data task; the medium-level transmission module includes a sending sub-module and a receiving sub-module; the sending sub-module includes: formulating a data sending start mechanism, reading the sending data in the smallest data processing blocks, providing the exchanged data to the interface according to the frame plan, continuously sending the subsequent data of the data task, and clearing the sent primary and secondary data or only the primary data; the receiving sub-module includes: reading the data according to the interface specification, saving the received primary and secondary data or only the primary data in blocks according to the flags, and continuously receiving the subsequent data of the data task; the high-level transmission module includes: formulating a data sending start mechanism, reading the sending data in blocks from the local station, writing it into the "receiving" exchanged data storage location of the lower-level station, continuously forwarding the subsequent data of the data forwarding task, and clearing the sent primary and secondary data or only the primary data; The calculation module includes: adjusting the auxiliary data to be used, formulating a data calculation start mechanism, calling the collected parameter data and the "received" auxiliary data, calculating the unprocessed primary data in blocks according to the calculation task processing algorithm, continuously calculating the subsequent data of the data task, adjusting the sent auxiliary data, and saving the sent, received, and calculated primary and secondary data or only the primary data.

8. The parallel processing device according to claim 7, wherein It also includes: Remote communication is adopted between stations, and together with its respective stations, it forms a multi-level station remote data communication system and a supercomputer cluster with independent distributed computing resources.

9. The parallel processing device according to claim 7, wherein It also includes: Short-range communication is adopted between stations, and together with its respective stations, it forms a multi-level station short-range data communication system and a supercomputer cluster with independent distributed computing resources.

10. The parallel processing device according to claim 7, wherein It also includes: Both remote and short-range communication are used between stations, and together with its respective stations, it forms a multi-level station local remote and local short-range data communication system and a supercomputer cluster with independent distributed computing resources.

Images