Parallel processing method and apparatus for multi-level data transmission and data computation

Abstract

The present application relates to an interactive concurrency method, and parallel processing method and apparatus for multi-level data transmission and data computation of big data or large models. In order to achieve the timeliness of big data or large model multi-access data transmission and multi-task computing, three application modes of a multi-level site / node remote, short-range and local remote / local short-range data communication system and a supercomputer / processor / computing core cluster having distributed computing resources are established. The interactive concurrency method establishes a data block forwarding and computing mechanism, uses relatively short time to complete big data or large model multi-level site / node concurrent transmission, and aggregates computing power of sites / nodes to complete big data or large model computing tasks, and thus has extremely high data transmission and computing benefits. The present application efficiently satisfies the demands on different application types of super-computing power for big data or large model multi-access data communication and multi-task computing.

Description

Parallel processing method and device for multi-level data transmission and data calculation TECHNICAL FIELD

[0001] The present application relates to a parallel processing method and device for multi-level data transmission and data calculation, and belongs to the technical field of big data, computer network, data communication and data calculation. TECHNICAL BACKGROUND

[0002] Current communication technology and computing technology are striving to cater to and promote the human need for rapid information processing. Artificially collected statistical information data, business data generated by technical platforms, and information data acquired by collectors all grow explosively. For example, mobile phones collect voice, cameras collect images, video recorders and cameras collect audio and video, radars detect targets, satellites detect the earth, spacecraft detect the universe, and various collectors continuously collect information. Information data distribution and sharing require transmission and calculation processing. According to the Shannon principle, information transmission speed is related to channel frequency bandwidth. In order to improve the timeliness of data transmission, wired or wireless channels must rely on abundant frequency spectrum resources. Current data transmission mostly uses optical fibers if it goes through the ground, and mostly rents microwave channels if it uses satellite communication. In order to improve the speed of data processing, a computer with strong data processing capability is needed, and in order to improve the speed of data calculation, a faster CPU (Central Processing Unit) or GPU (Graphics Processing Unit) and memory components are needed. Increasing memory capacity is beneficial to big data processing, and increasing the speed of peripheral data reading and saving requires using external storage with faster read-write access and its interface circuit, such as solid state disks and their supporting interfaces. In order to improve the time efficiency of data transmission and calculation, more expensive parallel transmission and parallel calculation systems can be used.

[0003] There are multi-level data transmission and data calculation requirements for business organization hierarchical setting and business data multitasking processing. Whether data needs multi-level transmission depends on many factors, such as front-end data processing and retransmission, ground special line cost related to distance, or setting up sub-center for retransmission, high cost of satellite receiving equipment, and subsequent transmission after receiving satellite data and transferring to ground special line. For example, multi-level transmission of provincial(departmental) satellite remote sensing system, satellite data is received by ground station, processed by data center, and transferred to its subordinate department level, prefecture level, county level, etc. through ground special line. Before data application, data needs not only multi-level distribution, but also multi-task data calculation. For example, satellite remote sensing data often needs a series of data preprocessing, including time domain, space domain, frequency domain and various data correction, as well as various ground object inversion, to provide various parameter data for earth simulation system. Early, multi-level data transmission is transmitted by independent station, after receiving the file of previous stage station, the data is transmitted to the next stage station. In order to improve the timeliness of data transmission, multi-level data transmission is often transmitted by synchronous transfer between stages, and data transmission is completed in real time. If the previous stage station has data calculation task, it can save the received data file by using large capacity memory, avoid wasting time, and use large power CPU or GPU processor to complete the corresponding data calculation, and then transfer to the next stage station. In the current multi-level data transmission, if there is data calculation task, it needs to wait for the completion of data file receiving, and the waiting time of transfer process and receiving process is relatively long, the data transmission time is not used for data calculation, and the timeliness and comprehensive benefit of data transmission and data calculation link of multi-level station are not considered.

[0004] Although data acquisition and processing are getting faster and faster, the timeliness of current multi-level data transmission and data calculation is still far from people's demand for industry application. Big data business systems need to deal with the resource pressure of data transmission and data calculation caused by centralized processing of business data in various places and congestion during peak periods. For big data in the fields of military, meteorology, territory, ocean, and environment, there is a poor timeliness in local and global simulation systems. In terms of technology, there is an urgent need for innovation and upgrading in data transmission, data time domain, space domain, and frequency domain correction, various parameter data inversion, and system simulation operation. For example, military target monitoring, earth resource monitoring, and disaster monitoring and prediction require high-speed data acquisition and processing, even at the cost of improving processing capacity. Especially, there is a big gap in the computing power of artificial intelligence big models, and there is a big gap in the decomposition of total target tasks and data transmission and calculation. Based on previous related research, it is more urgent to build an efficient multi-level data transmission and data calculation parallel processing application mode. Unlike big data parallel transmission mode and big data parallel calculation mode, it improves the transmission and calculation efficiency of big data together, which is of great significance to the promotion of national economy and social benefits. SUMMARY

[0005] The present application aims at data sources that require multi-address data transmission and multi-task data calculation. In order to overcome the closed nature of the data transmission process, an interactive fusion mechanism of data transmission and data calculation is established, a super application mode of data transmission and data calculation is constructed, the transaction efficiency of data transmission and data calculation is improved, and an efficient multi-level data transmission and data calculation parallel processing method and device are provided.

[0006] The present application includes an interactive concurrent method, a parallel processing method and a device for multi-level data transmission and data calculation. The interactive concurrent method is a parallel mechanism of the system, the parallel processing method includes four aspects of constructing system structure, adopting parallel mechanism, realizing system function, and establishing system module, and the parallel processing device includes four aspects of system structure, parallel mechanism, system function, and system module of the device.

[0007] The system structure includes: composed of multi-level computer (processor / computing core) sites (nodes); inter-site (inter-node) communication adopts long-range and short-range computer communication network, or long-range and short-range non-computer communication network; the site (node) and the long-range or short-range communication between them constitute a multi-level site (node) long-range or short-range data communication system and a distributed computer (processor / computing core) cluster; the multi-level site (node) and its data flow direction are a multi-level tree right type static structure; site (node) data processing is divided into data transmission and data calculation modules; the parallel processing dynamic mode of data staggered transmission and calculation at each level.

[0008] The parallel mechanism includes: data block provides opportunities for large data concurrent transmission calculation; from the auxiliary data and the corresponding relationship between the main data block, the minimum data processing block is divided into two basic forms: a group of main data, no or no need to transfer auxiliary data, a group of auxiliary data and its corresponding group of main data; data block transmission per site ( / node), per site ( / node) calculation, forming an interleaved transmission computing mechanism, to complete the whole data transmission calculation task. Also stated the mathematical model of parallel transmission calculation.

[0009] The parallel mechanism of large model includes: data block provides opportunities for large data concurrent transmission calculation. The minimum data processing block of large model is the smallest semantic unit (Token), which can be the smallest semantic unit of text, image, audio and video data. Block the whole data with the smallest semantic unit, determine each smallest semantic unit and its quantity. Data block transmission per site ( / node), per site ( / node) calculation, forming an interleaved transmission computing mechanism, to complete the whole data transmission calculation task. Also stated the mathematical model of parallel transmission calculation.

[0010] The system function includes: each level of site ( / node) undertakes data transmission and data calculation business; the way of data staggered transmission and calculation at each level completes the multi-level site ( / node) data transmission and total amount task calculation; the communication mode between sites ( / nodes) determines the operation, effect and benefit of the system, and shows different structure mode functions; the site ( / node) and its remote or near range communication determine the combination of multi-level site ( / node) remote data communication, near range data communication and distributed supercomputing.

[0011] The system module, namely the system module of each level of site ( / node), includes system preset, transmission module and calculation module.

[0012] The system preset includes: the program process form, communication transceiver mode, signal transceiver matching and data exchange mode are determined in advance, the calculation task algorithm is prepared in advance, and the file name specification naming is implemented.

[0013] The transmission module is divided into low-level version, intermediate version and high-level version, and the version is selected according to the requirement. The low-level version transmission module includes a sending submodule and a receiving submodule. The sending submodule includes: formulating a data sending starting mechanism, reading sending data according to the minimum data processing block, packing and framing the data for transmission according to the minimum data processing block, continuously sending subsequent data of the data task, cleaning the sent main and auxiliary data (or only the main data), etc. The receiving submodule includes: completely receiving each frame of data, unframing and unpacking according to the frame header and the package header, saving the received main and auxiliary data (or only the main data) according to the block, and continuously receiving subsequent data of the data task. The intermediate version transmission module includes a sending submodule and a receiving submodule. The sending submodule includes: formulating a data sending starting mechanism, reading sending data according to the minimum data processing block, providing exchange data to the interface according to the frame planning, continuously sending subsequent data of the data task, cleaning the sent main and auxiliary data (or only the main data), etc. The receiving submodule includes: reading data according to the interface specification, saving the received main and auxiliary data (or only the main data) according to the block, and continuously receiving subsequent data of the data task. The high-level version transmission module includes: formulating a data sending starting mechanism, reading sending data from the current site (node) according to the block, writing the exchange data into the "receiving" storage place of the lower site (node), continuously forwarding subsequent data of the data task, and cleaning the sent main and auxiliary data (or only the main data).

[0014] The calculation module includes: adjusting the auxiliary data needed, formulating a data calculation starting mechanism, calling the collected parameter data and the "received" auxiliary data, calculating the main data that has not been processed according to the calculation task processing algorithm, continuously calculating subsequent data of the data task, adjusting the sent auxiliary data, and saving the sent (or read), received (or written), calculated main and auxiliary data (or only the main data).

[0015] Compared with the prior art, the present application has the following innovations and benefits: (1) a parallel mechanism of data transmission and data calculation is established, the closed nature of the data transmission process is broken, a block forwarding mechanism of data transmission is established, a convenient door is provided for parallel processing of data transmission and data calculation, intensive forwarding opportunities are provided for multi-level site ( / node) data concurrent transmission, and time slots required for data processing are provided for data calculation; (2) using the parallel mechanism of data transmission and data calculation, large data multi-level site ( / node) concurrent transmission can be completed in relatively little time, and large data calculation tasks can be completed by gathering site ( / node) computing power, which has very high data transmission and data calculation efficiency; (3) using transmission time and computing resources, three implementation application modes are constructed by using different communication methods: ① a combination mode, a multi-level site ( / node) remote data communication system and a computer ( / processor / computing core) cluster of distributed computing resources, which simultaneously provide efficient data communication and super-power data calculation; ② a strong calculation mode, a near-range high-efficiency data communication system, a computer ( / processor / computing core) cluster of distributed computing resources, which plays a special super-power data calculation; and ③ a comprehensive mode, a multi-level site ( / node) local remote and local near-range data communication system, a computer ( / processor / computing core) cluster of distributed computing resources, which simultaneously provides efficient data communication and high-quality super-power data calculation; (4) the present application provides a parallel mechanism and its application mode for large data multi-address transmission and multi-task calculation, and efficiently solves the different types of needs of super-power for large data multi-address data communication and large data multi-task calculation.

[0016] Other features and advantages of the present application will become more apparent after a detailed description of the embodiments of the present application is given in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0017] Fig. 1 is a general principle block diagram;

[0018] Fig. 2 is a system structure block diagram;

[0019] Fig. 3 is a parallel mechanism block diagram;

[0020] Fig. 4 is a system function block diagram;

[0021] Fig. 5 is a system module block diagram;

[0022] Fig. 6 is a flowchart of a low-level version transmission module;

[0023] Fig. 7 is a flowchart of a middle-level version transmission module;

[0024] Fig. 8 is a flowchart of a high-level version transmission module;

[0025] Fig. 9 is a flowchart of a calculation module. DETAILED DESCRIPTION

[0026] The following further describes the embodiments from general to specific, first describes the data, and then describes the 9 related statements based on the general overall implementation scheme, and then describes the detailed scheme of the application in 3 application modes determined by inter-site ( / node) communication. It should be noted that the description of these embodiments is helpful for understanding the application, but some specific descriptions do not constitute a limitation on the application.

[0027] 1. General method and device

[0028] The data includes auxiliary data and main data, or only main data, depending on the specific data source and transmission, and the data processing task of the calculation. Auxiliary data is parameter data used when calculating main data. On the one hand, big data requires multi-address data transmission and multi-task data calculation processing. Data is divided into real-time data and non-real-time data (or historical data) from the time effectiveness. It is mainly various collected data on the ground, air and space, business data generated by technical platforms, and information data collected by artificial statistics, such as satellite and radar detection data, or production and operation industry data. The main data of satellite detection is remote sensing data, and the auxiliary data is telemetry data; there are many types of satellite detectors, such as CCD (Charge Coupled Device), WFV (Wide Field of View), MERSI (Medium Resolution Spectral Imager) remote sensing data. Satellite and radar remote sensing data require various data calculations such as correction, inversion and simulation. On the other hand, artificial intelligence big model data processing. The collected data is the main data, the big model decomposes the total target task, including 5 intermediate core process data processing of text preprocessing, Tokenization, vectorization, neural network inference and output generation, and needs to complete the data delivery to the computing device and dispatch the execution of the calculation task.

[0029] The general method and device of the application include system structure, parallel mechanism, system function, system module, etc. 4 aspects, see Figure 1.

[0030] 1.1. System structure

[0031] Referring to Figure 2, the system structure includes:

[0032] ① Computer ( / processor / computing core) site ( / node) composition: multi-level sites ( / nodes) are composed of 1st level sites ( / nodes), 2nd level sites ( / nodes), …, Nth level sites ( / nodes) (N is a positive integer, and N≥2), each site ( / node) is composed of one or more computers ( / processors / computing cores) and the like.

[0033] ②Inter-site communication system: Inter-site communication uses remote computer communication network (wide area network), or remote non-computer communication network (non-network version remote data communication system), short-range computer communication network (local area network or metropolitan area network), short-range non-computer communication network (non-network version short-range data communication system). The communication line has wireless and wired, various specifications line according to the needs of selection and use.

[0034] ③Overall structural function system: The site ( / node) and its remote communication between sites ( / nodes) constitute a multi-level site ( / node) remote data communication system and a distributed computing resource computer ( / processor / computing core) cluster, even a supercomputer ( / processor / computing core) cluster; or the site ( / node) and its short-range communication between sites ( / nodes) constitute a multi-level site ( / node) short-range data communication system and a distributed computing resource computer ( / processor / computing core) cluster, even a supercomputer ( / processor / computing core) cluster; or the site ( / node) and its remote and short-range communication between sites ( / nodes) constitute a multi-level site ( / node) local remote, local short-range data communication system and a distributed computing resource computer ( / processor / computing core) cluster, even a supercomputer ( / processor / computing core) cluster.

[0035] ④Tree branch type static structure: The multi-level site ( / node) and its data flow show a multi-level tree branch type structure, and the multi-level one-branch tree structure is a special form.

[0036] ⑤Site ( / node) data processing module: The site ( / node) system module is divided into data sending (or reading), receiving (or writing), computing and other modules.

[0037] ⑥Parallel processing dynamic mode: Data is transmitted from the first level site ( / node) to the Nth level site ( / node) in parallel, and the data processing tasks of the site ( / node) are dispatched in staggered computing.

[0038] 1.2. Parallel mechanism

[0039] Referring to FIG. 3, the parallel mechanism includes:

[0040] (1) Minimum data processing block unit. The data exchange and data sending (or reading), receiving (or writing), computing and other data processing services of each level site ( / node) are processed in the form of minimum data processing block as a unit. The whole scene [ / frame / track] data is divided into blocks by taking the minimum data processing block as a processing unit, the main and auxiliary data are transmitted synchronously or immediately, the data concurrent transmission computing opportunity is provided, and the parallel processing of data transmission and data computing is realized, thereby obtaining high efficiency of data transmission and data computing.

[0041] (2) Minimum data processing block setting. The minimum data processing block includes auxiliary data and main data, or only includes main data. The data is blocked according to the characteristics of the data structure, such as the correspondence between auxiliary data and main data. ① If there is no correspondence between auxiliary data and main data, the determination of the minimum data processing block only considers the influencing factors of the data calculation efficiency, and the minimum data processing block is only a group of main data; ② If there is a correspondence between auxiliary data and main data, auxiliary data does not need to be transmitted, and the minimum data processing block is a group of main data corresponding to auxiliary data; ③ If auxiliary data needs to be transmitted, the minimum data processing block is a group of auxiliary data and a group of main data corresponding thereto. The data amount of the minimum data processing block is relatively small, which is more conducive to reducing the time delay and improving the timeliness of the overall data transmission and calculation.

[0042] (3) Data transmission and calculation interaction mechanism. Taking the minimum data processing block as a unit, the whole scene / sheet / track data is transmitted and calculated in an interleaved manner to form a parallel processing linkage mechanism of data transmission and data calculation of each level of station / node. ① Data block transmission: each scene / sheet / track data is transmitted block by block, and the transmission task in the station / node is completed block by block; in the transmission of N stations / nodes, each block of data is transmitted from station to station / node. ② Data block calculation: each scene / sheet / track data calculation task is assigned to each station / node according to the planning, and the assigned calculation task is processed block by block, and the data calculation task is completed together by each station / node. ③ Interleaved transmission and calculation: each scene / sheet / track data is transmitted block by block, each station / node calculates the data block by block, and the transmission and calculation task of the main data and auxiliary data (or only the main data) is completed in an interleaved transmission and calculation manner of each station / node.

[0043] (4) Parallel transmission and calculation mathematical mode. ① Mathematical mode is established. Data is divided into M blocks, and N stations / nodes are used for parallel transmission and calculation. According to the space-time relationship of transmission and calculation between stations / nodes and blocks, the time of whole data parallel transmission and calculation is obtained

[0044] wherein: N and M are integers greater than or equal to 2, 1≤i≤N, 1≤j≤M; T Ci is the channel time consumption between block data stations / nodes; T Di，j is the block data transmission and reception time; T Oi，j is the block data calculation timeout, T Oi，j =T SCi，j -T Di，j , T SCi，j =T C1i，j +T C2i，j , T SCi，j is the block data calculation cache time, T C1 i，j is the block data calculation time, and T C2i，jFor block data cache, and T SCi，j -T Di，j ≤0, T Oi，j =0; the right side of formula (1) 1 is the total time of the channel between each level of site (node), the right side 2 is the total time of each level of site (node) receiving and transmitting the block data, the right side 3 is the total timeout of each level of site (node) calculating the block data, and the right side 4 is the total delay of 1 site (node) except the first block data; combining the same items, we get

[0045] Each block data has the same size, if the same level of site (node) uses the same computer (processor / computing core) to process each block data, then The transmission speed of each level of site (node) is related to the size of block data. It shows that reducing the size of block data will directly reduce the time of data receiving and transmitting of each level of site (node). Although each block data has the same size, the equipment used by each level of site (node) is different, and the data transmission speed may be different. T Oi，1 is the total timeout of each level of site (node) calculating the block data, which is related to the data task calculation cache speed and data transmission speed of each level of site (node). When T Oi，1 ≤0, it means that the data task calculation cache does not delay the data transmission, and when T Oi，1 >0, it means that the data task calculation cache delays the data transmission link. It shows that improving the computing resources will directly reduce the data calculation timeout. T NM is the total delay of 1 site (node) except the first block data, which is related to the average data transmission speed of each level of site (node) and the size of block data. Each block data has the same size, and the data transmission speed of the same site is the same. In the simplified case, T NM can be summarized as the sum of the total time of the channel between each level of site (node) for 1 block data, the total time of each level of site (node) receiving and transmitting 1 block data, the total timeout of each level of site (node) calculating 1 block data, and the total delay of 1 site (node) except the first block data.

[0046] ③Data transmission and calculation have high efficiency. The whole process is a parallel state of data transmission and calculation cross concurrent processing, and the efficiency of data transmission and calculation is evaluated according to the whole process. This mathematical model reflects the time efficiency of data transmission and data calculation in the whole process. From the mathematical model, the overall data transmission and calculation efficiency of the site (node) channel delay is the same, and there is no reduction, but only (N+M-1) block data is used for receiving and transmitting, and only N block data is used for calculating the timeout. The results show that using data transmission time and data calculation at the same time not only increases the data calculation efficiency, but also increases the data transmission efficiency of multi-level sites (nodes); reducing the size of block data will correspondingly improve the data transmission and calculation efficiency of multi-level sites (nodes).

[0047] 1.2A. Parallel mechanism [for large model]

[0048] Referring to FIG. 3, the parallel mechanism includes:

[0049] (1) Minimum data processing block unit. Designing data exchange and data sending (or reading), receiving (or writing), computing, and other data processing services of each level of site ( / node) are processed in a specified format with the minimum data processing block as a unit. The overall data is divided into blocks with the minimum data processing block as a processing unit. Initially, there is no auxiliary data synchronized or immediately transmitted with the main data, providing an opportunity for concurrent data transmission and computation, relying on parallel processing of data transmission and data computation, and achieving high efficiency of data transmission and data computation.

[0050] (2) Minimum data processing block setting. The minimum data processing block has only main data. The large model data structure features semantic units. ① The minimum semantic unit is determined as the data processing unit, which is the minimum data processing block; ② The minimum data processing block can be the smallest semantic unit of text, image, audio, and video when the overall data is divided into blocks; ③ The actual collected data determines the various minimum semantic units and their quantities in this example. The minimum data processing block has a relatively small data volume, which is more conducive to reducing latency and improving the timeliness of overall data transmission and computation;

[0051] (3) Data transmission and computation interaction mechanism. The overall data is transmitted and computed in an interleaved manner with the minimum data processing block as a unit, forming a parallel processing linkage mechanism for data transmission and computation of each level of site ( / node). ① Data block transmission: the overall data is transmitted block by block, and the transmission task within the site ( / node) is completed block by block; in N-site ( / node) transmission, each block of data is transmitted from station to station ( / node). ② Data block computation: the overall data computation task is assigned to each site ( / node) according to the plan, and the assigned computation task is processed by block, and the data computation task is completed by each site ( / node) together. ③ Interleaved transmission and computation: the overall data is transmitted block by block by each site ( / node), and each site ( / node) computes data by block to complete the transmission and computation task of the main data and auxiliary data (or only the main data) in an interleaved transmission and computation manner.

[0052] (4) Parallel transmission and computation mathematical mode.

[0053] The contents of ①②③ here are the same as the contents of ①②③ in "(4) Parallel transmission and computation mathematical mode" of "1.2. Parallel mechanism".

[0054] 1.3. System function

[0055] Referring to FIG. 4, the system function includes:

[0056] ① Intra-site ( / node) service classification function: Each level of site ( / node) undertakes main and auxiliary data (or only main data) sending (or reading), receiving (or writing), calculation and saving, etc.

[0057] ② Parallel processing dynamic function: Data flows from the first level of site ( / node) to the Nth level of site ( / node), and each level of site ( / node) relies on a multi-level tree branch structure to complete the interlaced data transmission and data calculation of sites ( / nodes), and then complete the multi-level site ( / node) data transmission and total task calculation.

[0058] ③ Communication mode construction function: The communication mode between sites ( / nodes) determines the application mode under the communication technology architecture, which is related to the operation, role and benefit of each part of the system; the communication between sites ( / nodes) and between them is long-range or short-range or both, which respectively shows the functions of the two combined mode, strong calculation mode and comprehensive mode of multi-level site ( / node) distributed computing resource data transmission and data calculation parallel processing.

[0059] ④ Overall structure mode function: two combined modes, each level of site ( / node) concurrently executes remote data transmission and concurrent calculation task allocation to superimpose distributed computing resources to process big data; or strong calculation mode, each level of site ( / node) concurrently executes short-range data dispatching and concurrent calculation task allocation to superimpose distributed computing resources to process big data; or comprehensive mode, each level of site ( / node) concurrently executes part of remote data transmission and part of short-range data dispatching, concurrent calculation task allocation to superimpose distributed computing resources to process big data.

[0060] 1.4. System module

[0061] Referring to FIG. 5, the system module, i.e., each level of site ( / node) system module, includes system preset, transmission module and calculation module.

[0062] 1.4.1. System preset

[0063] From the development process of data transmission and calculation and other information processing systems, from simple to complex, from independent system to application system on the basis of system. The present application is a non-independent system that implicitly exists with the basic system and application system, and does not necessarily be mentioned.

[0064] The system preset includes:

[0065] ① Determine the process form: data processing needs to determine the programming route and implementation of multi-task, multi-process and multi-thread.

[0066] ②Determine the communication transceiver mode: data communication needs to determine the specific communication line between sites (nodes), signal modulation and demodulation, and channel error correction coding and decoding, etc. The data transmission and reception communication mode is required to be matched.

[0067] ③Determine the signal transceiver matching: the signal transceiver needs to determine the signal quality and matching of the specific communication line, and arrange the signal low, medium and power amplifier appropriately.

[0068] ④Determine the data exchange mode: data exchange needs to determine the data exchange mode between data sending (or reading), receiving (or writing) and calculation, which can choose array or file or variable or memory, etc. as the exchange data storage place.

[0069] ⑤Compile the calculation task algorithm: different calculation tasks are allocated to each level of site (node), and the calculation task processing algorithm of each level of site (node) needs to be compiled.

[0070] ⑥File name specification naming: data file name involves data calling, saving, exchanging, etc. and should be named according to rules or regulations.

[0071] 1.4.2. Transmission module

[0072] The transmission module is divided into low, medium and high versions according to the degree of transparency of data transmission process for users, and the version is selected as needed. From the perspective of system user involvement, the basic system supporting data transmission link is briefly discussed.

[0073] (I) Low-level transmission module

[0074] Referring to FIG. 6, the data frame structure is designed, and the whole process of data sending and receiving such as data reading, packing and framing, data transmission, unframing and unpacking, and saving data is designed.

[0075] 1) Sending sub-module

[0076] According to the system preset conditions, the main and auxiliary data (or only the main data) is sent in a non-network communication environment.

[0077] ①Data sending start mechanism: when it is judged that the exchange data storage place exists data to be sent, the site (node) at this level executes the data sending transaction of the site (node) at this level.

[0078] ②Read and send data by block: according to the minimum data processing block main and auxiliary data (or only the main data) packing and framing plan, read the main data of the "sending" storage place where the data calculation is completed according to the format, and if the auxiliary data needs to be sent, read the auxiliary data of the corresponding "sending" storage place.

[0079] ③ Data packing and framing transmission: The packing and framing device or sending software processes the data synchronously or immediately after transmission in the minimum data processing block. When the main data is transmitted alone, the minimum data processing block is independently packed into one or more data packets in the packet format; or when the main and auxiliary data are transmitted in the same frame, the minimum data processing block is packed together to form one data packet in the packet format; or when the main and auxiliary data are transmitted in separate frames, the minimum data processing block is packed to form one or several independent data packets in the packet format; or the minimum data processing block is only about one sixth of the frame data area, and the minimum data processing block is used as a data sub-packet, and several sub-packets form a data packet. The data packet and channel are coded and framed in the frame format, and the minimum data processing block data is transmitted synchronously or immediately.

[0080] ④ Continuous sending of subsequent data: The subsequent data of the main and auxiliary data (or only the main data) is continuously read, packed, and framed to complete the transmission of the entire scene / sheet / track or the entire data.

[0081] ⑤ Cleaning up the sent data: The auxiliary data, main data (or only the main data) that have been sent, calculated, and saved in the exchange data storage are timely deleted to release the computing resources.

[0082] 2) Receiving sub-module

[0083] According to the system preset conditions, the main and auxiliary data (or only the main data) are received in a non-network communication environment.

[0084] ① Receiving full frame data: Each frame of data sent by a non-network system is completely received.

[0085] ② Receiving data unframing and unpacking: The unframing and unpacking device or receiving software unframes and unpacks each frame of data received according to the frame header and packet header marks, distinguishes the main and auxiliary data according to the data packet marks, and records the marks of the minimum data processing block data belonging to the same block.

[0086] ③ Saving the received data by block: The main and auxiliary data (or only the main data) are stored by the minimum data processing block according to the marks, and the auxiliary data and the main data (or only the main data) are saved in the corresponding data storage.

[0087] ④ Continuous receiving of subsequent data: The subsequent data of the main and auxiliary data (or only the main data) are continuously received, unframed and unpacked, and saved to complete the reception of the entire scene / sheet / track or the entire data.

[0088] (II) Intermediate version transmission module

[0089] Referring to FIG. 7, the user end utilizes the network communication protocol in the network environment or the general data communication system to exchange the main and auxiliary data (or only the main data) through the hardware or software interface according to the specification, and completes the data sending and receiving.

[0090] 1) Sending sub-module

[0091] Send main and auxiliary data (or only main data) in network or non-network communication environment according to system preset conditions.

[0092] ①Data sending start mechanism: when it is judged that there is data to be sent in the exchange data storage, the station (node) at this level executes the data sending transaction of the station (node) at this level.

[0093] ②Read and send data by block: according to the minimum data processing block main and auxiliary data (or only main data) packaging and framing plan, read the main data in the "sending" storage where the data calculation is completed, and if auxiliary data needs to be sent, read the auxiliary data in the corresponding "sending" storage.

[0094] ③Provide exchange data to the interface: the user end writes exchange data to the hardware or software interface specification, which is responsible for data packaging and framing and transmission by network communication protocol or general data communication machine.

[0095] ④Continue to send subsequent data: continuously read the subsequent data of the main and auxiliary data (or only main data), provide exchange data to the interface, and complete the whole scene [ / frame / track] or whole data sending.

[0096] ⑤Clean up the sent data: delete the auxiliary data, main data (or only main data) that have been sent, calculated and saved in the exchange data storage in time to release the calculation resources.

[0097] 2)Receiving sub-module

[0098] Receive main and auxiliary data (or only main data) in network or non-network communication environment according to system preset conditions.

[0099] ①Standard read interface data: the user end reads the unpacked data transmitted to the interface by network communication protocol or general data communication machine.

[0100] ②Save received data by block: store the main and auxiliary data (or only main data) according to the mark by the minimum data processing block, and save the auxiliary data and main data (or only main data) in the corresponding data storage.

[0101] ③Continue to receive subsequent data: continue to receive and save the subsequent data of the main and auxiliary data (or only main data), and complete the whole scene [ / frame / track] or whole data receiving.

[0102] (Three) Advanced transmission module

[0103] Referring to FIG. 8, under the support of the local system and bus, data transmission between the internal and external is transparent and smooth. The advanced platform can build transparent access of different addresses in network environment, and directly read and write data in the advanced platform environment as if it is a local machine.

[0104] ①Data sending start mechanism: When it is judged that there is data to be sent in the exchange data storage, the station (node) at this level executes the data sending transaction of the station (node) at this level.

[0105] ②Reading of the data to be sent: The transceiver software reads the main data in the "sending" storage of the station (node) at this level which has completed data calculation in the smallest data processing block as the processing unit according to the format, and reads the auxiliary data in the corresponding "sending" storage if the auxiliary data needs to be sent.

[0106] ③Writing of the data to be sent: The auxiliary data and the main data are written into the corresponding storage of the lower-level station (node) according to the format, or only the main data is written into the "receiving" storage according to the format.

[0107] ④Continuous forwarding of subsequent data: The auxiliary data and the main data (or only the main data) of each block of subsequent data of the whole scene (frame / track) are continuously forwarded to the lower-level station (node) in the smallest data processing block as the processing unit.

[0108] ⑤Cleaning up the data that has been sent: The auxiliary data and the main data (or only the main data) in the storage that have been sent, calculated and saved are deleted in time to release the calculation resources.

[0109] 1.4.3. Calculation module

[0110] Referring to FIG. 9, the calculation module comprises:

[0111] ①Adjustment of auxiliary data on demand: The stations (nodes) at each level collect the auxiliary data needed for adjustment in advance according to the progress of data calculation at each level.

[0112] ②Data calculation start mechanism: When it is judged that there is data to be calculated in the exchange data storage, the station (node) at this level executes the calculation transaction of the station (node) at this level.

[0113] ③Calculation of data according to the assigned task: The station (node) at this level calculates the corresponding main data saved in the "receiving" storage according to the calculation task processing algorithm assigned to the station (node) at this level according to the format, and saves the main data after calculation in the "sending" storage according to the format.

[0114] ④Continuous calculation of subsequent data: The station (node) at this level continuously calculates each block of subsequent data of the whole scene (frame / track) or the whole in the smallest data processing block as the processing unit.

[0115] ⑤Adjustment of the auxiliary data to be sent: If the auxiliary data needs to be sent, the original or adjusted auxiliary data is saved in the "sending" storage.

[0116] 6. Save the sent, received, calculated data: Each level of site (node) saves the received (or written) whole scene [ / frame / track] or whole body data and auxiliary data (or only body data) processed by the site (node) in file form in time according to the format to the file receiving, processing and forwarding directory, and the file name follows the file naming rules and marks the corresponding data level.

[0117] 2. Implementation application mode

[0118] The parallel processing method and device overall mode are summarized from four aspects of system structure, parallel mechanism, system function, system module, and show obvious different characteristics in specific implementation form and technical means from three high, medium and low versions of data transmission. Especially in the long-range and short-range communication mode between sites (nodes), it promotes the application system of the application to change in structure and function, and the system structure and function are more specific and characteristic in space. The communication mode between sites (nodes) determines the application mode under the communication technology framework, which is related to the operation, role playing and benefit of each part of the system, and forms three application modes.

[0119] The first application mode shows different characteristics in data transmission and data calculation. The long-range communication between sites (nodes) emphasizes data transmission tasks as the core and adds data calculation benefits. In the system structure, due to the long-range communication between sites (nodes), the sites (nodes) and the long-range communication between them form a multi-level site (node) long-range data communication system and a computer (processor / computing core) cluster of distributed computing resources, even a supercomputer (processor / computing core) cluster. In the system function, the long-range communication between sites (nodes) shows the function of the two-in-one mode of data transmission and data calculation parallel processing of multi-level site (node) distributed computer resources; the two-in-one mode, each level of site (node) concurrently executes long-range data transmission and concurrent computing task distribution, to superimpose the super computing power of distributed computing resources to process big data.

[0120] The third application mode is to highlight data transmission and data calculation in the local area. The inter-site (inter-node) long-range and short-range communication can highlight data transmission and data calculation in the local area. In the system structure, the inter-site (inter-node) long-range and short-range communication forms a multi-level site (node) local long-range and local short-range data communication system, and a computer (processor / computing core) cluster of distributed computing resources, even a supercomputer (processor / computing core) cluster. In the system function, the inter-site (inter-node) long-range and short-range communication shows the comprehensive mode function of multi-level site (node) distributed computing resources for parallel processing of data transmission and data calculation. In the comprehensive mode, each level of site (node) concurrently executes partial long-range data transmission and partial short-range data dispatch, and concurrently computes and dispatches tasks, so as to superimpose the supercomputing power of distributed computing resources to process big data.

[0121] The third application mode is to highlight data transmission and data calculation in the local area. The inter-site (inter-node) long-range and short-range communication can highlight data transmission and data calculation in the local area. In the system structure, the inter-site (inter-node) long-range and short-range communication forms a multi-level site (node) local long-range and local short-range data communication system, and a computer (processor / computing core) cluster of distributed computing resources, even a supercomputer (processor / computing core) cluster. In the system function, the inter-site (inter-node) long-range and short-range communication shows the comprehensive mode function of multi-level site (node) distributed computing resources for parallel processing of data transmission and data calculation. In the comprehensive mode, each level of site (node) concurrently executes partial long-range data transmission and partial short-range data dispatch, and concurrently computes and dispatches tasks, so as to superimpose the supercomputing power of distributed computing resources to process big data.

[0122] 3. Related statements

[0123] 3.1. Site (node) computer (processor / computing core)

[0124] The related site (node) of the application is composed of at least one computer (processor / computing core), but there is no limitation on the number of site (node) computers (processors / computing cores). In actual application, the high-performance configuration of the site (node) should be considered, and the computer (processor / computing core) responsible for data calculation transaction should be equipped with sufficient resources such as high-speed memory and high-level CPU or GPU, MIC (integrated multi-core), TPU (tensor processor), DPU (data processor), and even new type of processor, supercomputer or high-performance parallel computer to improve the computing capacity of the site (node). The site (node) of the application includes the best and non-best computer configuration.

[0125] It is a predictable structure of the present application that a certain site (node) is arranged not to perform data calculation task at each site (node) of data transmission and data calculation.

[0126] 3.2. Communication line between sites (nodes)

[0127] The communication line between sites (nodes) includes wired and wireless communication lines. The wired line is a communication line that uses a condensed state medium with good electrical conductivity or light conductivity to transmit information, such as optical fiber, twisted pair, coaxial cable, etc. The wireless line is a communication line that uses electromagnetic wave spectrum resources to transmit in the air, such as radio wave, microwave, and infrared, etc. The dedicated line includes optical fiber, DDN (Digital Data Network), ISDN (Integrated Services Digital Network), frame relay, packet switching, PSTN (Public Switched Telephone Network), etc. The communication line can be rented by the operator, such as renting a dedicated line, renting a satellite or ground microwave channel, etc. Wireless channels with approved frequency bands or specific data communication using public frequency bands can be used. In-house bus and other dedicated lines can also be used. The wired communication line can be self-built. The communication line includes future new types of communication lines.

[0128] 3.3. Data communication system

[0129] The data communication system refers to network and non-network data transmission systems, which are divided into three types: network, general, and private. The network data communication system refers to various forms of communication systems that meet the computer communication network standards, such as various communication systems of the current IEEE (Institute of Electrical and Electronics Engineers) 802 standard. The general data communication system is a space data collection communication system such as CCSDSAOS (Consultative Committee for Space Data Systems Advanced On-Orbit System), or a digital television data transmission system such as DTMB (China Digital Terrestrial Multimedia Broadcast), DVB-S2 (European Second Generation Satellite Digital Television Broadcast), DVB-T2 (European Second Generation Terrestrial Digital Television Broadcast), etc. The private data communication system refers to a data communication system that is not general and independent or uses a private communication protocol. From the user's perspective of data transmission transparency, the general data communication system uses an intermediate version of the transmission module, and the private data communication system uses a low-level version of the transmission module. In order to improve data communication capability, the data communication system can use parallel technologies such as space division multiplexing, time division multiplexing, frequency division multiplexing, and phase division multiplexing.

[0130] 3.4. Minimum data processing block calculation time

[0131] The remote data transmission speed is relatively slow compared to the computer (processor / computing core) computing speed. The data computing time of the smallest data processing block in the business processing module should be less than its data transmission time, the purpose is to avoid delaying the timeliness of the overall data transmission. In the task of data transmission and data calculation, although the data transmission task is primary and key, but the execution of data calculation task cannot delay the whole data transmission time, otherwise the site (node) data calculation task needs to be adjusted or better computing resources are selected to ensure high time efficiency of data transmission process. The present application includes that the data computing time of the smallest data processing block can be less than or equal to, greater than its data transmission time.

[0132] 3.5. Data exchange mode

[0133] The business modules can exchange data by file, such as database table file, text file and binary file, and can also exchange data by array, variable symbol, memory and other ways. Considering the read-write speed, the storage medium for exchanging data is preferably memory, and other media such as solid state disk can also be used.

[0134] 3.6. File naming rules

[0135] The data file name should be named in a standard way, such as the full name of the file name including the file name and the suffix name, the file name including the data source, the data latitude and longitude, the data date, the data time, the data category, the data code, etc. The interval symbol can be used "-". The suffix name represents the file category and is identified according to the convention. Taking satellite remote sensing data as an example, the data source is the satellite name and the detector name, the data latitude and longitude is the latitude and longitude of the monitored place, the data date is the specific date of remote sensing data collection, the data time is the specific time of remote sensing data collection, the data category is the data level or data processing batch, the data code is the data product number, and the suffix name represents the data file category. The file name, array name and variable symbol used for exchanging data, and the file name for saving data should comply with the standard naming. The present application is not limited to specific file naming and naming method, including using non-standard named file name.

[0136] 3.7. Multi-channel data processing

[0137] In practical applications, the present application can be extended to a comprehensive system to process multiple different source data in a time-sharing manner. The present application takes one data as an example, which can be extended to multiple different data from the same source, such as multiple acquisition data from different detectors on the same satellite, or multiple different data from different sources, such as multiple acquisition data from different detectors on different satellites. Each satellite remote sensing data and telemetry data may have different parameters and formats. The present application can use multiple source data processing mechanisms and technical solutions, as well as time conflict processing mechanisms. In multi-source data processing, intermediate sites (nodes) can insert data transmission tasks, which still belong to the technical solutions of the present application. In multi-source data processing, some transmission data may not be arranged for data calculation tasks, which still belong to the technical solutions of the present application.

[0138] 3.8. Duplex mode of the present application

[0139] The present application only describes a parallel processing method and device for one-direction multi-level data transmission and data calculation. If the reverse direction conforms to the relevant description of the present application, the forward direction and the direction partially or mostly or entirely overlap with the equipment or system, which belongs to the duplex mode of the present application, and still belongs to the protection scope of the present application.

[0140] 3.9. Related protection scope

[0141] The parallel processing method and device for multi-level data transmission and data calculation are described from the aspects of system structure, parallel mechanism, system function, and system module, etc. in combination with the drawings, to realize the efficient parallel processing function of multi-level site (node) data transmission and data calculation. The implementation process of the present application is complex, and there are many different implementation ways. Some specific ways are used in the description of the embodiments, which aims to show actual examples, but the present application is not limited to the described implementation ways. If different implementation forms are designed according to the teaching of the present application, and these changes, modifications, replacements, and variations of the implementation ways do not deviate from the principles and spirits of the present application, they still fall into the protection scope of the present application.

[0142] 4. Example description

[0143] Each mode has two examples, a total of six embodiments.

[0144] 4.1. Example 1: Mode 1 Example 1. This embodiment takes the real-time data of YL1 (Example 1) satellite CCD detector as an example. YL1 satellite is in sun-synchronous orbit, and the real-time data of CCD detector from YL1 satellite ground station includes auxiliary data and main data. Auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, satellite time, and other recorded data, as well as monitoring data of attitude control system, communication system, and power system. Main data is remote sensing data collected by CCD detector on satellite. Satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction, for daily monitoring by station department. This embodiment describes a multi-level data transmission and parallel processing method and device for data transmission by CCSDS AOS&XTCE (XML-based telemetry and command exchange standard) remote general data communication system, including system structure, parallel mechanism, system function, and system module.

[0145] 4.1.1. System structure

[0146] The system structure includes:

[0147] ① Computer station composition: the station is composed of 5 levels of stations from the 1st level to the 5th level, and the station is usually composed of a network of 3 computers.

[0148] ② Inter-station communication system: the stations are connected by remote data communication system. CCSDS AOS&XTCE remote general data communication system is adopted, satellite channel or dedicated line is rented, or wireless channel is applied.

[0149] ③ Overall structure function system: the 5 levels of stations and their remote communication constitute a multi-level station remote data communication system and a supercomputer cluster of independent distributed computing resources.

[0150] ④ Tree branch type static structure: the 5 levels of stations form a multi-level branch tree structure.

[0151] ⑤ Station data processing module: the station system module is divided into data sending, receiving, and computing modules.

[0152] ⑥ Parallel processing dynamic mode: data is transmitted from the 1st level station to the 5th level station in parallel, and the data processing tasks of the station are dispatched in staggered computation.

[0153] 4.1.2. Parallel mechanism

[0154] The parallel mechanism includes:

[0155] (1) Minimum data processing block unit. The design of each level of site data exchange and data transmission, reception, calculation and other data processing services are processed in the minimum data processing block unit according to the specified format. The minimum data processing block is used as a processing unit to block the whole scene data, and the auxiliary data is synchronized or transmitted immediately after the transmission of the main data, providing an opportunity for concurrent data transmission and calculation, relying on parallel processing of data transmission and data calculation to achieve high efficiency of data transmission and data calculation.

[0156] (2) Minimum data processing block setting. The minimum data processing block includes auxiliary data and main data. The telemetry data is auxiliary data, and the remote sensing data is main data. According to the actual data acquisition situation, the corresponding relationship and quantity of telemetry data and remote sensing data collected on the satellite in this example are determined. The telemetry data related to the correction of CCD detector remote sensing data is integrated into 1 frame transmission, and the 30 frames of remote sensing data corresponding to this frame of telemetry data are determined as the minimum data processing block, which is 1 frame of telemetry data and 30 frames of remote sensing data. The data volume of the minimum data processing block is relatively small, which is more conducive to reducing the time delay and improving the timeliness of overall data transmission and calculation.

[0157] (3) Data transmission and calculation interaction mechanism. The minimum data processing block is used as a unit to stagger the continuous data transmission and data calculation of the whole scene data, forming a parallel processing linkage mechanism of data transmission and data calculation at each level of site. ① Data block transmission: each scene data is transmitted block by block, and the in-station transmission task is completed block by block; in N-site transmission, each block of data is transmitted from station to station. ② Data block calculation: each scene data calculation task is assigned to each site according to the plan, and the calculation task is allocated according to the block processing, and the data calculation task is completed together by each station. ③ Interleaved transmission and calculation: each scene data block is transmitted from station to station, and each station calculates the data according to the block, and the transmission and calculation tasks of the main data and the auxiliary data are completed in the form of interleaved transmission and calculation at each station.

[0158] 4.1.3. System function

[0159] The system function includes:

[0160] ① In-station service classification function: each level of site undertakes the tasks of main and auxiliary data transmission, reception, calculation and saving.

[0161] ② Parallel processing dynamic function: data flows from the first level of site to the fifth level of site, and each level of site relies on the multi-level branch structure to complete the interleaved data transmission and data calculation of the site, and to complete the data transmission of multiple levels of sites in succession, and to complete the 0-4 level data correction task calculation in succession.

[0162] ③Communication mode configuration function: The inter-site remote communication mode determines the application mode under the communication technology architecture, which is related to the operation of each part of the system, plays a role, and produces benefits; inter-site remote communication shows the two-in-one mode function of parallel processing of data transmission and data calculation of multi-level site independent distributed computer resources.

[0163] ④Overall structure mode function: Two-in-one mode, concurrent execution of remote data transmission at each level of site, concurrent calculation of task allocation, super-computing power processing of big data by superimposed distributed computing resources.

[0164] 4.1.4. System module

[0165] The system module, i.e. the system module of each level of site, includes system preset, transmission module and calculation module.

[0166] 4.1.4.1. System preset

[0167] The system preset includes:

[0168] ①Determine the program process form: Data processing needs to determine the programming route and implementation of multi-task, multi-process and multi-thread.

[0169] ②Determine the communication transceiver mode: Data communication needs to determine the specific communication line, signal modulation and demodulation, channel error correction encoding and decoding mode between sites, and requires data transmission and reception communication mode matching.

[0170] ③Determine the signal transceiver matching: Signal transceiver needs to determine the signal quality and matching of specific communication line, and arrange the signal low, medium and power amplifier appropriately.

[0171] ④Determine the data exchange mode: Data exchange needs to determine the data exchange mode between data transmission, reception and calculation, and select array as the exchange data storage place.

[0172] ⑤Compile the calculation task algorithm: Each level of site is assigned different calculation tasks, and the calculation task processing algorithm of each level of site needs to be compiled first.

[0173] ⑥File name specification naming: Data file name involves data calling, saving, exchanging, etc., and should be named according to rules or regulations.

[0174] 4.1.4.2. Transmission module

[0175] Select to use the intermediate version transmission module. The user end uses CCSDS AOS & XTCE general data communication system to exchange main and auxiliary data through hardware interface according to specifications, completes data transmission and reception.

[0176] The sending submodule includes:

[0177] According to system preset conditions, send main and auxiliary data in non-network communication environment.

[0178] ①Data sending start mechanism: when it is judged that there is data to be sent in the exchange data storage, the station at this level executes the data sending transaction of the station at this level.

[0179] ②Reading and sending data by block: according to the minimum data processing block main and auxiliary data packaging and framing plan, the main data in the "sending" storage where the data calculation is completed is read in the format, and if the auxiliary data needs to be sent, the auxiliary data in the corresponding "sending" storage is also read.

[0180] ③Providing exchange data to the interface: the user end writes the exchange data to the hardware interface specification, which is responsible for data packaging and framing and transmission by the general data communication machine.

[0181] ④Continuously sending subsequent data: continuously reading the subsequent data of the main and auxiliary data, providing the interface exchange data, and completing the whole scene data sending.

[0182] ⑤Cleaning up the sent data: deleting the auxiliary data, main data in the exchange data storage that have been sent, calculated and saved in time, and releasing the computer resources.

[0183] The receiving submodule includes:

[0184] According to system preset conditions, receive main and auxiliary data in non-network communication environment.

[0185] ①Standard reading of interface data: the user end reads the unpacked data transmitted to the interface by the data unpacker.

[0186] ②Saving received data by block: according to the mark, save the main and auxiliary data in the minimum data processing block, and save the auxiliary data and the main data in the corresponding data storage.

[0187] ③Continuously receiving subsequent data: continuously receiving and saving the subsequent data of the main and auxiliary data, and completing the whole scene data receiving.

[0188] 4.1.4.3. Calculation module

[0189] The calculation module includes:

[0190] ①Adjusting auxiliary data on demand: each level station collects the auxiliary data needed for adjustment in advance according to the progress of data calculation at each level.

[0191] ②Data calculation start mechanism: when it is judged that there is data to be calculated in the exchange data storage, the station at this level executes the calculation transaction of the station at this level.

[0192] ③According to the assigned task, the data is calculated: the required parameter data and the auxiliary data in the "receiving" storage are collected according to the format, the corresponding main data in the "receiving" storage which has not been processed is calculated according to the calculation algorithm of the calculation task assigned to the site, and the main data after calculation is saved in the "sending" storage according to the format.

[0193] ④Continuously calculate subsequent data: continuously calculate each block of data in the whole scene as the minimum data processing block.

[0194] ⑤Adjust the auxiliary data sent: if the auxiliary data needs to be sent, the original or adjusted auxiliary data is saved in the "sending" storage.

[0195] ⑥Save the sent, received and calculated data: each site saves the received, processed whole scene main data and auxiliary data in the file receiving, processing and forwarding directory in the form of a file according to the format, the file name follows the file naming rules, and the corresponding data level is marked.

[0196] 4.2. Example 2: Mode 1 Example 2. This embodiment takes the non-real-time data of YL2 (Example 2) satellite WFV detector as an example. YL2 satellite is in sun-synchronous orbit, and the non-real-time data of WFV detector from YL2 satellite ground station includes auxiliary data and main data. The auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, satellite time and other recorded data, and monitoring data of attitude control system, communication system and power system. The main data is the remote sensing data collected by the WFV detector on the satellite. The satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction, for daily monitoring by site (node) departments. This embodiment describes a parallel processing method and device for multi-level data transmission and data calculation of data transmission by wide area network DDN dedicated line, including system structure, parallel mechanism, system function and system module.

[0197] 4.2.1. System structure

[0198] The system structure includes:

[0199] ① Computer site (node) composition: the site (node) is composed of 4 levels of site (node), i.e. the 1st level site (node) to the 4th level site (node), and each level of site (node) is composed of 3 computers forming a network.

[0200] ② Communication system between sites (nodes): the sites (nodes) are connected by a remote computer communication network (wide area network). The remote network version of the number transmission system between sites (nodes) is composed of two ends of the network and intermediate subnets, and the DDN dedicated line is rented.

[0201] 3. Overall structure and function system: 4-level stations (nodes) and their remote communication constitute a multi-level station (node) remote data communication system and a supercomputer cluster of distributed computing resources.

[0202] 4. Tree branch type static structure: 4-level stations (nodes) form a multi-level branch tree structure.

[0203] 5. Station (node) data processing module: the station (node) system module is divided into data reading, writing, calculation and other modules.

[0204] 6. Parallel processing dynamic mode: data is transmitted from the first level station (node) to the fourth level station (node) in parallel, and the data processing tasks of the station (node) are dispatched in staggered calculation.

[0205] 4.2.2. Parallel mechanism

[0206] The parallel mechanism includes:

[0207] (1) Minimum data processing block unit. The data exchange and data transmission, reception, calculation and other data processing services of each level station (node) are processed in the form of a minimum data processing block unit. The whole scene data is divided into blocks with the minimum data processing block as the processing unit. The main and auxiliary data are synchronized or transmitted immediately, providing an opportunity for concurrent data transmission and calculation, relying on parallel processing of data transmission and data calculation, and achieving high efficiency of data transmission and data calculation.

[0208] (2) Minimum data processing block setting. The minimum data processing block includes auxiliary data and main data. Telemetry data is auxiliary data and remote sensing data is main data. According to the actual data acquisition situation, the corresponding relationship and quantity of telemetry data and remote sensing data collected on the satellite in this example are determined. The telemetry data related to the correction of WFV detector remote sensing data is integrated into one frame for transmission, and this frame of telemetry data corresponds to 50 frames of remote sensing data, so the minimum data processing block is determined to be one frame of telemetry data and 50 frames of remote sensing data. The minimum data processing block has relatively small data volume, which is more conducive to reducing latency and improving the timeliness of overall data transmission and calculation.

[0209] (3) Data transmission and calculation interaction mechanism. With the minimum data processing block as the unit, the whole scene data is staggered for continuous data transmission and data calculation, forming a parallel processing linkage mechanism of data transmission and data calculation of each level of site (node). ① Data block transmission: each scene data is transmitted block by block, and the transmission task within the site (node) is completed block by block; in N-site (node) transmission, each block of data is transmitted from site to site (node). ② Data block calculation: each scene data calculation task is assigned to each site (node) according to the plan, and the calculation task is allocated according to the block processing, and the data calculation task is completed together by each site (node). ③ Staggered transmission and calculation: each scene data block is transmitted from site to site (node), and each site (node) calculates data by block to complete the transmission and calculation tasks of main data and auxiliary data in a staggered transmission and calculation manner.

[0210] 4.2.3. System function

[0211] The system function includes:

[0212] ① In-site (node) service classification function: each level of site (node) undertakes the services of main and auxiliary data reading, writing, calculation and saving.

[0213] ② Parallel processing dynamic function: data flows from the first level of site (node) to the fourth level of site (node), and each level of site (node) completes the staggered data transmission and data calculation of the site (node) depending on the multi-level tree branch structure, and then completes the multi-level site (node) data transmission, and then calculates the 0-4 level data correction task.

[0214] ③ Communication mode construction function: the remote communication mode between sites (nodes) determines the application mode under the communication technology architecture, which is related to the operation, role and benefit of each part of the system; the remote communication between sites (nodes) shows the two-in-one mode function of parallel processing of data transmission and data calculation of multi-level site (node) distributed computer resources.

[0215] ④ Overall structure mode function: two-in-one mode, each level of site (node) concurrently executes remote data transmission and concurrently calculates the dispatched task, to supercomputing power processing big data by superimposed distributed computing resources.

[0216] 4.2.4. System module

[0217] The system module, i.e. the system module of each level of site (node), includes system preset, transmission module and calculation module.

[0218] 4.2.4.1. System preset

[0219] The system preset includes:

[0220] ① Determine the process form: data processing needs to determine the programming route and implementation of multi-task, multi-process, multi-thread.

[0221] ② Determine the communication mode: data communication needs to determine the specific communication line between sites (nodes), signal modulation and demodulation, and channel error correction coding and decoding, etc. The data transmission and reception communication mode is required.

[0222] ③ Determine the signal transceiver matching: signal transceiver needs to determine the signal quality and matching of specific communication line, and arrange the signal low, medium and power amplifier appropriately.

[0223] ④ Determine the data exchange mode: data exchange needs to determine the data exchange mode between data reading, writing and calculation, and select the file as the exchange data storage place.

[0224] ⑤ Compile the calculation task algorithm: different calculation tasks are assigned to each level of site (node), and the calculation task processing algorithm of each level of site (node) needs to be compiled.

[0225] ⑥ File name specification naming: data file name involves data calling, saving, exchanging, etc., and should be named according to rules or regulations.

[0226] 4.2.4.2. Transmission module

[0227] Select the use of high-level transmission module. The high-level platform can build transparent access to different addresses in network environment, and can read and write data directly as if it were local in high-level platform environment.

[0228] ① Data sending start mechanism: when it is judged that there is data to be sent in the exchange data storage place, the site (node) at this level executes the data sending transaction of the site (node) at this level.

[0229] ② Read transceiver data: the transceiver software takes the minimum data processing block as the processing unit, reads the main data of the "sending" storage place of the site (node) at this level which has completed data calculation according to the format, and reads the auxiliary data corresponding to the "sending" storage place.

[0230] ③ Write transceiver data: write auxiliary data and main data into the corresponding storage place of the lower level site (node) "receiving" according to the format.

[0231] ④ Continue to forward subsequent data: continue to forward the auxiliary data and main data of each block of subsequent data to the lower level site (node) with the minimum data processing block as the processing unit.

[0232] ⑤ Clean up the sent data: delete the auxiliary data and main data in the storage place which have been sent, calculated and saved in time to release computer resources.

[0233] 4.2.4.3. Computing module

[0234] The computing module includes:

[0235] ①Adjustment of auxiliary data on demand: each level of site ( / node) collects auxiliary data needed for adjustment in advance according to the progress of data level-by-level calculation.

[0236] ②Data calculation starting mechanism: when it is judged that there is data that needs to be calculated in the exchange data storage place, the site ( / node) at this level executes the calculation transaction of the site ( / node) at this level.

[0237] ③Calculation of data according to assigned tasks: according to the format, the parameters data and auxiliary data needed by the site ( / node) at this level are collected and saved in the "receiving" storage place. The corresponding main data that has not been processed and is saved in the "receiving" storage place is calculated block by block according to the calculation task processing algorithm assigned to the site ( / node) at this level, and the main data after calculation is saved in the "sending" storage place according to the format.

[0238] ④Continuous calculation of subsequent data: the minimum data processing block is taken as the processing unit to continuously calculate the whole scene subsequent data.

[0239] ⑤Adjustment and sending of auxiliary data: if auxiliary data needs to be sent, the original or adjusted auxiliary data is saved in the "sending" storage place.

[0240] ⑥Saving of sent, received and calculated data: each level of site ( / node) saves the whole scene main data and auxiliary data written by the site ( / node) at this level in the form of a file in the file receiving, processing and forwarding directory in a timely manner according to the format. The file name follows the file naming rules and indicates the corresponding data level.

[0241] 4.3. Example three: Mode 2 example 1. This embodiment takes the non-real-time data of YL1 (example 1) satellite CCD detector as an example. YL1 satellite is in sun-synchronous orbit. The real-time data of CCD detector from YL1 satellite when passing through ground station includes auxiliary data and main data. Auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time and other recorded data, as well as monitoring data of attitude control system, communication system and power system. The main data is remote sensing data collected by on-board CCD detector. The satellite remote sensing data transmission process arranges data correction, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction, to provide data service for user daily monitoring in time. This embodiment describes a parallel processing method and device for multi-level data transmission and data calculation through parallel port direct connection, including system structure, parallel mechanism, system function and system module.

[0242] 4.3.1 System structure

[0243] The system structure comprises:

[0244] ① Computer site composition: The site is composed of 4 levels of sites, i.e. the 1st level site to the 4th level site, and each level of site is composed of 3 computers to form a network.

[0245] ② Inter-site communication system: The data is transmitted between sites through parallel port direct connection, which is a short-range non-computer communication network connection, and a non-network version short-range data transmission system using private communication technology.

[0246] ③ Overall structure function system: The 4th level site and the short-range communication therebetween form a multi-level site short-range data communication system and a supercomputer cluster of independent distributed computing resources.

[0247] ④ Tree branch type static structure: The 4th level site forms a multi-level branch tree structure.

[0248] ⑤ Site data processing module: The site system module is divided into data reading, writing, calculation and other modules.

[0249] ⑥ Parallel processing dynamic mode: Data is transmitted from the 1st level site to the 4th level site in parallel, and the data processing tasks of the site are dispatched in staggered calculation.

[0250] 4.3.2. Parallel mechanism

[0251] The content herein is the same as that of "4.1.2. Parallel mechanism".

[0252] 4.3.3. System function

[0253] The system function comprises:

[0254] ① Intra-site business classification function: Each level of site undertakes main and auxiliary data reading, writing, calculation and saving and other businesses.

[0255] ② Parallel processing dynamic function: Data flows from the 1st level site to the 4th level site, and each level of site completes site staggered data transmission and data calculation depending on the multi-level tree branch type structure, and the multi-level site data transmission is completed in succession, and the 0-4 level data correction task calculation is completed in succession.

[0256] ③ Communication mode construction function: The communication mode between sites determines the application mode under the communication technology framework, which is related to the operation, role playing and benefit generation of each part of the system; the short-range communication between sites shows the strong calculation mode function of the multi-level site independent distributed computer resource data transmission and data calculation parallel processing.

[0257] ④ Overall structure mode function: Strong calculation mode, each level of site executes short-range data dispatching in parallel and calculates the task dispatching in parallel, to process big data with super computer power of superposed distributed computing resources.

[0258] 4.3.4. System module

[0259] The system module, i.e. the system module of each level station, includes system preset, transmission module and calculation module.

[0260] 4.3.4.1. System preset

[0261] The system preset includes:

[0262] The contents of ①②③ here are the same as those of "4.1.4.1. System preset" ①②③.

[0263] ④ Determine the data exchange mode: data exchange needs to determine the data exchange mode between data reading, writing and calculation, and select a file as the exchange data storage place.

[0264] The contents of ⑤⑥ here are the same as those of "4.1.4.1. System preset" ⑤⑥.

[0265] 4.3.4.2. Transmission module

[0266] Select to use a low-level transmission module. Design the data frame structure, data reading, packing and framing, data transmission, unframing and unpacking, and saving data, etc. the whole process of data sending and receiving.

[0267] The sending submodule includes:

[0268] In accordance with the system preset conditions, send the main and auxiliary data in a non-network communication environment.

[0269] ① Data sending start mechanism: when it is judged that the exchange data storage place exists data that needs to be sent, the station of this level executes the data sending transaction of this level station immediately.

[0270] ② Read and send data by block: according to the minimum data processing block main and auxiliary data packing and framing plan, read the main data of the "sending" storage place where the data calculation has been completed according to the format, and if the auxiliary data needs to be sent, also read the auxiliary data of the corresponding "sending" storage place.

[0271] ③ Data packing and framing transmission: process the data according to the minimum data processing block main and auxiliary data immediately after transmission. When the main and auxiliary data are framed and transmitted, the auxiliary and main data are formed into 1 to 50 independent data packets according to the packet format. According to the frame format, the data packets and channel coding are framed, and the synchronization or immediate transmission of the minimum data processing block data.

[0272] ④ Continue to send subsequent data: continuously read and pack the subsequent data of the main and auxiliary data, and complete the whole scene data sending.

[0273] ⑤ Clean up the sent data: delete the auxiliary data, main data that have been sent, calculated and saved in the exchange data storage place in time, and release the computer resources.

[0274] The receiving sub-module comprises:

[0275] According to the system preset condition, the main and auxiliary data are received in a non-network communication environment.

[0276] ① Full frame data receiving: each frame of data sent by the parallel port is received in full.

[0277] ② Receiving data unloading and unpacking: each frame of received data is unloaded and unpacked according to the frame header and packet header marks, the main and auxiliary data are distinguished according to the data packet marks, and the data memory marks of the same block of minimum data processing blocks are recorded.

[0278] ③ Saving received data by blocks: the main and auxiliary data are saved according to the minimum data processing blocks according to the marks, and the auxiliary data and the main data are saved in the corresponding data storage places.

[0279] ④ Continuous receiving of subsequent data: the subsequent data of the main and auxiliary data are continuously received, unloaded, unpacked and saved, and the whole scene data receiving is completed.

[0280] 4.3.4.3. Computing module

[0281] The content herein is the same as that of "4.1.4.3. Computing module".

[0282] 4.4. Example four: mode 2 example 2. This embodiment takes the small version language model data processing as an example. According to the user's query data, the artificial intelligence language model decomposes the total target task, including five aspects of intermediate core process data processing such as text preprocessing, Tokenization, vectorization, neural network inference and output generation, and needs to complete the data delivery and dispatch execution computing task of the computing device. The user's query data is the main data. The parallel processing method and device of multi-level data transmission and data calculation of InfiniBand transmission data are described, including system structure, parallel mechanism, system function and system module.

[0283] 4.4.1. System structure

[0284] The system structure comprises:

[0285] ① Computer site ( / node) composition: the site ( / node) is composed of 5 levels of sites ( / nodes), i.e. the first level of site ( / node) to the fifth level of site ( / node), and each site ( / node) is composed of 8 GPCs (GPU processing clusters).

[0286] ② Inter-site ( / node) communication system: short-range communication between sites ( / nodes) uses InfiniBand connection to transmit data, and InfiniBand is an I / O network communication system.

[0287] 3. Overall structure function system: 5-level stations (nodes) and their short-range communication between them constitute a multi-level station (node) short-range data communication system and a supercomputer cluster of distributed computing resources.

[0288] 4. Tree branch type static structure: 5-level stations (nodes) form a multi-level branch tree structure.

[0289] 5. Station (node) data processing module: the station (node) system module is divided into data sending, receiving, computing and other modules.

[0290] 6. Parallel processing dynamic mode: data is transmitted from the first level station (node) to the fifth level station (node) in parallel, and the data processing tasks of the station (node) are dispatched in staggered computing.

[0291] 4.4.2. Parallel mechanism

[0292] The parallel mechanism includes:

[0293] The contents of this place (1) (2) (3) are the same as those of "1.2A. Parallel mechanism [suitable for large model]" (1) (2) (3).

[0294] 4.4.3. System function

[0295] The system function includes:

[0296] 1. Station (node) internal service classification function: each level of station (node) undertakes the main data sending, receiving, computing and saving services.

[0297] 2. Parallel processing dynamic function: data flows from the first level station (node) to the fifth level station (node), and each level of station (node) relies on the multi-level tree branch structure to complete the staggered data transmission and data calculation of the station (node), and then complete the multi-level station (node) data transmission, and the computing power completes the 0-4 level data correction task calculation.

[0298] 3. Communication mode construction function: the short-range communication mode between stations (nodes) determines the application mode under the communication technology architecture, which is related to the operation, role and benefit of each part of the system; The short-range communication between stations (nodes) shows the strong computing mode function of the parallel processing of data transmission and data calculation of multi-level station (node) distributed computer resources.

[0299] 4. Overall structure mode function: strong computing mode, each level of station (node) executes concurrent short-range data dispatching and concurrent computing task dispatching to supercomputing power processing of distributed computing resources to process big data.

[0300] 4.4.4. System module

[0301] The system module, i.e. the system module of each level of station (node), includes system preset, transmission module and calculation module.

[0302] 4.4.4.1. System preset

[0303] The system preset includes:

[0304] The contents of ①②③ here are the same as those of "4.2.4.1. System preset" ①②③.

[0305] ④ Determine the data exchange mode: data exchange needs to determine the data exchange mode between data sending, receiving and calculation, and select an array as the storage place of exchanged data.

[0306] The contents of ⑤⑥ here are the same as those of "4.2.4.1. System preset" ⑤⑥.

[0307] 4.4.4.2. Transmission module

[0308] Select to use the advanced version of the transmission module. Under the support of the local system and bus, data transmission between the internal and external is transparent and smooth.

[0309] ① Data sending start mechanism: when it is judged that the storage place of exchanged data exists data to be sent, the station (node) of this level executes the data sending transaction of the station (node) of this level.

[0310] ② Reading of transceiving data: the transceiving software takes the minimum data processing block as the processing unit, reads the main data of the "sending" storage place of the station (node) of this level which has completed data calculation according to the format.

[0311] ③ Writing of transceiving data: write the main data into the corresponding storage place of the lower level station (node) "receiving" according to the format.

[0312] ④ Continuous forwarding of subsequent data: continuously forward the main data of each block of the whole data to the lower level station (node) in the minimum data processing block as the processing unit.

[0313] ⑤ Clean up the sent data: delete the main data in the storage place which has been sent, calculated and saved in time, and release the computer resources.

[0314] 4.4.4.3. Calculation module

[0315] The calculation module includes:

[0316] ① Adjust the auxiliary data as needed: each level of station (node) collects the required parameter auxiliary data in advance according to the progress of data calculation at each level.

[0317] ②Data computation start mechanism: when it is judged that there is data to be computed in the exchange data storage, the station (node) at this level executes the computation transaction of the station (node) at this level.

[0318] ③Computing data according to assigned tasks: the station (node) at this level collects various parameter data and auxiliary data in the "receiving" storage according to the format, computes the corresponding subject data saved in the "receiving" storage according to the computation task processing algorithm assigned to the station (node) at this level in blocks, and saves the subject data after computation in the "sending" storage according to the format.

[0319] ④Continuously computing subsequent data: continuously computing each block of data in the whole scene with the smallest data processing block as the processing unit.

[0320] ⑤Adjusting auxiliary data to be sent: if the subordinate station (node) needs predetermined parameter auxiliary data, the collected auxiliary data is saved in the "receiving" storage.

[0321] ⑥Saving the sent, received and computed data: the station (node) at each level saves the whole subject data and auxiliary data written by the station (node) at this level in the form of a file in the file receiving, processing and forwarding directory in time according to the format, the file name follows the file naming rules, and the corresponding data level is marked.

[0322] 4.5. Example 5: Mode 3 Example 1. This embodiment takes the real-time data of YL1 (Example 1) satellite CCD detector as an example. YL1 satellite is in sun-synchronous orbit, and the real-time data of the CCD detector from the ground station passed by YL1 satellite includes auxiliary data and subject data. The auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time and other recorded data, and monitoring data of attitude control system, communication system and power system. The subject data is remote sensing data collected by the on-board CCD detector. Satellite remote sensing data products are divided into five levels, and data correction is arranged in the transmission process, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric precision correction, and inversion of parameters such as air temperature, humidity, wind, cloud, precipitation, sunshine and surface background, for numerical prediction model system simulation. This embodiment describes a parallel processing method and device for multi-level data transmission and data computation, including system structure, parallel mechanism, system function and system module, etc. 4 aspects.

[0323] 4.5.1. System structure

[0324] The system structure includes:

[0325] ① Computer site composition: The site is composed of 6 levels of sites, from the 1st level site to the 6th level site. The first 5 levels of sites are mostly composed of 3 computers to form a network, and the last 1 level of site is composed of 5 GPU nodes.

[0326] ② Inter-site communication system: The sites are connected by long-range or short-range data communication systems. The first 5 levels of sites use the CCSDS AOS & XTCE general data communication system, rent satellite channels or apply for wireless channels, and the last 1 level of site uses short-range NVLink connection between the 5 nodes. NVLink is a data link layer bus and its communication protocol.

[0327] ③ Overall structure and function system: The first 5 levels of sites and their long-range communication form a multi-level site long-range data communication system and a supercomputer cluster of independent distributed computing resources; the last 1 level of site and its short-range communication form a multi-node short-range data communication system and a supercomputer cluster of distributed computing resources.

[0328] ④ Tree branch type static structure: The first 5 levels of sites form a multi-level tree structure, and the 5th and 6th levels of sites form a 5-branch type.

[0329] ⑤ Site data processing module: The site system module is divided into data sending, receiving, computing and other modules.

[0330] ⑥ Parallel processing dynamic mode: Data is transmitted from the 1st level site to the 6th level site in parallel, and the data processing tasks of the site are dispatched in staggered computing.

[0331] 4.5.2. Parallel mechanism

[0332] The content of this place is the same as that of "4.1.2. Parallel mechanism".

[0333] 4.5.3. System function

[0334] The system function includes:

[0335] ① Intra-site business classification function: Each level of site undertakes main and auxiliary data sending, receiving, computing and saving business.

[0336] ② Parallel processing dynamic function: Data flows from the 1st level site to the 6th level site, and each level of site relies on the multi-level tree structure to complete the staggered data transmission and data calculation of the site, and then completes the multi-level site data transmission, and the computing power completes the 0-5 level data correction task calculation, and the 5-6 level data inversion task calculation.

[0337] ③Communication mode configuration function: The inter-site short-range and long-range communication mode determines the application mode under the communication technology architecture, which is related to the operation, function and benefit of each part of the system; part of the inter-site uses long-range communication, part uses short-range communication, which shows the comprehensive mode function of parallel processing of data transmission and data calculation of multi-level site independent distributed computer resources.

[0338] ④Overall structure mode function: comprehensive mode, concurrent execution of part of the long-range data transmission and part of the short-range data dispatching at each level of site, concurrent calculation of task allocation, super computing power processing of big data by superimposed distributed computing resources.

[0339] 4.5.4. System module

[0340] The system module, i.e. the system module of each level of site, includes system preset, transmission module and calculation module.

[0341] 4.5.4.1. System preset

[0342] The content here is the same as that of "4.1.4.1. System preset".

[0343] 4.5.4.2. Transmission module

[0344] 1) The user end uses CCSDS AOS & XTCE data communication system and selects to use intermediate version transmission module. The main and auxiliary data are exchanged through hardware interface according to the specification to complete data sending and receiving.

[0345] The content of "sending submodule" and "receiving submodule" here is the same as that of "sending submodule" and "receiving submodule" in "4.1.4.2. Transmission module".

[0346] 2) Short-range uses NVLink special communication protocol connection and selects to use high-level transmission module. Under the support of local system and bus, data transmission between internal and external is transparent and smooth.

[0347] ① Data sending starting mechanism: when it is judged that there is data to be sent in the data storage place, the site executes the data sending transaction of this level of site.

[0348] ② Reading of received and sent data: the receiving and sending software takes the minimum data processing block as the processing unit, reads the main data of the "sending" storage place of the data calculation completed by this level of site according to the format, and reads the auxiliary data of the corresponding "sending" storage place if the auxiliary data needs to be sent.

[0349] ③ Writing of received and sent data: write auxiliary data and main data into the corresponding storage place of the lower level of site "receiving" according to the format.

[0350] ④Continuously forwarding subsequent data: Continuously forwarding the subsequent auxiliary data and main data of the scene data to the next level site with the minimum data processing block as the processing unit.

[0351] ⑤Cleaning up the sent data: Timely deleting the auxiliary data and main data stored in the sent, calculated and saved storage, releasing the computer resources.

[0352] 4.5.4.3. Computing module

[0353] The content of this place is the same as that of "4.1.4.3. Computing module".

[0354] 4.6. Example six: Mode 3 example 2. This embodiment takes the non-real-time data of the WFV detector of YL2 (Example 2) satellite as an example. YL2 satellite is in sun-synchronous orbit, and the non-real-time data of the WFV detector of YL2 satellite when passing through the ground station includes auxiliary data and main data. The auxiliary data is telemetry data, which includes satellite orbit parameters, satellite positioning, on-board time, and other recorded data, as well as monitoring data of attitude control system, communication system, and power system. The main data is the remote sensing data collected by the WFV detector on board the satellite. The satellite remote sensing data products are divided into five levels, and data correction is arranged in the transmission process, including system positioning, system projection, radiation calibration and correction, conversion to apparent reflectivity, and geometric fine correction, as well as the inversion of parameters such as air temperature, humidity, wind, cloud, precipitation, sunshine, and surface background, for the simulation of numerical prediction mode system. This paper describes the multi-level data transmission and parallel processing method and device of data transmission and data calculation of remote and short-range computer network transmission data, including system structure, parallel mechanism, system function and system module, etc.

[0355] 4.6.1. System structure

[0356] The system structure includes:

[0357] ① Computer site composition: The site is composed of 6 levels of sites from the 1st level site to the 6th level site, and each level of site is composed of a network of 3 computers.

[0358] ② Communication system between sites: The sites are connected by remote or short-range IEEE 802 standard computer communication network. The first 5 levels of sites are connected by wide area network, which is composed of two end networks and intermediate subnets, and the communication lines such as leased optical fiber or DDN special line are used; the last 5 sites are connected by local area network communication, and the communication lines such as optical fiber, twisted pair wire, and coaxial cable are used.

[0359] ③ Overall structure function system: The first 5 levels of sites and their remote communication constitute a multi-level site remote data communication system and a supercomputer cluster of independent distributed computing resources; the last 5 sites and their short-range communication constitute a multi-site short-range data communication system and a supercomputer cluster of independent distributed computing resources.

[0360] ④Tree static structure: the first to fifth level sites form a multi-level branch tree structure, and the fifth and sixth level sites form a five-branch type.

[0361] ⑤Site data processing module: the site system module is divided into data reading, writing, calculation and other modules.

[0362] ⑥Parallel processing dynamic mode: data is transmitted from the first level site to the sixth level site in parallel, and the data processing tasks of the site are dispatched in staggered calculation.

[0363] 4.6.2. Parallel mechanism

[0364] The parallel mechanism includes:

[0365] The content of this place (1) is the same as that of "4.1.2. Parallel mechanism" (1).

[0366] The content of this place (2) is the same as that of "4.2.2. Parallel mechanism" (2).

[0367] The content of this place (3) is the same as that of "4.1.2. Parallel mechanism" (3).

[0368] 4.6.3. System function

[0369] The system function includes:

[0370] ①Intra-site business classification function: each level site undertakes main and auxiliary data reading, writing, calculation and saving business.

[0371] ②Parallel processing dynamic function: data flows from the first level site to the sixth level site, and each level site relies on the multi-level tree branch structure to complete the staggered data transmission and data calculation of the site, and the multi-level site data transmission is completed in succession. The calculation power completes the 0-5 level data correction task calculation.

[0372] ③Communication mode construction function: the long-range and short-range communication mode between sites determines the application mode under the communication technology architecture, which relates to the operation, role and benefit of each part of the system; some sites use long-range communication, some use short-range communication, and the comprehensive mode function of parallel processing of multi-level site independent distributed computer resources is shown. Data transmission and data calculation.

[0373] ④Overall structure mode function: comprehensive mode, each level site concurrently executes part of the remote data transmission and part of the short-range data dispatch, and concurrently calculates the task dispatch, to supercomputing power processing big data by superimposed distributed computing resources.

[0374] 4.6.4. System module

[0375] The system module, i.e. the system module of each level of station, includes system preset, transmission module and calculation module.

[0376] 4.6.4.1. System preset

[0377] The system preset includes:

[0378] The contents of ①②③ here are the same as those of "4.1.4.1. System preset" ①②③.

[0379] ④ Determine the data exchange mode: data exchange needs to determine the data exchange mode between data reading, writing and calculation, and select a file as the storage place of exchanged data.

[0380] The contents of ⑤⑥ here are the same as those of "4.1.4.1. System preset" ⑤⑥.

[0381] 4.6.4.2. Transmission module

[0382] Select to use the advanced transmission module. The advanced platform can build transparent access of different addresses under network environment, and directly read and write data as if it is local under the advanced platform environment.

[0383] The contents of ①②③④⑤ here are the same as those of "4.5.4.2. Transmission module" ①②③④⑤.

[0384] 4.6.4.3. Calculation module

[0385] The contents here are the same as those of "4.1.4.3. Calculation module".

Claims

1. A method for concurrent interaction of multi-stage data transmission and data computation, characterized in that, Comprise: Minimum data processing block unit, minimum data processing block setting, data transmission calculation interaction mechanism 3 parts; The minimum data processing block unit, that is, the design of data exchange and data sending or reading, receiving or writing, calculation data processing business of each level site [ / node], is processed in the form of minimum data processing block unit; The whole data is divided into blocks with minimum data processing block as processing unit, auxiliary data is transmitted synchronously or immediately, providing data concurrent transmission calculation opportunity, relying on the parallel processing of data transmission and data calculation, obtaining high efficiency of data transmission and data calculation; The minimum data processing block setting, that is, the minimum data processing block includes auxiliary data and main data, or only includes main data; Block according to data structure characteristics, such as the corresponding relationship between auxiliary data and main data; ① If there is no corresponding relationship between auxiliary data and main data, the minimum data processing block is only a group of main data; ② If there is a corresponding relationship between auxiliary data and main data, auxiliary data does not need to be transmitted, and the minimum data processing block is a group of main data corresponding to auxiliary data; ③ If auxiliary data needs to be transmitted, the minimum data processing block is a group of auxiliary data and a group of main data corresponding to them; The data volume of minimum data processing block is relatively small, which is more conducive to reducing the time delay and improving the timeliness of overall data transmission calculation; The data transmission calculation interaction mechanism, that is, the minimum data processing block unit is used to form the parallel processing linkage mechanism of data transmission and data calculation of each level site [ / node]; ① Data block transmission: each scene [ / frame / track] data is transmitted block by block, and the transmission task in site [ / node] is completed block by block; In N site [ / node] transmission, each block of data is transmitted by site [ / node]; ② Data block calculation: each scene [ / frame / track] data calculation task is assigned to each site [ / node] according to the plan, and the assigned calculation task is processed by block, and the data calculation task is completed by site [ / node] together; ③ Interleaved transmission calculation: each scene [ / frame / track] data is transmitted block by block, each site [ / node] calculates data by block, and the transmission calculation task of main data and auxiliary data, or only main data, is completed by interleaved transmission calculation of each site [ / node].

2. A method for concurrent interaction of multi-stage data transmission and data computation, characterized in that, Comprise: Minimum data processing block unit, minimum data processing block setting, data transmission calculation interaction mechanism 3 parts; The minimum data processing block unit, that is, the design of data exchange and data sending or reading, receiving or writing, calculation data processing business of each level site [ / node], is processed in the form of minimum data processing block unit; The whole data is divided into blocks with minimum data processing block as processing unit, auxiliary data is transmitted synchronously or immediately, providing data concurrent transmission calculation opportunity, relying on the parallel processing of data transmission and data calculation, obtaining high efficiency of data transmission and data calculation; The minimum data processing block setting is that the minimum data processing block of the large model is only the main data, and the data structure feature is a semantic unit; ① The minimum semantic unit (Token) is determined as the data processing unit to determine the minimum data processing block; ② The minimum data processing block can be the minimum semantic unit of text, image, audio and video for the whole data blocking; ③ The minimum semantic unit and the number thereof of the present example are determined from the actually collected data; the data amount of the minimum data processing block is relatively small, which is more conducive to reducing the time delay and improving the timeliness of the whole data transmission calculation; The data transmission and calculation interaction mechanism is that the whole data is staggered and continuously transmitted and calculated in units of minimum data processing blocks to form a parallel processing linkage mechanism of data transmission and data calculation of various levels of sites / nodes; ① Data block transmission: the whole data is transmitted block by block, and the transmission task in the site / node is completed block by block; In N-site / node transmission, each block of data is transmitted from site to site; ② Data block calculation: the whole data calculation task is dispatched to each site / node according to the planning, and the calculation task is allocated according to the block processing, and the data calculation task is completed by each site / node together; ③ Staggered transmission and calculation: the whole data is transmitted block by block from site to site, and the data is calculated block by block at each site, and the transmission and calculation task of the main data is completed in the staggered transmission and calculation mode of each site.

3. The method of claim 1 or 2, wherein, Also includes: Parallel transmission calculation mathematical model, namely: data is divided into M blocks, N sites[ / nodes] parallel transmission calculation, according to the space-time relationship between sites[ / nodes] and blocks, the whole data parallel transmission calculation time is obtained In the formula: N, M are integers greater than or equal to 2, 1≤i≤N, 1≤j≤M; T Ci is the time spent for inter-site [node] channel for block data; T Di，j is the time spent for block data transceiving; T Oi，j Calculate the timeout for block data, T Oi，j = T SCi，j -T Di，j , T SCi，j = T C1 i，j + T C2i，j , T SCi，j Calculate the cache time for block data, T C1i，j Calculate the time for block data, T C2 i，j Calculate the cache time for block data, and it is stipulated that T SCi，j -T D i，j ≤ 0, T Oi，j = 0; the right term 1 of formula (1) is the total time consumption of the channel between the stations [ / nodes] at all levels, the right term 2 is the total time for the block data transmission and reception of the stations [ / nodes] at all levels, the right term 3 is the total timeout for the block data calculation of the stations [ / nodes] at all levels, and the right term 4 is the total delay of the block data except the first block of 1 station [ / node] on average; after combining the same terms, we get Each block of data has the same size, if the same computer [processor / computing core] is used to embed each block of data at the same level of site [node], then there are In the simplified case, T NM The total time can be summarized as the sum of the total time of the channel between the nodes of each level of data, the total time of the transmission and reception of each level of data, the total time of the calculation of each level of data, and the total delay of the transmission and reception of each block of data except the first block at the node.

4. A method for parallel processing of multi-stage data transmission and data computation, characterized in that, Including: System structure, parallel mechanism, system function, system module 4 parts; The system structure includes: composed of multiple levels of computer / processor / computing core sites / nodes; the communication between sites / nodes adopts remote and short-range computer communication network, or remote and short-range non-computer communication network; the site / node and the remote or short-range communication between them constitute a multi-level site / node remote or short-range data communication system and a distributed computer / processor / computing core cluster, or even a supercomputer / processor / computing core cluster; the multi-level site / node and the data flow direction are of a multi-level tree branch type static structure; the site / node data processing is divided into data transmission and data calculation modules; the data is staggered and transmitted and calculated in a parallel processing dynamic mode; Adopting the parallel mechanism includes: data blocking provides an opportunity for large data concurrent transmission and calculation; the minimum data processing block is divided into three forms: ① a group of main data, ② a group of auxiliary data corresponding to the main data, and ③ a group of auxiliary data and a group of main data corresponding thereto; the data is transmitted and calculated block by block from site to site, forming a staggered transmission and calculation mechanism to complete the whole data transmission and calculation task; The parallel mechanism of the large model includes: data block provides an opportunity for large data concurrent transmission calculation; the minimum data processing block of the large model is the minimum semantic unit, which can be the minimum semantic unit of text, image, audio and video data; the overall data is blocked by the minimum semantic unit, and the number of each minimum semantic unit is determined; the data block is transmitted and calculated station by station ( / node), forming an interleaved transmission computing mechanism, and completing the whole data transmission and calculation task; The system functions include: each level of station [ / node] undertakes data transmission and data calculation business; the data is transmitted and calculated by the interleaved transmission and calculation method, and the total amount of data transmission and calculation task of multiple stations [ / nodes] is completed; the communication mode between stations [ / nodes] determines the operation, effect and benefit of the system, and shows different structure mode functions; the station [ / node] and the remote or short-range communication between stations [ / nodes] determine the combination of remote data communication, short-range data communication and distributed supercomputing of multiple stations [ / nodes]; The system module is established, that is, the system module of each level of station [ / node], including system preset, transmission module and calculation module; The system preset includes: the program process form, the communication sending and receiving mode, the signal sending and receiving matching and the data exchange mode are determined in advance, the calculation task algorithm is prepared in advance, and the file name specification naming is implemented; The transmission module is divided into low-level version, middle-level version and high-level version, and the version is selected and used as needed; the low-level version transmission module includes a sending submodule and a receiving submodule; the sending submodule includes: formulating a data sending start mechanism, reading and sending data according to the minimum data processing block, packing and framing data transmission according to the minimum data processing block, continuously sending subsequent data of the data task, and cleaning the sent main and auxiliary data or only the main data; the receiving submodule includes: completely receiving each frame of data, unpacking according to the frame header and the package header, saving the received main and auxiliary data or only the main data according to the block, and continuously receiving subsequent data of the data task; the middle-level version transmission module includes a sending submodule and a receiving submodule; the sending submodule includes: formulating a data sending start mechanism, reading and sending data according to the minimum data processing block, providing exchange data to the interface according to the frame planning, continuously sending subsequent data of the data task, and cleaning the sent main and auxiliary data or only the main data; the receiving submodule includes: reading data according to the interface specification, saving the received main and auxiliary data or only the main data according to the block, and continuously receiving subsequent data of the data task; the high-level version transmission module includes: formulating a data sending start mechanism, reading and sending data from the current station [ / node] according to the block, writing the received data into the "receiving" exchange data storage of the lower station [ / node], continuously forwarding subsequent data of the data task, and cleaning the sent main and auxiliary data or only the main data; The calculation module includes: adjusting the auxiliary data needed, formulating a data calculation start mechanism, calling the collected parameter data and "receiving" auxiliary data, calculating the main data not processed according to the calculation task processing algorithm according to the block, continuously calculating the subsequent data of the data task, adjusting the sent auxiliary data, and saving the sent, received and calculated main and auxiliary data or only the main data.

5. A parallel processing device for multi-stage data transmission and data computation, characterized by, It includes: System structure, parallel mechanism, system function, system module 4 parts; The device system structure includes: composed of multiple levels of computer [processor / computing core] sites [nodes]; inter-site [node] communication uses long-range, short-range computer communication networks, or long-range, short-range non-computer communication networks; sites [nodes] and their long-range or short-range communication constitute a multi-level site [node] long-range or short-range data communication system and distributed computer [processor / computing core] cluster, even supercomputer [processor / computing core] cluster; The multi-level site [node] and its data flow are of a multi-level tree branch type static structure; Site [node] data processing is divided into data transmission and data calculation modules; Data is transmitted and calculated in parallel processing dynamic mode by staggered transmission; The parallel mechanism of the device includes: data block provides an opportunity for large data concurrent transmission and calculation; The smallest data processing block is divided into three forms: ① a group of main data, ② a group of auxiliary data corresponding to the main data, and ③ a group of auxiliary data and a group of main data corresponding to them; Data blocks are transmitted and calculated at each site [node], forming an interlaced transmission and calculation mechanism to complete the entire data transmission and calculation task; The parallel mechanism of the large model includes: data block provides an opportunity for large data concurrent transmission and calculation; The smallest data processing block of the large model is the smallest semantic unit, which can be the smallest semantic unit of text, image, audio and video data; The entire data is divided into blocks based on the smallest semantic unit, and the number of each smallest semantic unit is determined; Data blocks are transmitted and calculated at each site [node], forming an interlaced transmission and calculation mechanism to complete the entire data transmission and calculation task; The system function of the device includes: each level of site [node] undertakes data transmission and data calculation business; The multi-level site [node] data transmission and total amount task calculation are completed by the staggered transmission and calculation of data at each level; The inter-site [node] communication mode determines the operation, effect and benefit of the system, and shows different structure mode functions; The long-range or short-range communication between sites [nodes] determines the combination of multi-level site [node] long-range data communication, short-range data communication and distributed supercomputing; The system module of the device is the system module of each level of site [node], including system preset, transmission module and calculation module; The system preset includes: pre-determined program process form, communication transceiver mode, signal transceiver matching and data exchange mode, pre-prepared calculation task algorithm, and implemented file name specification naming; The transmission module is divided into low-level version, intermediate version and high-level version, and the version is selected according to the requirement; the low-level version transmission module includes a sending submodule and a receiving submodule; the sending submodule includes: formulating a data sending starting mechanism, reading sending data according to the minimum data processing block, packing and framing the data for transmission according to the minimum data processing block, continuously sending subsequent data of the data task, and cleaning the sent main and auxiliary data or only the main data; the receiving submodule includes: completely receiving each frame of data, unpacking according to the frame header and the package header, saving the received main and auxiliary data or only the main data according to the block according to the mark, and continuously receiving subsequent data of the data task; the intermediate version transmission module includes a sending submodule and a receiving submodule; the sending submodule includes: formulating a data sending starting mechanism, reading sending data according to the minimum data processing block, providing exchange data to the interface according to the frame planning, continuously sending subsequent data of the data task, and cleaning the sent main and auxiliary data or only the main data; the receiving submodule includes: reading data according to the interface specification, saving the received main and auxiliary data or only the main data according to the block according to the mark, and continuously receiving subsequent data of the data task; the high-level version transmission module includes: formulating a data sending starting mechanism, reading sending data from the current site [ / node] according to the block, writing the received exchange data into the lower site [ / node], continuously forwarding subsequent data of the data task, and cleaning the sent main and auxiliary data or only the main data. The calculation module includes: adjusting the auxiliary data needed, formulating a data calculation starting mechanism, calling the collected parameter data and the received auxiliary data, calculating the main data which has not been processed according to the calculation task processing algorithm according to the block, continuously calculating subsequent data of the data task, adjusting the sent auxiliary data, and saving the sent, received and calculated main and auxiliary data or only the main data.

6. The parallel processing method according to claim 4 and the parallel processing apparatus according to claim 5, characterized by, Further comprising: Remote communication is adopted between sites [ / nodes], and a multi-level site [ / node] remote data communication system and a supercomputer [ / processor / computing core] cluster of distributed computing resources are formed with each site [ / node].

7. The parallel processing method according to claim 4 and the parallel processing apparatus according to claim 5, characterized by, Further comprising: Short-range communication is adopted between sites [ / nodes], and a multi-level site [ / node] short-range data communication system and a supercomputer [ / processor / computing core] cluster of distributed computing resources are formed with each site [ / node].

8. The parallel processing method according to claim 4 and the parallel processing apparatus according to claim 5, characterized by, Further comprising: Both remote and short-range communication are adopted between sites [ / nodes], and a multi-level site [ / node] local remote and local short-range data communication system and a supercomputer [ / processor / computing core] cluster of distributed computing resources are formed with each site [ / node].

Images