An SDN-based on-orbit big data processing simulation platform for low-orbit constellations

Abstract

The present invention discloses a SDN-based low-orbit constellation on-orbit big data processing simulation platform, including a constellation simulation layer, a network simulation layer and an application simulation layer. The constellation simulation layer provides a simulation of a real physical environment for the simulation platform, including satellite orbits, satellite visibility and inter-satellite link parameters. The simulation result of the constellation simulation layer provides a basis for network topology construction and network parameter configuration for the network simulation layer; the network simulation layer provides a dynamic network environment for the simulation platform, including changes in network topology, changes in bandwidth and delay between satellite nodes, and the network simulation layer provides an API to the application simulation layer through network control; the application simulation layer uses the API provided by the network simulation layer to implement distributed big data software and testing tools. The present invention deploys real data streams and real applications into the simulation platform, so that the traditional network simulation using software models for parameter calculation is transformed into the simulation of actual applications and data streams.

Description

Technical Field

[0001] The present application relates to the technical field of low-orbit satellite simulation, and in particular to an SDN-based low-orbit constellation on-orbit big data processing simulation platform. Background Art

[0002] In addition to carrying traditional satellite services such as broadband communications, remote sensing, and meteorological services, low-orbit constellations have also become an important part of the 6G ubiquitous intelligence vision. In order to cope with diverse business needs, transparent forwarding alone seems insufficient. Instead, storage and computing resources must be deployed in low-orbit constellations to carry out on-orbit processing. In recent years, the latest research results of aerospace microprocessors have enabled onboard computers based on commercial architectures to run complex general computing tasks rather than just signal processing. However, due to the limitations of the payload capacity of small low-orbit satellites and the space radiation environment, the scale of tasks that a single satellite can independently complete is extremely limited, which makes multi-satellite joint distributed computing and distributed storage an important development direction in low-orbit constellation network research. Distributed processing systems have been widely used in ground data centers. They use multiple devices to jointly store massive data and complete a large number of computing tasks. Commercial distributed systems such as Hadoop, Spark, and Flink have made great progress. The efficient operation of these distributed big data applications depends on a large amount of data exchange between nodes, which requires stable links and sufficient bandwidth. However, due to the periodic motion of satellites, the network state of low-orbit constellations is dynamic, which will affect the performance of distributed big data applications that are sensitive to network performance. Currently, there are no standardized simulation tools that can address the problem of integrating dynamic satellite networks with distributed big data applications.

[0003] At present, the functions of satellite network simulation software are concentrated on satellite orbit simulation and network simulation, and the two are not considered together. In addition, there is a lack of corresponding research on the applicability of distributed big data applications in satellite networks. Taking the popular MapReduce architecture in distributed computing as an example, its operating mechanism is divided into three processes: Map, Shuffle, and Reduce. The middle Shuffle process is to pass the intermediate data processed by the Mapper to the Reducer. This process occupies 50% of the time in the general distributed computing process and has a non-negligible impact on the completion time of the calculation. For satellite networks with dynamic link characteristics, the impact of the Shuffle process on the completion time of the calculation is not yet clear. At the same time, the cost of deploying big data applications in low-orbit constellations for actual testing is high. Summary of the invention

[0004] In view of the above problems, the present invention provides an SDN-based low-orbit constellation on-orbit big data processing simulation platform, which aims at the distributed on-orbit processing problems of low-orbit constellations and is used to verify the performance of big data applications in low-orbit constellation networks.

[0005] The present invention provides an SDN-based low-orbit constellation on-orbit big data processing simulation platform, the simulation platform includes a constellation simulation layer, a network simulation layer and an application simulation layer, wherein:

[0006] The constellation simulation layer provides the simulation platform with a simulation of a real physical environment, including satellite orbits, satellite visibility and inter-satellite link parameters. The simulation results of the constellation simulation layer provide a basis for network topology construction and network parameter configuration for the network simulation layer.

[0007] The network simulation layer provides a dynamic network environment for the simulation platform, including changes in network topology, changes in bandwidth and latency between satellite nodes, and provides an API to the application simulation layer through network control;

[0008] The application simulation layer uses the API provided by the network simulation layer to implement distributed big data software and testing tools, and the application simulation layer is also used to count application indicators.

[0009] A further technical solution of the present invention is: the constellation simulation layer is implemented by jointly simulating in the form of a main process and several slave processes, and the implementation process includes a scenario configuration process and a parameter calculation process, wherein in the scenario configuration process, STKEngine is used in the main process to configure the orbit and satellite parameters of the low-orbit constellation; in the parameter calculation process, the visibility and link parameters between each satellite in each time slot of the simulation are obtained from STK, and the parameter calculation process is disassembled into several parts and processed by the slave process.

[0010] A further technical solution of the present invention is: the main process divides the satellites waiting to be calculated into several groups according to the number of cores of several servers equipped with STK, and distributes the scenario files and the numbers of the groups to be calculated to each slave process, and the calculation results of the main process and the slave process are asynchronously transmitted to the network simulation layer through the Socket interface.

[0011] A further technical solution of the present invention is: the network simulation layer adopts a synchronous data receiving strategy to receive the parallel computing data of the constellation simulation layer, and the network simulation layer adopts a master process and slave process structure, wherein the master process of the network simulation layer and the master process of the constellation simulation layer obtain the orbit and satellite parameters of the low-orbit constellation through the socket interface, and start a corresponding number of slave processes of the network simulation layer to establish connections with the slave processes of the constellation simulation layer respectively.

[0012] A further technical solution of the present invention is: the network simulation layer adopts a time slot simulation strategy to divide the entire simulation time into several time slots, receives data parameters of the next time slot from the constellation simulation layer before the end of each time slot, and uses the acquired data parameters to calculate the network topology and network parameter configuration to generate a satellite network based on SDN.

[0013] A further technical solution of the present invention is: in the process of generating an SDN-based satellite network, an open source SDN network simulation tool mininet is used to build a network including a host, an SDN switch and an SDN controller and configure network parameters, and each application is packaged into a container image to access the SDN network. The mininet uses OpenvSwitch as a virtual satellite node, establishes a network connection according to visibility and link parameters, and connects a docker image to each OpenvSwitch.

[0014] A further technical solution of the present invention is: the network simulation layer provides an API to the application simulation layer through network control, and the specific implementation method includes:

[0015] Use the SDN controller written in Python to monitor the data of the entire network and control the scheduling of network flows, deploy RYU to the network, and use the northbound API in the Python script to call the functions of RYU;

[0016] The SDN controller and the switch complete information exchange through the Openflow protocol, specifically including: the switch forwards the data packets that cannot be forwarded to the SDN controller through packet_in, and the SDN controller obtains the forwarding rules through calculation and sends the flow table and group table to OpenvSwitch to realize the control of the OpenvSwitch forwarding strategy;

[0017] OpenvSwitch forwards data based on flow tables and group tables.

[0018] A further technical solution of the present invention is: the RYU is also used to realize the identification and scheduling of application traffic, classify the application traffic by identifying the port number of the application through the control information of Openflow, and allocate the maximum bandwidth of each type of traffic as needed.

[0019] A further technical solution of the present invention is: the application simulation layer uses a distributed big data processing platform based on Hadoop to implement distributed computing and distributed storage, and the Hadoop is made into a docker image and connected to the network simulation layer.

[0020] A further technical solution of the present invention is: the application simulation layer deploys multiple erasure code storage strategies in HDFS, tests the performance of data download and repair in the satellite network by controlling the distribution of data blocks, and uses the HiBench test suite to test the shuffle traffic and task execution time of the Hadoop distributed computing component MapReduce.

[0021] The present invention provides a SDN-based low-orbit constellation on-orbit big data processing simulation platform. In order to support the huge number of satellites in the low-orbit constellation, the simulation capability of STK is expanded. Specifically, a framework for parallel computing that can be used for multi-core servers is designed using the programming interface of STKEngine, which realizes the simulation of thousands of satellite orbits, visibility and link parameters at the same time; the real-time constellation information provided by STK is mapped to a virtual network based on Mininet (SDN deployment tool), Mininet is transformed to enable it to access any application container, and a flow table distribution mechanism suitable for the satellite topology characteristics is configured; real big data applications are deployed in the network using container technology, and in order to enable the simulation platform to reflect the real application performance, the distributed big data platform Hadoop is transformed and packaged as a docker image, and a control script is designed to realize the rapid deployment and configuration of computing / storage nodes in the network. In summary, the present invention develops a multi-dimensional joint comprehensive simulation platform for the distributed big data processing characteristics in the low-orbit constellation. The platform has three layers: constellation simulation layer, network simulation layer and application simulation layer. The data flow and control flow interaction of the three-layer simulation architecture are realized through Python control scripts and APIs. The beneficial effects of the present invention are: creatively deploying real data streams and real applications into the simulation platform, transforming the traditional network simulation using software models for parameter calculation into the simulation of actual applications and data streams; in order to facilitate flexible and efficient simulation, a parallel simulation script is designed based on the simulation engine of STK, which improves the parameter calculation capability of STK for large-scale constellations. At the same time, the relevant data is encapsulated as an interface and provided to the network simulation layer; in order to combine real big data applications into the simulation platform, the containerization technology is introduced to realize the rapid deployment of a large number of distributed processing nodes in the virtual network constructed by SDN, thereby improving the availability of the simulation platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the structure of an SDN-based low-orbit constellation on-orbit big data processing simulation platform in an embodiment of the present invention;

[0023] Figure 2 It is a schematic diagram of the distributed storage and distributed computing process in the application simulation layer in an embodiment of the present invention;

[0024] Figure 3is a schematic diagram of satellite network delay simulation in an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of file retrieval throughput of different network topology scales in an embodiment of the present invention;

[0026] Figure 5 It is a schematic diagram of computing task throughput of different network topology scales in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0029] The embodiment of the present invention provides the following embodiments for an SDN-based low-orbit constellation on-orbit big data processing simulation platform:

[0030] As an on-orbit processing simulation platform, the core goal is to evaluate the application performance in the low-orbit constellation network environment, which inevitably requires three parts of simulation capabilities: satellite constellation simulation, satellite network simulation, and satellite application simulation. Therefore, the embodiment of the present invention constructs an on-orbit processing simulation platform based on software defined network (SDN) technology. SDN divides the network into a control plane and a data plane, which provides excellent scalability for integrating other tools. The embodiment makes full use of this idea in the design of the simulation platform, that is, embedding distributed applications into the SDN environment. Not only that, the real constellation parameters are mapped to SDN with the help of the STK (STK is the abbreviation of Satellite Tool Kit, i.e., satellite tool kit) API, in order to achieve the joint simulation of the above three parts.

[0031] Specifically, the embodiment establishes a three-layer simulation platform architecture, and designs an application programming interface between each layer as the basis for upper-layer simulation. In the simulation platform, several existing simulation software are integrated and cleverly combined through secondary development. A series of open simulation scripts enable the simulation platform to automatically provide synchronous real-time simulation from low-orbit constellations to processing tasks.

[0032] like Figure 1 As shown in the figure, the architecture and functions of the simulation platform are described:

[0033] The simulation platform includes a constellation simulation layer, a network simulation layer and an application simulation layer, wherein:

[0034] The constellation simulation layer provides the simulation platform with a simulation of a real physical environment, including satellite orbits, satellite visibility and inter-satellite link parameters. The simulation results of the constellation simulation layer provide a basis for network topology construction and network parameter configuration for the network simulation layer.

[0035] Furthermore, the constellation simulation layer is implemented by jointly simulating with a master process and several slave processes, and the implementation process includes a scenario configuration process and a parameter calculation process. In the scenario configuration process, STKEngine is used in the master process to configure the orbit and satellite parameters of the low-orbit constellation; in the parameter calculation process, visibility and link parameters between each satellite in each time slot of the simulation are obtained from STK, and the parameter calculation process is broken down into several parts and processed by slave processes.

[0036] Furthermore, the master process divides the satellites waiting to be calculated into several groups according to the number of cores of several servers equipped with STK, and distributes the scenario files and the numbers of the groups to be calculated to each slave process. The calculation results of the master process and the slave process are asynchronously transmitted to the network simulation layer through the Socket interface.

[0037] In the specific implementation process, the constellation simulation layer provides a simulation of the real physical environment for the entire simulation platform, including satellite orbits, satellite visibility and inter-satellite link parameters. Its simulation results can provide the basis for network topology construction and network parameter configuration for the network simulation layer. In order to achieve this goal, the powerful satellite simulation capabilities of STK are used. The satellite toolkit is a commercial satellite simulation tool developed by AGI, which can provide simulation of satellite orbits, visibility, coverage, link parameters, etc. In order to facilitate developer integration, STK provides a variety of application interfaces that can be called through multiple languages ​​such as MATLAB, Python, and C++.

[0038] Faced with the huge number of satellites in the low-orbit constellation, manual configuration in the STK software is a time-consuming and laborious task, so a Python script and a database were written that use the STKEngine APIs for automated scenario configuration.

[0039] During the specific implementation process, the script reads the parameters in the database according to the user configuration, and then establishes the STK simulation scenario through the API. However, STKEngine itself can only execute a single process, which hinders the real-time simulation of thousands of satellite nodes in the low-orbit constellation. In order to make full use of the powerful computing power of multi-core processors and server clusters, the inventor designed a parallel computing script based on STKEngineAPI. Specifically, the inventor prepared several servers equipped with STK, and all of them ran Python-based parallel computing scripts.

[0040] In the simulation process design, the constellation simulation is divided into two processes, namely the scenario configuration process and the parameter calculation process, and a master process and several slave processes are used for joint simulation. In the scenario configuration process, STKEngine is used in the master process to configure the orbit number, orbit height, number of satellites, satellite receiver and transmitter parameters of the low-orbit constellation. These parameters come from the pre-loaded database and each satellite is numbered. This process takes about several minutes. In the parameter calculation process, it is necessary to obtain the visibility and link parameters between each satellite in each time slot of the simulation from STK. This process will take a lot of time and become the bottleneck of the entire simulation. This simulation process can be disassembled into several parts, which is conducive to the use of multi-process processing. The master process will divide the satellites waiting to be calculated into several groups according to the number of cores of each server, and distribute the scenario files and the numbers of the groups they need to calculate to each slave process. It is worth noting that the slave processes here are running on different servers in the cluster, and each server runs several slave processes according to the number of CPU cores. The calculation results of the master process and the slave process will be asynchronously transmitted to the satellite network simulation layer through the Socket interface. In practical applications, the startup order and calculation speed of each slave process are different, so a synchronization strategy is designed in the satellite network simulation layer to ensure the complete constellation information of each time slot.

[0041] The network simulation layer provides a dynamic network environment for the simulation platform, including changes in network topology, changes in bandwidth and latency between satellite nodes, and provides an API to the application simulation layer through network control;

[0042] Furthermore, the network simulation layer adopts a synchronous data receiving strategy to receive the parallel computing data of the constellation simulation layer, and the network simulation layer adopts a master process and slave process structure, wherein the master process of the network simulation layer and the master process of the constellation simulation layer obtain the orbit and satellite parameters of the low-orbit constellation through the socket interface, and start a corresponding number of slave processes of the network simulation layer to establish connections with the slave processes of the constellation simulation layer respectively.

[0043] Furthermore, the network simulation layer adopts a time slot simulation strategy to divide the entire simulation time into several time slots, receives data parameters of the next time slot from the constellation simulation layer before the end of each time slot, and uses the acquired data parameters to calculate the network topology and network parameter configuration to generate an SDN-based satellite network.

[0044] Furthermore, in the process of generating an SDN-based satellite network, the open source SDN network simulation tool mininet is used to build a network including hosts, SDN switches and SDN controllers and configure network parameters. Each application is packaged into a container image and connected to the SDN network. The mininet uses OpenvSwitch as a virtual satellite node, establishes a network connection according to visibility and link parameters, and connects a docker image to each OpenvSwitch.

[0045] Furthermore, the network simulation layer provides an API to the application simulation layer through network control. The specific implementation method includes: using an SDN controller written in Python to monitor the data of the entire network and control the scheduling of network flows, deploying RYU to the network, and using the northbound API to call the function of RYU in the Python script; the SDN controller and the switch complete information interaction through the Openflow protocol, specifically including: the switch forwards the data packets that cannot be forwarded to the SDN controller through packet_in, the SDN controller obtains the forwarding rules through calculation and sends the flow table and group table to Open vSwitch to realize the control of the OpenvSwitch forwarding policy; OpenvSwitch forwards data according to the flow table and group table.

[0046] Furthermore, the RYU is also used to implement the identification and scheduling of application traffic, classify application traffic by identifying the port number of the application through the control information of Openflow, and allocate the maximum bandwidth of each type of traffic as needed.

[0047] In the specific implementation process, the function of the satellite network simulation layer is to provide a dynamic network environment for the entire platform, including changes in network topology, changes in bandwidth and latency between satellite nodes, which are calculated based on the data provided by the constellation simulation layer. In addition, this layer also deploys a network control module that can load network traffic scheduling algorithms and provide APIs to the application layer.

[0048] In order to receive the parallel computing data of the constellation simulation layer, a synchronous data reception strategy is designed. Specifically, a multi-threaded communication method is designed, which is similar to the constellation simulation layer and also adopts the structure of master process and slave process. Among them, the master process of this layer and the master process of the previous layer obtain the constellation information and slave process information through the socket interface, and then start the corresponding number of slave processes to establish connections with the slave processes of the previous layer.

[0049] Since the calculation results generated by multi-threading are transmitted asynchronously, a time slot simulation strategy is designed in this layer, that is, the entire simulation time is divided into several time slots, and the data of the next time slot is received from the upper layer before the end of each time slot. The information obtained will be used to calculate the network topology and link parameters to generate a satellite network based on SDN. In order to conveniently implement the function, the open source SDN network simulation tool mininet is used, which can quickly build a network consisting of hosts, SDN switches, and SDN controllers and freely configure network parameters. More importantly, mininet itself is developed based on LinuxContainer technology, so various applications can be packaged into container images to access the SDN network. When building the network, mininet uses OpenvSwitch (OVS) as a virtual satellite node and establishes network connections according to visibility and link parameters, thus forming a satellite network with switching and forwarding functions. In order to give satellite nodes computing and communication capabilities, each OVS is connected to a host, and they are regarded as the same node because the link between them has sufficient bandwidth and low latency. Here, mininet is improved by replacing the host with a docker image.

[0050] In the SDN architecture, the controller will monitor the data of the entire network and control the scheduling of network flows. In order to achieve a unified development language, RYU is deployed in the network. It is an SDN controller written in Python, so the northbound API can be used in Python scripts to call RYU functions. The controller and the switch complete information exchange through the Openflow protocol. Specifically, the switch will forward the data packets that cannot be forwarded to the controller through packet_in. After that, the controller calculates the forwarding rules and sends the flow table and group table to OVS. This process realizes the control of the OVS forwarding strategy, and OVS is only responsible for forwarding data according to the flow table and group table. RYU can also realize accurate identification and scheduling of application traffic. The application port number is identified through the control information of Openflow to classify the application traffic, which can allocate the maximum bandwidth of each type of traffic as needed. This can achieve performance optimization of satellite big data applications at the network level.

[0051] The application simulation layer uses the API provided by the network simulation layer to implement distributed big data software and testing tools, and the application simulation layer is also used to count application indicators.

[0052] Furthermore, the application simulation layer uses a distributed big data processing platform based on Hadoop to implement distributed computing and distributed storage, and the Hadoop is made into a docker image and connected to the network simulation layer.

[0053] Furthermore, the application simulation layer deploys multiple erasure code storage strategies in HDFS, and tests the performance of data download and repair in the satellite network by controlling the distribution of data blocks. At the same time, the HiBench test suite is used to test the shuffle traffic and task execution time of the Hadoop distributed computing component MapReduce.

[0054] In the specific implementation process, the function of the satellite application simulation layer is to implement actual distributed big data software and testing tools in the network, and to collect statistics including application traffic, execution time and other indicators. The introduction of container technology makes the distributed applications deployed in this layer flexible and replaceable. The satellite application simulation layer uses a Hadoop-based configuration, and other distributed applications are similar. Hadoop, a popular distributed big data processing platform, is made into a docker image and connected to the simulation network. Hadoop has two important functions: distributed storage and distributed computing. A variety of erasure code storage strategies are deployed in the Hadoop Distributed File System (HDFS), and the performance of data download and repair in the satellite network is tested by controlling the distribution of data blocks. In addition, the HiBench test suite is used to test the shuffle traffic and task execution time of the Hadoop distributed computing component MapReduce. Figure 2 The system structure of distributed computing and distributed storage is demonstrated.

[0055] In the specific implementation process, a part of the data is stored in each satellite node. During the simulation process, the controller will submit several jobs to MapReduce as a user. The NameNode working on the master node will be divided into several tasks through the JobTracker and assigned to each DataNode working on the slave node. The data flow generated in this process will be managed by the satellite network simulation layer.

[0056] In order to verify the function of the simulation platform, a series of distributed storage and computing simulation experiments were carried out on the platform. We installed Ubuntu 20.04 LTS on a Dell R940 server to run the network simulator and the big data application platform, and two desktop computers were used to run STK. In the simulation process, the Starlink Phase I planned constellation was used as the satellite network. The constellation and simulation environment parameters are given in Table 1.

[0057] Table 1 Simulation parameters

[0058]

[0059]

[0060] During the simulation, the STK parallel simulation framework was used to build and run the Starlink constellation on two desktop computers. Then, the satellite link parameters were obtained according to the time slot, and a time-varying virtual satellite network was built using mininet. Finally, the Hadoop platform was deployed in this network to simulate the performance of distributed storage and distributed computing tasks.

[0061] A virtual network based on SDN was built, including 1584 satellite nodes. In the simulation, the simulator refreshes the network parameters every 1 minute. It is unrealistic to call the nodes of the entire constellation for actual processing tasks. The nodes involved in the work distribution are often a number of nodes in a local range. Local networks of different sizes ranging from 4*4 to 8*8, that is, 16 to 64 nodes, were delineated from the entire Stralink constellation. The satellite at the head is configured as the master node, and the rest are slave nodes. Figure 3 The average delay from the master to the slave node during the 120-minute simulation is shown. Since the master node is responsible for allocating processing tasks in Hadoop, the delay simulation can reflect the task response speed of distributed applications. It can be seen that the average delay changes periodically with the movement of the satellite, which is consistent with the periodic change of the distance between satellites. At the same time, as the network scale expands, the average delay also gradually increases. The impact of this phenomenon on the processing capacity of distributed applications is explained in the following simulation.

[0062] The host in Mininet is configured as a Hadoop container, and the performance of HDFS and MapReduce are simulated respectively. The simulation parameters are shown in Table 1.

[0063] 1. Data download of HDFS: 100 files are stored in the local network, each with a size of 384MB. Three different RS codes are configured for these files to divide them into several data blocks, and different data blocks of each file are randomly stored on different nodes. For example, when RS (3,2) is used, a file will be encoded into 5 blocks, including 3 data blocks and 2 check blocks, each of which is 128MB. When retrieving a file, the request node will download 3 blocks at the same time to rebuild the file. In the process of each file retrieval, the master node is used to initiate the request, and 1 file is randomly requested each time. Since the storage and request of files are random, the chance of each node being read is uniform during repeated retrieval.

[0064] The task throughput of two storage modes, RS(6,3) and triple replication, under different networks is compared. Other erasure codes have similar conclusions. For comparability, the bandwidth of the ground data center is set to be the same as that of the low-orbit constellation, but it has a tree topology and extremely low link latency. Figure 4It can be seen that in ground data centers, networks of different sizes have little effect on throughput, and RS(6,3) has a slight performance improvement of up to 2.3% compared to triple replication (see the dotted line in the figure). In low-orbit constellations, the increase in network size will reduce task throughput due to increased latency. The solid line shows that in low-orbit constellations, the performance advantage of erasure coding over triple replication is more obvious, up to 22.9%, and the performance improvement rate gradually increases with the expansion of the network size. Since multiple data blocks can be read in parallel when retrieving files, the erasure code storage strategy provides a higher throughput than triple replication.

[0065] 2. MapReduce task completion: Take the sort task as an example to study the performance of distributed computing, because it generates a large intermediate value data stream, which can better reflect the impact of network performance on task throughput. HiBench is used to configure a 2GB workload for each task, and only one task at a time randomly selects one-quarter, one-half, or all of all nodes as working nodes.

[0066] from Figure 5 It can be seen that when the network scale is small, adding working nodes can increase the task throughput, but when the network scale is large, adding nodes will lead to a decrease in task throughput. The large network scale brings high latency and reduces the efficiency of data transmission between nodes. At the same time, it is observed that even in low-latency ground data centers, the expansion of the network scale will still lead to a decrease in task throughput. This is because more data flows are generated and it is easier to cause local congestion. It can be seen that the reasonable scheduling of working nodes and the data flows between them can alleviate the computing performance problems brought by low-orbit constellations.

[0067] There are three types of workloads: 100M, 1G, and 2G. They are completed by several worker nodes. HiBench runs only one task each time, and it computes in parallel on 15 randomly selected worker nodes. Obviously, in a 4*4 network, there are only 15 worker nodes besides the master, and they all participate in the calculation every time. Due to the randomness of the selection, each node has an even chance to participate in the calculation.

[0068] Similar to the HDFS experiment, satellite nodes at different locations in the network are configured as workers. It should be emphasized that in the Sort type task, each worker acts as a Mapper and a Reducer. As can be seen from the figure, the change pattern of the three curves is similar to that of the HDFS experiment. The fluctuation of satellite link latency will significantly affect the performance of distributed computing.

[0069] A SDN-based low-orbit constellation on-orbit big data processing simulation platform is provided in combination with the above embodiments. In order to support the huge number of satellites in the low-orbit constellation, the simulation capability of STK is expanded. Specifically, a parallel computing framework that can be used for multi-core servers is designed using the programming interface of STKEngine, which realizes the simulation of thousands of satellite orbits, visibility and link parameters at the same time; the real-time constellation information provided by STK is mapped to a virtual network based on Mininet (SDN deployment tool), Mininet is transformed to enable it to access any application container, and a flow table delivery mechanism suitable for the satellite topology characteristics is configured; real big data applications are deployed in the network using container technology, and in order to enable the simulation platform to reflect the real application performance, the distributed big data platform Hadoop is transformed and packaged as a docker image, and a control script is designed to realize the rapid deployment and configuration of computing / storage nodes in the network. In summary, the present invention develops a multi-dimensional joint comprehensive simulation platform for the distributed big data processing characteristics in the low-orbit constellation. The platform has three layers: constellation simulation layer, network simulation layer and application simulation layer. The data flow and control flow interaction of the three-layer simulation architecture are realized through Python control scripts and APIs. The beneficial effects of the present invention are: creatively deploying real data streams and real applications into the simulation platform, transforming the traditional network simulation using software models for parameter calculation into the simulation of actual applications and data streams; in order to facilitate flexible and efficient simulation, a parallel simulation script is designed based on the simulation engine of STK, which improves the parameter calculation capability of STK for large-scale constellations. At the same time, the relevant data is encapsulated as an interface and provided to the network simulation layer; in order to combine real big data applications into the simulation platform, the containerization technology is introduced to realize the rapid deployment of a large number of distributed processing nodes in the virtual network constructed by SDN, thereby improving the availability of the simulation platform.

[0070] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A SDN-based low-orbit constellation on-orbit big data processing simulation platform, characterized in that: The simulation platform includes a constellation simulation layer, a network simulation layer and an application simulation layer, wherein: The constellation simulation layer provides the simulation platform with a simulation of a real physical environment, including satellite orbits, satellite visibility and inter-satellite link parameters. The simulation results of the constellation simulation layer provide a basis for network topology construction and network parameter configuration for the network simulation layer. The network simulation layer provides a dynamic network environment for the simulation platform, including changes in network topology, changes in bandwidth and latency between satellite nodes, and provides an API to the application simulation layer through network control; The application simulation layer uses the API provided by the network simulation layer to implement distributed big data software and testing tools, and the application simulation layer is also used to count application indicators; The constellation simulation layer is implemented by jointly simulating in the form of a master process and several slave processes. The implementation process includes a scenario configuration process and a parameter calculation process. In the scenario configuration process, STKEngine is used in the master process to configure the orbit and satellite parameters of the low-orbit constellation; in the parameter calculation process, visibility and link parameters between each satellite in each time slot of the simulation are obtained from STK. The parameter calculation process is disassembled into several parts and processed by the slave process. The master process divides the satellites to be calculated into several groups according to the number of cores of several servers equipped with STK, and distributes the scene files and the numbers of the groups to be calculated to each slave process. The calculation results of the master process and the slave process are asynchronously transmitted to the network simulation layer through the Socket interface; The network simulation layer adopts a synchronous data receiving strategy to receive the parallel computing data of the constellation simulation layer. The network simulation layer adopts a master process and a slave process structure, wherein the master process of the network simulation layer and the master process of the constellation simulation layer obtain the orbit and satellite parameters of the low-orbit constellation through the socket interface, and start a corresponding number of slave processes of the network simulation layer to establish connections with the slave processes of the constellation simulation layer respectively; The network simulation layer adopts a time slot simulation strategy to divide the entire simulation time into several time slots, receives data parameters of the next time slot from the constellation simulation layer before the end of each time slot, and uses the acquired data parameters to calculate the network topology and network parameter configuration to generate a satellite network based on SDN; The network simulation layer provides an API to the application simulation layer through network control, and the specific implementation method includes: Use the SDN controller written in Python to monitor the data of the entire network and control the scheduling of network flows, deploy RYU to the network, and use the northbound API in the Python script to call the functions of RYU; The SDN controller and the switch complete information exchange through the Openflow protocol, specifically including: the switch forwards the data packets that cannot be forwarded to the SDN controller through packet_in, and the SDN controller obtains the forwarding rules through calculation and sends the flow table and group table to Open vSwitch to realize the control of the Open vSwitch forwarding strategy; Open vSwitch forwards data based on flow tables and group tables; The RYU is also used to implement the identification and scheduling of application traffic, identify the port number of the application through the control information of Openflow to classify the application traffic, and allocate the maximum bandwidth of each type of traffic as needed.

2. The SDN-based low-orbit constellation on-orbit big data processing simulation platform according to claim 1 is characterized in that: In the process of generating an SDN-based satellite network, the open source SDN network simulation tool mininet is used to build a network including hosts, SDN switches and SDN controllers and configure network parameters. Each application is packaged into a container image and connected to the SDN network. The mininet uses Open vSwitch as a virtual satellite node, establishes a network connection according to visibility and link parameters, and connects a docker image to each Open vSwitch.

3. The SDN-based low-orbit constellation on-orbit big data processing simulation platform according to claim 1 is characterized in that: The application simulation layer uses a distributed big data processing platform based on Hadoop to implement distributed computing and distributed storage, and the Hadoop is made into a docker image and connected to the network simulation layer.

4. The SDN-based low-orbit constellation on-orbit big data processing simulation platform according to claim 3 is characterized in that: The application simulation layer deploys multiple erasure code storage strategies in HDFS, and tests the performance of data download and repair in the satellite network by controlling the distribution of data blocks. At the same time, the HiBench test suite is used to test the shuffle traffic and task execution time of the Hadoop distributed computing component MapReduce.

Images