System and method for information slingshot on network tapestry and momentary granularity

Abstract

The present invention relates to systems and methods for information slingshots at network tapestries and momentary granularity. Systems and methods for file transfer and processing in a network environment are disclosed. In one embodiment, the system may include one or more processors. The one or more processors may be coupled to a first device. The one or more processors may be configured to retrieve a file from a file queue. The file may be stored in a local memory of the first device. The file may be transferred from a second remote device via remote direct memory access. The one or more processors may be further configured to determine whether the file is complete. The one or more processors may be further configured to remove the file from the file queue if the file is determined to be complete.

Description

[0001] This application is a divisional application of the invention patent with application date of December 9, 2016, application number 201680066545.X, and invention name "System and method for information slingshot at network tapestry and instantaneous granularity".

[0002] This application claims priority to U.S. Provisional Application No. 62 / 266,060, filed December 11, 2015, which is incorporated herein by reference.

[0003] This application claims priority to U.S. Provisional Application No. 62 / 296,257, filed on February 17, 2016, which is incorporated herein by reference.

[0004] International Patent Application Nos. PCT / US15 / 64242, PCT / US16 / 12178, PCT / US16 / 15276, PCT / US16 / 15278, PCT / US16 / 26467, PCT / US16 / 26481, PCT / US16 / 26489, PCT / US16 / 37180, and PCT / IB 16 / 00110 are hereby incorporated by reference in their entirety. Technical Field

[0005] The present disclosure relates generally to networking, and more particularly to accelerating data transfer over medium and long distances by fundamentally changing the nature of the exchanges between sender and receiver devices. The method migration is moving from round-trip packet transmission to one-way slingshot file sending, and using the principles of high performance computing to increase processing throughput.

[0006] The present disclosure specifically describes a one-way information slingshot that saves complete data files of any size to remote storage over a network tapestry at near line speed. The information slingshot can be integrated with existing network fabrics, or can be a standalone mechanism working independently, or a hybrid of the two.

[0007] The present disclosure also relates generally to software and networking, and more particularly to managing instantaneous time granularity of higher level functions. Background Art

[0008] Information slingshot on the web tapestry

[0009] Conventional techniques and the limitations or shortcomings of those techniques demonstrate why different approaches may be beneficial. The features, methods, and applications of the present disclosure described herein briefly describe how to construct and use such techniques. It also provides perspectives on how the present disclosure is saturated with specific improvements to overcome the limitations of conventional techniques.

[0010] The present disclosure may be beneficial for meeting various networking needs. This article provides the use of the present disclosure by the financial industry as an example to illustrate the features and how they overcome the shortcomings of conventional technologies. However, the scope and scale of application and use of the present disclosure transcends this narrow focus and is applicable to many other use cases.

[0011] The Internet is a network of networks. A network fabric can be defined as a network under the control of an administrative entity, or a network protocol, such as Internet Protocol (IP) or networks used within a geographic area. Fabrics can be woven together through peer-to-peer cooperation of network fabrics connected together at the edge. Methods such as Network Address Translation (NAT) can be used.

[0012] A Global Virtual Network (GVN) is a type of network that exists on top of the Internet (OTT), weaving the layers of the fabric into a network tapestry with entry and exit points (EIPs) at each fabric edge.

[0013] One network fabric may be a wide area network (WAN) consisting of dedicated lines between two local area networks (LANS) points, such as Multiprotocol Label Switching (MPLS) or Digital Data Network (DDN), etc. These dedicated links may also be connected to other networks peer-to-peer, based on a one-to-one relationship defined as a single hop, or via a one-to-many network relationship of multi-hop peering, with many possible paths or routes in between two or more endpoints.

[0014] A network fabric can also define the scale and scope of the type of network fabric that is end-to-end. Ethernet defines a type of network, but this can also be further classified by protocol, such as Internet Protocol (IP) on Ethernet, such as Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) and other network types. Built on top of IP are protocols such as Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). TCP over IP (TCP / IP) is more verbose and has greater built-in error checking and handling for the reliability of data sent relative to UDP over IP (UDP / IP), which is more configured for data streaming, such as audio or video data, where the loss of a packet will not cause significant negative effects.

[0015] In addition to the different protocols and IP versions built on top of Ethernet, Ethernet itself has Ethernet, Gigabit Ethernet (which can have speeds of 1 or 10 or 40 or 100 Gigabits), with other versions expected to be introduced. Ethernet is writing common mandatory protocols to connect various devices together, and they can communicate with each other.

[0016] InfiniBand (IB) is an Ethernet replacement protocol that utilizes different physical NIC ports, plugs, and cables, and IB works in a different way. IB is a preferred choice for connecting many nodes to a high performance computing / computer (UPC) environment. IB allows local remote direct memory access (RDMA) between nodes that bypass the network and the operating system (O / S) stack of the host device for access, where RDMA memory (or other) devices are connected. There are also alternatives, such as Fibre Channel and other network technologies, which can provide maintenance of fast local RDMA transfers and save files to a parallel file system (PFS) or RAM over long distances or to another remote storage device.

[0017] Although Fiber Channel is an alternative to IB, for the purposes of this disclosure, this disclosure focuses on IB as the network type of choice for connecting UPC nodes, especially over large distances. IB is used as an example for demonstration, but any alternative network type that can provide a similar basic connection can be used as an alternative to support information slingshots.

[0018] Reliability is extremely important when comparing network types. Another major driver that influences the choice of network type and network protocol is also related to the time it takes for communications to travel across the physical path distance. Latency is a measure of the time it takes for data to travel in one direction, or it can also be expressed as the round-trip time (RTT) over a specified distance between two points. Bandwidth is a measure of the data transfer rate of a network segment, communication route, network fabric, or other network path.

[0019] Some physical laws are cited as basis in this article. Because time and distance are significant, this disclosure uses the following baseline:

[0020] - The distances in this article are measured in miles under the experimental system

[0021] - Distance measures in this article can be numbers with or without commas and / or decimals, or can be expressed as integers

[0022] - An exception to the distances used is the refractive index of the optical cable described below, where distances are expressed in meters under the metric system

[0023] -Time is measured in seconds, expressed as whole, fractional and / or decimal seconds

[0024] - The time units quoted in this article are seconds, milliseconds (ms) and microseconds (μs)

[0025] - Any granularity finer than microseconds, such as nanoseconds (ns), may be important in certain practical applications of the present disclosure, but for the purpose of presentation, the finest granularity herein is μs.

[0026] In computing, the current main or standard measure of time for networking is milliseconds (ms), and for processing is microseconds (μs). The granularity of very short time can be measured, for example, as fractions of a millisecond (every 1 / 20 or 1 / 10 or 1 / 100) or decimals (0.05, 0.1, 0.01). The following table is a reference to some possible values ​​and their corresponding equivalent conversions.

[0027] # illustrate Second Milliseconds (ms) Microseconds (μs) 1 1 / 10 of a second 0.10000 100 100,000 2 1 / 20 of a second 0.05000 50 50,000 3 1 / 100 of a second 0.01000 10 10,000 4 10 microseconds 0.00001 0.010 10 5 100 microseconds 0.00010 0.100 100 6 1000 microseconds 0.00100 1.000 1,000

[0028] Table 1 - Measurement of time

[0029] The table below compares the speed of light in a vacuum to the speed of light inside the glass core of an optical fiber. This table illustrates the physical limits of optical fiber efficiency to establish a baseline for the theoretical best speed one could hope to achieve for light traveling through an optical fiber.

[0030]

[0031] Table 2 - Fiber line speed considering fiber delay due to refraction: Based on data from the following website: http: / / www.m2optics.com / blog / bid / 70587 / Calculating-Optical-Fiber-Latency

[0032] Although the refractive index of the fiber optic cable may vary slightly, an average value is assumed in this article as follows: For an efficiency of 68.05%, the average value is approximately 203 to 204 m / μs, relative to the speed of light of 299.792 m / μs.

[0033] Some special industries have applications that have an absolute requirement for the best quality of service (QoS) with the lowest possible latency and zero packet loss. These special industries need to exchange application-specific information, such as financial information, transaction execution to confirmation, scientific computing, etc. In many cases, dedicated fiber lines will be used for transmission to achieve higher reliability and higher throughput than expected over the open Internet.

[0034] An alternative to terrestrial fiber optic line communications is to relay via satellite transmission. Slingshot is capable of operating over satellite links, and communication speeds may approach the speed of light using such an underlying transmission medium. Satellite links are susceptible to bandwidth limitations, weather risks, cosmic interference, and other challenges. For purposes of this disclosure, the intersection of the Information Slingshot is to deploy its display over a terrestrial fiber optic line network.

[0035] Ethernet and its combination of networking technologies are the most widely used network type, deployed from local area networks within offices, data centers, and other clusters of devices to global backbone link networks operated by Internet Service Providers (ISPs) across the global Internet. Despite its widespread deployment, the Internet Protocol - IPv4 became a victim of its own success as it faced an almost complete exhaustion of the number of available IP addresses. Technologies such as NAT for devices in LANs have helped alleviate this problem, but have not yet solved it, and IPv4 addresses are still in very short supply. IPv6 appears to have an abundant supply of available IP addresses at its core, however, IPv6 has not been widely deployed due to several factors, one of which is the infrastructure investment in legacy equipment that only handles IPv4 rather than both IPv4 and IPv6. Ethernet became the dominant network type, with widespread use common in LANs and the wider Internet because it is a relatively easy to implement and globally deployed standard. As more and more devices utilize a protocol or network type, network effects also come into play, which makes the decision for others to adopt it easier, further incentivizing high capacity, leading to commoditization with the effect of lowering prices.

[0036] In data centers, centralized computing, storage, processing, and other operations are distributed on various rack-mounted servers, requiring faster transmission than Ethernet to connect these servers together behind the channel so that they can share data and communicate. Fiber Channel and InfiniBand (IB) are two such technologies that provide ultra-low latency and high-capacity bandwidth. IB's lossless and parallel transmission provides strong advantages, allowing the use of remote direct memory access (RDMA), and also provides opportunities to deploy and utilize a globally distributed parallel file system (PFS). The limitation of IB is that it is only deployed at shorter distances measured in meters. This was later extended to several kilometers. Until recently, IB "long-distance" links were limited to within cities or between two metro areas close to each other, where data centers were connected to each other via ultra-fast IBs through dedicated lines between them. Recent technology allows the distance of IB to be extended and up to 20,000 kilometers between two IB gateways through dark fiber lines.

[0037] Conventional Ethernet IP protocol technology is known to have shortcomings, especially over long distances. The problem with IP is inherent in how packets are sent. IP communications using its most commonly used protocol TCP / IP require round trips and redundancy, resulting in a higher than necessary RTT.

[0038] IP transmits data in packets. Each packet consists of a header, a data payload, and possibly other information. The packet size allowed by the IP network fabric is defined as the Maximum Transmission Unit (MTU) and may be limited to a few hundred bytes to 4KB or more, but if the payload is larger than the set packet limit allows, in order to transmit the payload from point A to point B, the payload needs to be split into multiple parts of two or more packets, and each subsequent transmission and acknowledgment requires a round trip. Packet size also affects whether packets are subject to fragmentation. Fragmented packets are often reassembled if the fragmentation is forced by an intermediate device that has a lower MTU than other devices between network segments in the transmission path through the chain of devices. This shortcoming of IP can cause packets to be lost and must be retransmitted. Even if all packets arrive, fragmentation adds congestion to the middle and may flood the receiving device with more packets than it was expecting. For example, if the sending device uses the common setting of 1500-byte MTU and breaks the payload into a stream of packets that are 1500 bytes each, the sending device will stream them out as fast as the stack and NIC can handle, up to the available bandwidth of the "last mile" connection. If a device in the middle of the Internet has an MTU setting of 1492, then every packet that flows through this device will be fragmented or divided into multiple packets. As a result, the receiving device receives twice as many packets as the sending device streams. This can saturate the receiving device, causing congestion and potential losses, or at least slowdowns. Furthermore, network devices are rated to send so many packets per second, no matter how "full" the packets are. Fragmented packets are typically less than full to capacity, further reducing potential bandwidth, causing a bumping effect that can hinder network performance.

[0039] The reliability and authentication of TCP / IP packets provides peace of mind, but at the expense of speed and enforced rigidity, as some of the disadvantages of long-distance IP conventional technology are that there are limits due to physical limitations and the nature of the network protocols.

[0040] In particular, packetized transmissions, such as multi-part packet transmissions that split larger payloads, run the risk of being fragmented by intermediate devices, which may not be able or configured to handle large payloads.

[0041] It is known that Ethernet / IP round trip transmission has QoS degradation over long distances with delays, known problems such as congestion, which causes delays, especially with distance. TCP / IP can catch lost packets and resend them. Although this is good from a reliability perspective, it adds delay and congestion to the network.

[0042] It is also important to point out how Ethernet handles data packets, as it has a store and forward model where packets are received, inspected, and forwarded only after the entire packet, including the full payload, is completely received and inspected. The latency within the computer / router / network to handle Ethernet traffic packets is on the order of 100 microseconds (μs) per device. This 100μs delay lag per intermediate device for a packet to be received and passed on may seem minor, but over a long distance path, this delay can be magnified by passing through many devices. NAT also adds another small but still measurable point of friction. The delays caused by the client-server approach - prepare packet, send, transmit, receive, assemble packet, consume - add bloated and unnecessary steps to the communication flow within the stack of the device's O / S.

[0043] There is still room for improvement in the current IP protocol, including the need for lower latency, eliminating packet loss, and solving other problems such as inefficiencies in packetization and bidirectional transmission, solving bandwidth-delay product (BDP) and related algorithms that govern file transfers.

[0044] Although current trading methods have been designed to be as efficient as possible, broadcasting market information via UDP / IP multicast, and sending trade orders and confirmations in the smallest possible packets with the lowest latency via TCP / IP, the problem still exists. UDP / IP does not have the same error correction mechanism as TCP / IP, so neither the receiver nor the sender knows if a packet has been lost. Packet loss is more likely during times of heavy traffic, which is precisely the time when it is most difficult for traders to have complete information urgently.

[0045] There are also changes in calculating which network protocols are necessary to solve the troubles in moving large amounts of data.

[0046] New paradigms are needed due to emerging technologies such as Blockchain (and other distributed registry technologies), Industrial Internet of Things (HOT), telemedicine, etc. where there is a requirement for fast, reliable transmission of ever-increasing data payloads. Slingshot is a mechanism for sending information in one direction as a single file of nearly arbitrary size.

[0047] Information Slingshot uses RDMA over IB or similar underlying network types to store data files of unlimited size on a remote storage device such as PFS, and has a File Queue Manager (FQM) or similar mechanism on the device in the target remote area pull the file from the PFS. Error correction is built into the storage device, which then governs the process of writing the file, just like the interaction between the locally connected storage device and the O / S within the host device. The significance of Slingshot is that the internal process distribution is called distributed process. One-way transmission is another feature of Slingshot, which achieves the efficiency of UDP / IP while achieving the reliability of TCP / IP and solving the need for confirmation of the packets sent. Slingshot uses parallel stream file storage, which has the following effect: the last byte arrives at the same or nearly the same time as the first byte sent.

[0048] Slingshot can leverage any underlying network that can support RDMA and has properties similar to IB. High Performance Computing (HPC), such as RDMA, the principles of IB and other properties provide a foundation on which novel approaches are to be built. Note that the storage on the PFS device can be physical or virtual, or HDD, SAS, SSD, RAM disk or any other device that can provide a fast storage medium and allow RDMA or equivalent type of direct remote access.

[0049] For demonstration purposes of this disclosure, IB is utilized as an example network to run Slingshot on. The IB bandwidth under load can achieve 90 to 95 percent of the theoretical BW maximum. IB is characterized by passing through a switch, where the IB device receives the header of the packet, uses a logical forwarding decision and continues to transmit the payload through the pipeline, even before receiving the complete payload. Therefore, IB has extremely low latency compared to Ethernet. It has much lower redundancy than TCP / IP or UDP / IP. It runs on top of a dark fiber connection. Compared to Ethernet on dark fiber, it is still faster, and if local RDMA is utilized, the latency can be measured as one-way for the effective transmission, rather than two-way like RTT for Ethernet TCP / IP.

[0050] In the context of metal data transmission, the present disclosure is a framework to leverage tapestry and utilize advanced connectivity methods to address the communication needs of accelerating global trade execution and other needs.

[0051] One important difference is that Slingshot does not do traditional file transfers, but rather stores the files in remote storage. This Slingshot approach transfers larger chunks of data faster from one region to another in a very efficient manner as a bunch of various data payload "files" are transferred.

[0052] Significant gains are obtained over long distances. Reference is made to the cited exemplary embodiment figures and their explanations for further details.

[0053] In this article, for example Fig.29 The entry and exit points (EIPs) at the edges of the fabric between EIP 2970 in the GVN integrate information slingshots into the tapestry of network types such as GVN. In addition, using the information slingshot mechanism as a foundation, other features can be supported, such as bouncing (U.S. Provisional Patent Application No. 62 / 327,907), information data beacons (U.S. Provisional Patent Application No. 62 / 327,846), bouncing routing (U.S. Provisional Patent Application No. 62 / 327,911), etc.

[0054] An HPC cluster in which all nodes are capable of acting as control nodes places files in a PFS storage location for retrieval by other nodes. Although nodes can pull files from a PFS in a remote area, the added latency of pulling and marking and moving files makes it less meaningful. It is best for the sender to save the file in a PFS as close to the target as possible. A feature of the present disclosure is that the nodes of the PFS storage instance are aware of each other, and that the PFS storage instance is close to the target device where the sender wishes to read the saved file.

[0055] Any file presented to the information slingshot mechanism from any source will be saved remotely in the target area as a complete file. The file queue manager will retrieve the file from the queue on PFS or RAM or any other alternative storage medium. It will process the file, check whether it is complete at the time of processing. The complete file will be available from the next computation chain.

[0056] The Information Slingshot overcomes the shortcomings of conventional technologies because it provides an efficient and fast workable framework for sharing large amounts of information between devices over long distances.

[0057] Significantly reduces transaction latency. Long distance transmission is close to line speed, does not require breaking files into multiple parts, and does not packetize or have assembleable packets when sending. The present disclosure also eliminates packet bloat of packet headers and multi-part delimiter boundaries. Straight-through transmission is much more efficient than store-and-forward IP packet processing. Advanced error detection and correction on file writes further provides higher efficiency.

[0058] The nature of information exchange changes from sequential packet transmission to long-distance parallel multi-threaded file transfer, and parallel files are saved to storage in remote areas more efficiently.

[0059] There is also the added benefit of higher availability of files accessed via RDMA, which contains structures and extended payloads to facilitate the transfer of larger data. RDMA has ultra-fast reads and writes to / from storage devices without having to go through the host because it bypasses the stack at both the origin and destination.

[0060] Bottlenecks and resistance points due to the nature of the IP protocol are also avoided by the present disclosure.By saving complete files to a remote PFS for another device to pull and use, packetization is no longer required.

[0061] Decoupling the round-trip RTT nature of packetized communication to a one-way send file reduces transmission time. Another less obvious advantage is that the body data may be variable in file size.

[0062] With conventional technology, if the data to be transmitted is larger than the packet's payload size specifies, it will be broken into multiple parts, which will require multiple round trips to achieve the transmission. This adds a lot of time to complete the transmission and more processing to generate and then verify and reassemble the multi-part message. Information Slingshot provides a known and consistent transmission time between points, without requiring the sending device to separate the file, and the receiving device to reassemble it.

[0063] It overcomes the delay caused by the cumulative total of various transmission times and processing times for a packet chain. In the case of calculating the packetized Ethernet IP packet transmission time as RTT time * total number of packets, the information slingshot gives a more accurate timing mechanism.

[0064] Information Slingshot can send entire files, without breaking the data into multiple part payloads, as Ethernet IP does, by sending them in packet streams. This also has the positive effect of keeping the data moving at a very low level, without requiring the CPU, RAM or other elements of the device's stack to do anything.

[0065] The information slingshot further avoids the need to receive a stream of packets. Ethernet consumes valuable time and resources when it has to receive a stream of packets and then authenticate each other, confirm that they are in the correct order, and then reassemble the data file.

[0066] The core advantages of Information Slingshot (RDMA OTT IB) are parallel streams during transmission, low latency, lossless transmission, zero-copy networking, reduced load on hardware via cut-through switching, RDMA bypassing the host device stack at both ends, local use of parallel file system (PFS), one-way transmission without the need to confirm packets, no RTT, only one-way transmission, and no "packet size" limit.

[0067] The nature of the transfer via Slingshot can change how the transfer is approached. The data file header and body data can be more verbose and can include a file header and an explicit end-of-file (EOF) marker. Including parameters, information arrays, algorithm logic frameworks, and more body files can allow for more feature-rich data transfers. Data file trailers with checksums, security, and other integrity-related items can also be included within the transferred file.

[0068] The present disclosure can provide communication advantages. Near real time across long distances at near line speeds can be achieved. At short distances, time (microseconds) can be saved from existing methods, resulting in rapid access to data in remote areas. Information Slingshot changes how transactions are performed by speeding up transmissions, simplifying addressing, allowing larger file payloads, and providing other advantages.

[0069] The accompanying drawings and example embodiments further describe the conventional technology, the disadvantages of the conventional technology, and the features of the present disclosure, and how these features overcome the disadvantages of the conventional technology.

[0070] Network Slingshot and derived technologies may provide benefits such as, but not limited to: a) Network Slingshot is a one-way fast data delivery in a slingshot; b) there may be no need to packetize traffic, unlimited file size, arrival of the first byte of transmission at near line speed, and the fastest possible last bit arrival time via dynamic parallel transmissions scaled based on available BW overhead; c) Slingshot-based High Performance Networking (HPN) provides numerous advantages; and d) other slingshot-related technologies may be discussed during a more formal public meeting.

[0071] Financial and other network technologies based on information slingshots can provide benefits such as, but not limited to:

[0072] 1. Information transmission via data beacon pulsators - address issues with data feeds from the market - etc. Advantages, no loss due to congestion or packet loss, data is sent as files with unlimited file size. No matter how far between source and destination, there is point-to-point, 100% reliability at near line speed.

[0073] 2. Order of trade execution - occurs at near line speed, enabling larger message types, which can allow complex algorithms to be sent from a source host to a target host at the market edge where a series of instructions can be executed, sending large amounts of logic at once as a unified block of data compared to individual data packets.

[0074] 3. Blockchain - Although a lot of interest and attention has been invested in this technology, one problem that has not been adequately addressed is that the high degree of associated data results in a fairly large file size, and current IP protocols are not the most efficient for transmission. Slingshot and corresponding technologies allow the transmission of entire blockchain files at near line speed with 100% reliability.

[0075] 4. Enhanced integration must also be locally integrated into the workflow of existing legacy networks within the area from client / server devices to the slingshot mechanism. The network components of the information slingshot are designed to enhance the long distances between various networks between regions or areas and speed up the flow of data between them.

[0076] 5. Bounce can also be integrated into the existing IP framework as a seamless transparent segment, so that: a) permission is integrated into the existing legacy IP path workflow; and b) intermediate optimizations seamlessly and transparently exploit the benefits of bounce for client traffic passing through the bounce.

[0077]

[0078] Table 3 - Comparison of Ethernet IP Round Trip Time (RTT) and Fiber Backbone (FBB) Legacy: Data Source:

[0079] https: / / www.sprint.net / lg / lg_start.php-ping tests be tween points

[0080] http: / / www.verizonenterprise.eom / about / network / latenc y / #latency-static tables for reference https: / / wondernetwork.com / pings / Hong+Kong-dynamictables-can set base city and retrieve stats

[0081] https: / / ipnetwork.bgtmo.ip.att.net / pws / network_dela y.html-tables*(US)

[0082] https: / / ipnetwork.bgtmo.ip.att.net / pws / global_netw ork_aVgs.html-tables-global averages

[0083] Source of FBB old style pictures:

[0084] FBB one-way legacy calculated specifically for this disclosure.

[0085] Instantaneous granularity

[0086] The present disclosure describes various methods to command fixed or variable data processing cycles with extremely fine granularity for instantaneous time. The time allocated for a cycle can be dynamically calculated based on several factors including the time required to process items during the cycle. Post-processing during the time cycle can be considered, using this information to algorithmically calculate the maximum number of items that can be processed. Information is available both to and from the application layer.

[0087] Software in the form of scripts or programs or routines or other types of code blocks (SW) can be executed using a one-time trigger. In some cases, this is sufficient. However, some types of SW need to be run at regular and repeated time intervals.

[0088] Built into UNIX and LINUX systems is a scheduler called a command run on a scheduler (CRON) that will execute software to execute at a specific point in time. It can also be set up for intervals. An alternative to CRON jobs would be to have a daemon running as a background process that has an internal clock with set times and other triggers at which point other software jobs can be started. In theory, these would work like CRON, but could be set up in a more customizable way.

[0089] It is also possible to write a script so that once executed, it keeps looping based on a condition that never becomes true to force an exit, or based on a very large number of times that it allows it to loop with a decreasing number of times each time it loops. In cases where the loop time granularity is less than the minimum one minute required for CRON table establishment, a combination of CRON starts and internal loops can be used. For example, if a specific event occurs every five seconds, a one-minute CRON can be triggered, and within the SW script, the loop iterates twelve times (60 seconds / 12 loops = 5 second intervals).

[0090] A script can also run, and then at the end of its execution, call a new instance of itself to run again. This can have a sleep function at the end to pause between when it next runs, or a sleep of zero seconds can be set, implying a new call to itself to start immediately at the end.

[0091] The associated manager can act as a monitor by doing things like monitoring a directory, which can trigger the launch of another script for processing files when its file list is filled with a certain number of files. Thus, the conditional launch of the SW script is based on a non-time-based factor to be reached. This can change based on when the queue is filled with a certain number of items in the queue, filled with the total memory size in RAM or queue memory, or based on free space remaining or other factors.

[0092] For looping or execution of software scripts, internal and external conditions can be set, such as maximum execution time or mechanisms to try and confirm that a script or function or other routine is only run once. This checks to confirm that there are no duplicate instances running to avoid overlapping operations of two running at the same time, which may interfere with each other.

[0093] Critical to hardware operation is the internal system clock, on which many functions rely. Internal clocks can drift over time, and others can run slower or faster than they should, so they are updated by running software scripts periodically.

[0094] Set execution times and intervals are either hard-coded into the script or as dynamic variables, allowing interdependent scripts / routines / managers / modules or other software to utilize the same instantaneous parameters. These instantaneous dynamic variables can be set in a database, included in a flat file, calculated or generated by other means.

[0095] A time trigger can have its uses, but occasionally there is a need to keep running, re-run the code, script or routine, so the trigger needs to be automated. In order to preserve system resources and also have more control over these triggers, a framework associated with time is needed to allow scheduling.

[0096] CRON can be run at a specific time of a specific day or every X time units or by various other calendar-related time periods. The minimum time unit specified by CRON is one minute. This is acceptable for tasks related to a specific system, such as log rotation, scheduled backups, or other routine programs that can and should be automated. However, some tasks require much finer granularity than one minute and require more monitoring or control at the userland or user space software application layer.

[0097] In the case of statistical or conditional looping scripts, or scripts that call themselves at the end of the current cycle, there are several problems, but one of the main ones is that the next call is made at an undetermined time. There are ways to track the run time and add sleep functions to try to fire the next loop or call iteration at regular intervals. However, this only works for cycles that run in less time than the interval. In the case that the cycle takes longer than the expected interval, the next tick will be late, and thus may be out of the next cycle timing. Managing this process can become complicated very quickly. So it creates issues of granularity and dependencies.

[0098] Daemons run background processes very similar to CRON, but require their own scheduler and manager to control when they run.

[0099] Conditional execution based on factors such as list length, amount of queue items, free disk space, or other factors can be efficient in some cases, inefficient and inconsistent in others. Some items can be stuck in the queue for inconsistent amounts of time. And this doesn't take other factors into account, so sometimes server resources are underutilized, and then at other times, are crushed by overutilization, possibly causing other negative impact issues.

[0100] The CRON loops within the script that are triggered every minute are sequential, so if 12 loops are scheduled in 60 seconds, five seconds are allocated for each loop. As mentioned above, if the loop only takes 3 seconds, a 2-second sleep can be issued at the end of the loop to maintain the 5-second regularity. Real problems arise when one or more loops are greater than 5 seconds, which may push the script time longer than 60 seconds. The next time, the CRON scheduler will call the next instance of the script, which will begin its cycle of 12 loops. Adverse problems with currently running scripts may cause significant problems due to overlapping processing of files, or in the case of transaction processing, if two scripts are performing transactions, there may be unexpected copies, or isolated steps belonging to independent processes or other problems.

[0101] Conventional techniques have other drawbacks, which primarily deal with millisecond (ms) granularity, or even finer time granularity such as microsecond (μs).

[0102] In computing, the current primary or standard measure of time for networking is milliseconds (ms), and for some processing microseconds (μs). The granularity of very short time can be measured, for example, as fractions of a millisecond (every 1 / 20 or 1 / 10 or 1 / 100) or decimals (0.05, 0.1, 0.01). When scheduling the execution of software scripts, routines, etc., when they are triggered to run, greater control is needed. For very fine time granularity, such as when interval time is measured in ms or μs, an intelligent manager is needed to adjust the instantaneous time granularity.

[0103] Such a manager requires an accurate and up-to-date system clock. Built into the manager is a time updater to keep the system clock as synchronized as possible with the actual time. Best-of-breed protocol algorithms can be utilized, such as the crossover algorithm or the Marzul lo algorithm.

[0104] This moment manager also relies on the system resource manager, understanding the system processing tasks and other physical capabilities (and constraints), and other factors to estimate how long a script or routine or other software call cycle should take to complete its operation.

[0105] By also looking at items to process before processing, a more reasonable time estimate can be made. In the algorithmic calculation of how long an instant should take, three main factors are utilized when calculating the difference t, or the amount of time from the start to the end of the instant.

[0106]

[0107] The instant granularity is based on several factors. The start time of an instant is based on the completion of the previous cycle of the instant invocation or according to a scheduled fixed time interval.

[0108] There are two types of cycles:

[0109] Fixed-time cycles set a finite number of items based on an estimate of how long it will take to process (P) each item and then deal with post-processing (Q). This limit ensures that each cycle does not run over time.

[0110] A variable time period allows all items in the batch to be processed. The R delay ensures that the next cycle pulls a new batch and all items in the last processed batch have been processed (P) and the post-processing (Q) is complete. For example, for files in the queue, the list can be processed on page P, and then the files can be marked, moved, deleted, or touched in other ways so that they are not pulled in the next batch. The time delay (R) ensures that there is enough delay between the last item processed and post-processed in the previous batch and the first item to be processed in the next batch.

[0111] The component of this moment manager is to maintain a history of runtime data for each cycle. These recorded records can be used to analyze times of underutilization and peak utilization. It can also detect delays caused by unprocessed items being cut off due to the limited amount of time to process each cycle. If this happens too often, then it is an indicator that more processing power is needed, the software is not efficient, or it can indicate other problems such as a slow database or other need for software refactoring, or other remedial measures to solve the problem.

[0112] The instant can have as fine granularity as the application requires, and can be set to run at fixed intervals, at variable intervals based on cycle length or other factors. This instant granularity allows for consistent network beacon flashing, slingshot related functions, and other time-dependent functions.

[0113] For more features, see the example embodiments given in the drawings and their description. Instantaneous granularity relies on timers, time circuits, timekeeping software, logic, time updaters, and other timekeeping and synchronization techniques to work properly so that they provide accurate system time for internal clocks and other mechanisms. Instantaneous granularity is built on top of these time mechanisms.

[0114] The instant granularity is set to a fixed time period through dynamic variables and based on ever-changing conditions. The number of items in the queue can also be set by dynamic variables or other factors. Continuous testing, monitoring resources, recording current performance, comparing to past performance, determining system load requirements and other intelligent methods provide a great ability to determine how long each instant should take to process (P), post-process (Q) and delay (R). Knowing how much time each item should take to be processed during the cycle called by the instant allows the setting of limits on the items processed during subsequent instant cycles.

[0115] Adjustable in-flight moment managers and metrics indicate system health, identify upgrade needs, and can also be engineered into distributed processing configurations to pull resources on demand based on need and availability. Algorithmic analysis at work governs how many items can be processed, balancing available resources with how much those resources can physically handle. Predictability and reliability are the results.

[0116] The intervals between ticks can also be adjusted accordingly based on dynamic user requirements and the contextual reality of the situation. Such a high degree of control over the tick manager at very fine time granularity, such as ms or μs, is advantageous for several reasons, such as:

[0117] ● Prevent overlap in batches during fixed instant calls

[0118] ●Can detect and control the escape process

[0119] ● Code inefficiencies can be identified, allowing the development of a framework to be used as a basis for research, diagnosis, and more importantly, problem solving.

[0120] • This provides a way to utilize the hardware (HW) more efficiently.

[0121] • Other advantages will be apparent to those skilled in the art.

[0122] The instantaneous granularity forms the basis of reliable time intervals on which other software applications can be built, for example, beacons set to pulse with a period of Xms for network information, or pings or other types of time-sensitive applications.

[0123] It is also possible to make the monitoring data available at the software layer so that it adjusts its operations to the current conditions and enables time-related information to be exploited at a very high level.

[0124] The option of fixed time periods or adjustable time periods may be beneficial in different situations. By having a manager run and monitor the cycles, during peak usage periods when cycles run longer than expected, those subsequent cycles can be run for shorter times during times of less demand to bring the cycles back to regular periods and catch up if they were previously delayed.

[0125] This manager and its data are easily integrated into existing architectures as a stream manager to realize benefits.By having more control over the momentary granularity, other applications and benefits can also be realized. Summary of the invention

[0126] Disclosed are systems and methods for file transfer and processing in a network environment. In one embodiment, the system may include one or more processors; the one or more processors may be coupled to a first device. The one or more processors may be configured to retrieve a file from a file queue. The file may be stored in a local memory of the first device. The file may be transferred from a second remote device via remote direct memory access. The one or more processors may be further configured to determine whether the file is complete. The one or more processors may be further configured to remove the file from the file queue if the file is determined to be complete.

[0127] According to other aspects of this embodiment, the second remote device may be a backbone switching server.

[0128] According to other aspects of this embodiment, the system may further include a dedicated link between the first device and the second remote device.

[0129] According to other aspects of this embodiment, the file may include a header, a body, and a file trailer.

[0130] According to other aspects of this embodiment, the file tail may include a mark indicating the end of the file.

[0131] According to other aspects of this embodiment, the file may include a key-value pair.

[0132] According to other aspects of this embodiment, the file trailer may include a checksum to verify the integrity of the body.

[0133] According to other aspects of this embodiment, the remote direct memory access can be implemented based on at least one of a wireless bandwidth or a fiber channel.

[0134] According to other aspects of this embodiment, the file may include executable code.

[0135] According to other aspects of this embodiment, the one or more processors may be further configured to: if it is determined that the file is complete, perform at least one of the following operations: set a flag on the file, move the file to a folder, or delete the file from the local storage.

[0136] In another embodiment, a system and method for managing the granularity of time allocated for processing tasks is disclosed. In one embodiment, the system may include one or more processors configured to determine a duration required to process a task. The duration may be calculated based on a first duration for processing the task, a second duration for post-processing associated with the task, and a third duration as a delay.

[0137] According to other aspects of this embodiment, the task may include batch processing.

[0138] According to other aspects of this embodiment, the duration may be in at least one of microseconds or milliseconds.

[0139] According to other aspects of this embodiment, the duration may be adjusted based on a history of runtime data for the duration.

[0140] According to other aspects of this embodiment, the second duration may include the time required to update the task status. BRIEF DESCRIPTION OF THE DRAWINGS

[0141] In order to facilitate a more complete understanding of the present disclosure, reference is now made to the accompanying drawings, in which like elements are referenced with like numerals or reference characters. These drawings should not be construed as limiting the present disclosure, but are intended to be illustrative only.

[0142] Figure 1 A distance comparison of various segments of a long distance data path is shown.

[0143] Figure 2 An exemplary problem for long distance transactions is shown.

[0144] Figure 3 An exemplary conventional technology implementation in the real world is shown.

[0145] Figure 4 Shows the inefficiency of current technology.

[0146] Figure 5 A GVN topology is shown, including a backbone segment over the Internet or dark fiber.

[0147] Figure 6 Conventional technology is shown - the file is sent as a multi-part transmission via packetized transmission over TCP / IP.

[0148] Figure 7 Shown is the topology on the tapestry - HPC power.

[0149] Figure 8 An exemplary workflow for files saved over long distance to PFS is shown.

[0150] Fig. 9 Traditional client server REQ RESP and backend DB replication (multi-master) are shown.

[0151] Fig.10 Information via RDMA is shown slingshotting to Db.

[0152] Fig.11 Shows information slingshot-queuing files via RDMA.

[0153] Fig.12 It shows the independence of information slingshot-queue files via RDMA, and SRV_ACC performs cross-DB verification.

[0154] Fig.13 An exemplary use case for the Information Slingshot is shown - the relay of market information from the market to the trading desk.

[0155] Fig.14 An exemplary use case for the Information Slingshot is shown - a trade execution request from a client source to the marketplace.

[0156] Fig.15 An exemplary use case of the Information Slingshot is shown - trade results from the market back to the client.

[0157] Fig.16 An exemplary workflow of the process of writing information to a file and then reading it is shown.

[0158] Fig.17 An exemplary workflow for writing information to a file is shown.

[0159] Fig.18 An exemplary workflow for reading a file and processing it is shown.

[0160] Fig.19 An exemplary embodiment of the information slingshot is shown with respect to the file header / file segments: header, body data, and file end.

[0161] Fig. 20 An exemplary embodiment of an information slingshot is shown with respect to slingshotting, storing in a PFS, and processing files from a repository.

[0162] Fig.21 An exemplary embodiment of an information slingshot is shown with respect to harvesting a file queue to retrieve subject data for analysis and use.

[0163] Fig. 22 An exemplary embodiment of an information slingshot relative to file body data and EOF is shown.

[0164] Fig.23An exemplary embodiment of an information slingshot relative to a file process flow is shown.

[0165] Fig.24 Illustrated is a potential problem with file pull queues such as copy processing.

[0166] Fig.25 An exemplary embodiment of synchronization that batch-pull only files that are not marked as being read during a moment is shown.

[0167] Fig.26 An exemplary workflow for a post-critical backend process (to write to remote) is shown.

[0168] Fig. 27 An exemplary workflow for a post-criticality (to read from PFS and process) backend process is shown.

[0169] Fig.28 An exemplary system diagram for an information slingshot is shown.

[0170] Fig.29 The structured combination of hierarchical network weaving into a network tapestry is shown.

[0171] Fig.30 Exemplary sequential layers of various network textures forming a tapestry are shown.

[0172] Fig.31 A conventional technique - CRONS is shown.

[0173] Fig.32 A conventional technique - cycle is shown.

[0174] Fig.33 A conventional technique is shown - recursion - where the script calls itself.

[0175] Fig.34 shows the problem of looping.

[0176] Fig.35 The difference time = P + Q + R is shown.

[0177] Fig.36 The timing of various cycles is shown, where X≠Y.

[0178] Fig.37 The timing of various cycles is shown, where X is consistent.

[0179] Fig.38 The distance thresholds at which gain is achieved from the slingshot are shown.

[0180] Fig.39 The synchronization of batches relative to the marker file is shown.

[0181] Fig.40The instantaneous granularity is shown.

[0182] Fig.41 The instantaneous granularity is shown relative to a variable time interval.

[0183] Fig.42 An exemplary algorithm is shown with respect to momentary adjustments.

[0184] Fig.43 Artificial intelligence analysis, granularity and cycle are shown.

[0185] Fig.44 Exemplary components of the transient component are shown.

[0186] Fig.45 An exemplary momentary use in beacon flashing is shown.

[0187] Fig.46 An exemplary instantaneous use of a beacon in multiple pulses is shown.

[0188] Fig.47 An exemplary momentary use in beacons at intersections is shown.

[0189] Fig.48 An exemplary system diagram for instantaneous granularity is shown. DETAILED DESCRIPTION

[0190] Figure 1 A distance comparison for each segment of a long distance data path is shown. This example embodiment describes a dedicated line between two endpoints of a network connecting either end. It is a common misconception that long distance communications over optical fiber travel at the speed of light, however, the true line speed is at best only 68% efficient compared to the speed of light (see Table 2 herein). For long distance communications data transmission, such as conveying financial information, making a transaction request, receiving confirmation, or some equivalent type of information exchange, a lot of work has been done to make the path as efficient as possible. Although improvements may be possible through continued innovation in either local area, the relative amount of time received in one or both of these local areas (the difference time (Δt)) is much less than the time of the long-distance WAN0154.

[0191] This example embodiment shows a network in region A 0100 linked to a network in region B 0120, connected by a dedicated path through a fiber backbone link FBB 0130. FBB 0130 is connected at both ends by endpoint devices (EP) 0102 and EP 0122, which act as anchors at either end 0132 and 0138 of a bridge through FBB 0130.

[0192] The shortest distances are within each of the zones, Local Zone A 0150 and Local Zone B 0152. The longest distances are on the long range WAN segment 0154, and it is on this segment that the greatest improvement can be achieved (e.g., Transport Method - Maximum Distance via Longest Segment - FBB 0130).

[0193] According to the timing of end-to-end transmission of data, the network from the device in area A 0100 to the device in area B 0120 is composed of three blocks of network segments. Local area A 0100 reaches the endpoint 0102 via the communication path 0106 from the device in 0100.

[0194] The long haul between zones is from EP 0102 via communication path 0132 from EP 0102 to FBB 0130 and at the other end via 0138 to EP 0122. Finally, local network segment 0126 is from EP 0122 to devices in zone B 0120.

[0195] In situations such as financial transaction instructions or information transmission, the long-range WAN 0154 segment may represent a significant and major portion of the time required end-to-end 0100 to 0120. Therefore, it is necessary to improve the connectivity of the long-range WAN 0154 segment.

[0196] Figure 2 shows an exemplary problem of long-distance transactions. This figure is based on Figure 1 In the example embodiment, the paths at both ends of the network are extended from EP 02100 and EP 02200 via paths 02302 and 02306.

[0197] At each end of area A where EP 02100 is located and area B where EP 02200 is located, it links the EP to LAN 02102 / 02202, and also shows the entry and exit points (EI P) 02106 / 02206 on each LAN, leading to the Internet 02108 / 02208 at EIP 02106 and EIP 02206 via connection paths such as 02116 and 02216, followed by a standard gateway to the point of presence (POP), leading to the currently widely used Internet method.

[0198] For example, each LAN 02102 / 02202 and Internet zone 02108 / 02208 shown in the figure has various devices connected to those networks. For each LAN, such as LAN 02102, devices are connected to it via 02156 and 02158, respectively, such as server SRV_A02122, client C_A02126, and parallel file system node (PFS) PFS 02128. The Internet, such as Internet 02108, has two servers SRV_A02132 and SRV_A02136 connected to it via communication paths 02162 and 02166.

[0199] In actual usage, there may be the same amount, or more or less devices connected to each network area LAN 02102, Internet 02108, LAN 02202, Internet 02208 than described herein. The amount and type of these connected devices are for illustrative purposes only.

[0200] Other devices may be utilized, such as routers, firewalls, managed switches, servers, and other types of devices that can govern how traffic can flow within and between various networks.

[0201] Figure 3 An exemplary conventional technical implementation in the real world is shown.This example embodiment describes the placement and connection between Slingshot nodes located in three regions, North America, Europe and Asia, some of the world's major financial markets.

[0202] As described in the legend box 03000 in the lower right corner, each financial market described herein is described from a networking perspective as a global node 03008. Near the global node, two rings represent the type of connection quality zone, for example, the radius is measured from the center where the market or exchange is located. This is only for simplification, because many factors determine the size and shape of these zones. However, the two zones can be distinguished from each other because the closest one is the high performance zone 03006 and the other is the best service zone 03002.

[0203] The markets are connected to each other via high performance network links to share pricing information and also facilitate the processing of trade orders and provide trade confirmations of trades and other information from a remote location to another location market.

[0204] The farther a market is from a querying client or server or other type of device, the longer it takes for information to flow, including pricing, volume or other real-time market information, trade requests, completion confirmations of whether an order was filled or not, or other types of data exchanges.

[0205] The areas indicated in this article are:

[0206] ●NYC 03100, New York, New York, USA

[0207] ●LDN 03110, London, England, UK

[0208] ●TOK 03120, Tokyo, Japan

[0209] ●HKG 03130, Hong Kong, China

[0210] There are many other markets around the world that are of interest, but those identified are shown for illustration purposes. A path is also shown between each market, such as 03200 between NYC 03100 and LDN 03110. In reality, there are multiple paths representing submarine cables between two points. This is intended to simplify the examples illustrated herein.

[0211] The lower the latency between points, the faster market information can be known, and the faster trade-related information can be executed and processed.

[0212] Figure 4 Showing the inefficiency of current technology.This figure describes the conventional technique of packetization of data transmitted over an IP network.

[0213] The communication path described in this article is from starting point 04110 to end point 04210.

[0214] LAN_A 04100 connects to WAN 04000 and reaches LAN_B 04200 as a process flow to illustrate packetized information flow. Information is prepared on the device at step 04112 and sent to or via server 04120. This diagram shows the packet flow being sent during IP Send / IP Receive.

[0215] Paths 04322, 04332, 04352, 04522, 04532, 04552, 04622, 04632, 04652 are the transmissions of one of the multiple multi-packet transmission packets. Paths 04326, 04336, 04356, 04526, 04536, 04556, 04626, 04636, 04656 are packet acknowledgments indicating that they were received. The lost packet will not reach the destination device, so this acknowledgment will not be sent back to the originating device. The packet may also reach the destination, but be used to acknowledge that it was lost in transit.

[0216] The maximum size allowed for a single packet payload is equal to the total allowed packet size minus the header size minus the file trailer size.

[0217] To calculate the total payload size:

[0218] Payload size = allowed payload size - header size - file tail size

[0219] The total number of packets required to send a data file of size X is:

[0220]

[0221] Therefore, there is a need to group larger data sizes into packet streams.

[0222] This figure shows an example where packets can be relayed between devices rather than just passed, where the data is broken down into individual packets in more than one stage and then reassembled into a data file once all packets are received at the process, as shown in IP Send Packet 04116 to IP Receive Packet 04118.

[0223] For example, SRV_A 04120 may query a market or other data source for information and cache it locally before copying the information to SRV_B 04220 via IP Send Packet 04006 to IP Receive Packet 04008.

[0224] Continuing along this line, the client can query SRV_B04220 to process information 04212 through IP send packet 04216 to IP receive packet 04218.

[0225] The key point is that grouping data into packets allows for large payloads, which can then be reassembled at the destination or source multiple times along the communication path.

[0226] Figure 5 A GVN topology is shown, including a backbone segment over the Internet or dark fiber. In International Patent Application PCT / US15 / 64242 SYSTEM AND METHOD FOR CONTENTRETRIEVAL FROM REMOTE NETWORK REGIONS, there is a feature where more than one file is aggregated together into a larger file to be sent from one geographic region to another via a "chain cache" through a file transfer. For this feature to be beneficial, the file transfer needs to be as fast as possible. As a bunch of various data payload "files" are transferred, the information slingshot method of the present disclosure moves larger data blocks from one end of the world to the other faster than conventional technology methods.

[0227] You can Figure 5 The present disclosure is utilized within a topology as follows:

[0228] The SRV_BBX 05508 in the zone or region ZD2 05500 may be connected to the SRV_BBX 05608 in the other zone or region ZD3 05600 via a dark fiber connection 05818 through a dark fiber fabric 05808 .

[0229] SRV_BBX 05508 uses the present disclosure to bypass the stack of SRV_BBX 05608 via path 05650 through 05818 and directly write the file to parallel file storage PFS 05620 via remote direct memory access (RDMA).

[0230] SRV_IBX 05608 uses the present disclosure to bypass the stack of SRV_IBX 05508 via path 05550 through 05818 and directly write the file to parallel file storage PFS 05520 via remote direct memory access (RDMA).

[0231] Path 05812 may be IPv4 or some standardized Internet protocol over which traffic flows from SRV_AP 05510 to and / or from SRV_AP 05610 via Path 05812 at the top of GVN 05802 via a tunnel or other type of communication path. This embodiment illustrates that various types of network fabrics may be combined into a larger network tapestry. These fabrics may be seamlessly woven together as described in U.S. Provisional Patent Application No. 62 / 174,394. The information slingshot may be a standalone approach or may be integrated as a network segment within a larger network path comprised of network segments.

[0232] This example embodiment shows the topology of a global virtual network (GVN), its various devices, communication paths and other embodiments. It shows how to connect various geographic areas or regions or landmasses together on various types of paths.

[0233] legend:

[0234] ZL0-LAN zone 0, ZL1-LAN zone 1, ZI0-Internet zone 0

[0235] ZI 1-Internet Zone 1, ZI2-Internet Zone 2, ZD3-Internet Zone 3, ZD 2-Internet Data Center 2, ZD3-Internet Data Center Zone 3

[0236] Figure 6 Conventional technology is shown - the file is sent as a multi-part transmission via packetized transmission over TCP / IP. Figure 4 An extension to IP Multipart File Transfer that enables it to be sent via IP Multipart File Transfer when the data payload contains a structured data file that is larger than the payload size allowed.

[0237] In this case, additional header and possibly more trailer information is needed to indicate which piece of the multi-part file is contained in the payload segment of the packet. When the file is "saved", it is saved to a hierarchical file system (HFS) 06120, 06320, and the send is to read the saved local file from this HFS and packetize the multi-part file transfer in as many packets as are needed to send the file 06200. When the file is received, the process receive file 06220 will read each packet, confirm that they are in order, and will then take each part of the multi-part transfer and stitch the file back together part by part.

[0238] The reassembled file is saved in the HFS 06320 in the host device 06310.

[0239] Figure 7 Shown is a topology on a tapestry - HPC Power. This example embodiment describes two EPDs 07100 and 07200 connected to a network topology, such as SRV 07300 within a Global Virtual Network (GVN) 07000.

[0240] This is a simplification of the network, but is designed to illustrate how PFS can distribute storage between four regions.

[0241] Each communication path in the tapestry can run various fabric protocols. A PFS, such as file storage on PFS 07810, PFS07812 inside LAN 07120 or LAN 07812, or PFS 07800 or PFS 07802 in the cloud, can be accessed via Remote Direct Memory Access (RDMA), through protocols such as InfiniBand (IB), or other protocols that allow local RDMA access to distributed nodes in a Parallel File System (PFS). The key point is that the foundation of High Performance Computing (HPC) can be achieved through the topology enabled by the GVN.

[0242] Figure 8 An exemplary workflow for saving a file to a PFS over a long distance is shown. This figure describes saving a file 08100 from one device 08000 to a PFS 08200 in another location via RDMA, followed by retrieval of the file from the PFS 08200 by another device 08300.

[0243] The steps in the flow from start 08000 may be a device generating a file, delivering a file, pulling a file or otherwise processing a file, such as file 08100. Path 08120 describes saving a file directly to a PFS device via RDMA, writing directly to a storage node device from one region to another.

[0244] Retrieving files from queue 08300 at PFS 08200 step via path 08220 may be local or remote relative to the region where the PFS device is located.

[0245] Ending 08600 means accessing the file, such as reading, loading, or otherwise using the file.

[0246] Fig. 9 Traditional client server REQ RESP and backend DB replication (multi-master) are shown. This example embodiment describes the current IP request / response framework via REQ1 09500 and RESP2 09502 paths.

[0247] Files saved by SRV_APP 1 09020 on a local HFS such as HFS 09040, and by SRV_APP 2 09220 on HFS 09240. SRV_APP may represent an application server or other type of device and is labeled for illustration purposes. WAN 09100 may be a LAN / Internet / dedicated line or other type of network connection.

[0248] MMDBR# refers to multiple master database copies. In such a deployment, the database table and field structure definitions are equivalent and identical. Each host has a starting number on the row, automatically incremented by an offset value equal to the number of hosts. For example, in the case of only two databases attached to the master host, DB_1 would have row numbers 1, 3, 5, etc. DB_2 would have ID row numbers 0, 2, 4, 6, 8, 10, etc. Another anti-collision method can also be used instead.

[0249] Database replication can be automated or manual or other process where the contents of each DB_1 09050 and DB_2 09250 are copied to the other. To protect data from being overwritten, row offsets accomplish this protection by ensuring that there are no identical row IDs between the two ends.

[0250] Fig.10 Shown is a slingshot of information via RDMA - pointing to Db. This example embodiment takes a approach to try and speed up communications. This could be RDMA sending the entire database table and contents directly to the storage medium for the database in the remote region. This is good for table structures and all data rows of a database that are static, unused or otherwise locked. This approach can be used when copying to an offline slave machine and then bringing it (back) online.

[0251] However, this approach is not efficient for a single row or a small number of rows and may cause synchronization problems.

[0252] Therefore, prior to this approach, a lock on the database during transmission was required at MMDBR1 10510 and MMDBR2 10512 to transfer messages between DBs 10050 and 10250 to ensure they were synchronized. This added undesirable drag to the system.

[0253] This example may not be an ideal utilization of the slingshot.

[0254] Fig.11 Shown is the information slingshot via RDMA - queuing files. The Backbone Switch Server (SRV_BBX) inculcates the property of excellent connectivity over large pipes and has strong communication capabilities.

[0255] This diagram shows that SRV_BBX 1 11080 writes files directly to PFS 11802 via RDMA 1 11600 , and SRV_IBX 2 12280 cross-publishes different files to PFS 11800 via RDMA 2 11610 .

[0256] The data payload is efficiently transmitted. A transmission record is kept on each side, indicating which files are received. Multiple master DB replications have DB_1 11050 using odd rows and DB_2 11250 using even rows. In this example, there is a reliance on the database replication mechanism to track multi-host replication events. Therefore, synchronization problems may ensue. There is also the possibility of a significant lag between the time when the device can perform a remote write and obtain a long delay in its own confirmation that may be significantly harmful. Therefore, there is a speed difference and a reliability gap that needs to be overcome between coupling ultra-fast RDMA writes and traditional database information replication.

[0257] This example embodiment does not illustrate an ideal implementation of Slingshot. Saving to a remote PFS device is efficient, but making the SRV_APP in the target region aware of the file is not fast or reliable enough.

[0258] Fig.12 Information slingshot via RDMA - independence of queued files, SRV_ACC doing cross-DB verification. This example diagram illustrates the need for complete decoupling of database records in an active sense to and from. In this example, SRV_BBX, for example SRV_BBX 1 12080 will send a file to PFS 12802, the record will be stored on SRV DB 1 12030, into the local database DB_1 12050. It will leave at that time.

[0259] Because of the reliability of RDMA, once a file is saved to a PFS storage node, it is believed to be there. Account servers (SRV_ACC), such as SRV_ACC_1 12060 and SRV_ACC 2 12260 communicate with each other to catalog the received transfers. This is also not the ideal approach for Slingshot and points to the need to decouple the Slingshot approach from the current approach taken over IP and embrace the capabilities of Slingshot.

[0260] Fig.13 An exemplary use case for the Information Slingshot is shown - market information relay from the market to the trading desk. As an example of a slingshot, in the financial industry, market information such as pricing and volume can be requested and received via TCP / IP, or it can be sent from source to destination by UDP / IP multicast. TCP / IP requests and retrievals require round-trip RTT time. UDP / IP transmission is only one-way for useful information (acknowledgements are not as important to the sender), however, there is still significant drag due to the inefficiencies of IP and packetized transmission.

[0261] Using current packetized IP technology, even UDP / IP multicast is not the most efficient.

[0262] This diagram contains an example embodiment that describes a geographic area where device DO 13100 in region A 13000 collects information and then flows forward to transmit the information to devices in region B 13200.

[0263] The step of collecting field of view information 13120 to prepare file content 13130 ​​for ejecting the file to remote PFS 13180 is on DO 13100. Ejection is via 13882 to a parallel file system (PFS) node PFS 13880 on storage device S2 13800. RDMMA over IB 13882 or an equivalent network type has numerous advantages over transferring files via TCP / IP or UDP / IP over Ethernet, primarily because the file does not need to be packetized, but also because of how packetization is handled and the efficiency of RDMA over IB.

[0264] Files stored on PFS 13880 are retrieved on device D2 13300 by a process that retrieves files one by one from PFS 13380 via path 13382. A single file or a batch of multiple files may be retrieved simultaneously.

[0265] Subsequent steps 13332 to 13325 to 13320 describe using the file in the remote region from the origin of the file. It is no longer necessary for the devices DO 13100 and D2 13300 to communicate with each other about the scheduling or transmission or use of the file itself.

[0266] Fig.14An exemplary use case of the information slingshot is shown - a trade execution request from a client source to the market. This example diagram also shows the slingshot in the context of a financial industry use case, demonstrating the simple efficiency of placing a trade order from device D2 14300 and ultimately being used by device D1 14100. It starts at 14310 and ends at 14100. It is Fig.13 The reverse path of the path described in .

[0267] The key point is that both devices work independently of each other. There is only a communication event when D2 14300 wishes to give instructions to execute a transaction on D1 14100, but it is not from D2 14300 directly to D1 14100, but D2 14300 saves one or more data files on the parallel file system (PFS) 14000. Once the file is written, D2 14300 is able to move on to other things.

[0268] The D1 14100 has automated processes that run at time intervals. Fig.24 and 25 Knowing this interval, this automated process will look at the PFS 14880 to check if 14180 has any new unmarked (not marked as read) files to be retrieved and processed by it.

[0269] This approach is very efficient for trading orders, such as trade execution requests. Device D1 may be as close to the market as possible. Device D2 may be a client terminal or a server very close to the client accessing it. The key point is that the trade payload can be sent as close to line speed as possible.

[0270] This is advantageous because large file sizes can be sent, and this mechanism can be used to send / receive ever-growing height (data size) blockchains, algo-bots (intelligent algorithms to be executed remotely), and other payloads larger than the packet size.

[0271] Fig.15 shows an exemplary use case of the Information Slingshot - trading results from the market back to the client. This example is very similar to Fig.13 and 14 , because devices D1 15100 and D2 15300 work independently. The communication between them is via a file 15882 published by one device 15100 to the PFS 15880, and the retrieval of this file by a process running 15380 on the other device 15300. They communicate, but all bloat is cut from the exchange.

[0272] The difference in this figure is the example embodiment confirmation from the marketplace 15120 and the trade execution notification 15320.

[0273] D1 15100 and D2 15300 may all occur within the same device or via distributed operations between different devices in the areas represented by D1 15100 and D2 15300.

[0274] Fig.16 An exemplary workflow is shown of information being written to a file, then read, processed, etc. This diagram simplifies Slingshot into a linear process flow and is general in nature with no specific reference to finance or any other industry.

[0275] This diagram begins the process at start 16000, reaching the information 16100 step via path 16010 to evaluate and prepare information 16200 via 16110, either as a copy of a file or to construct the information into a file specifically designed for remote storage via RDMA to PFS 16300 via 16210. This saved file will be retrieved from the queue retrieve file 16500 on the PFS by a process via path 16310. The file content will be parsed by step parse file content 16600 via path 16510. End step 16900 reached via path 16610 implies that the file can be used. The key point is that this process can be simplified into a few key steps.

[0276] Fig.17 shows an exemplary workflow for writing information to a file. This diagram is based on Fig.16 . It further simplifies the process by only showing the stages from information 17100 to the time when the file is saved in PFS17300.

[0277] Fig.18 Shows an exemplary workflow for reading files and processing them. This figure is also based on Fig.16 It further simplifies the flow, but this time only shows the stage starting from when a file on PFS 18300 is retrieved by the process querying the file 1850 in the queue of PFS. The file content will be parsed by step parsing file 18600 and ends at end 18900.

[0278] Fig.19 An exemplary embodiment of the information slingshot is shown with respect to file header / file segments: header, body data, file trailer. This exemplary embodiment of the information slingshot describes a data file organized in three defined segments; header information 19100, payload 19200 containing body data, and file trailer 19300.

[0279] This file may be stored in RAM, memory, saved to disk, or otherwise stored in another form of memory or storage on the device or other device.

[0280] Packet header 19100 may contain information about the host origin, host destination, timestamp, and other information.

[0281] Security information may be stored in fields in both the header 19100 and the trailer 19300. This security information may hold a reference to a key used to decrypt the payload 19200 and other information. The payload (body data) may be sent fully or partially encrypted or unencrypted.

[0282] The payload checksum in the file trailer is used to verify the integrity of the body data.

[0283] Other features may take advantage of the flexibility provided.

[0284] A significant advantage is that the payload can have the algorithm sent as part of the payload.

[0285] The EOF marker in the end of the file 19300 will indicate that the file has arrived, is complete and ready to be verified / checked for accuracy and then ultimately used.

[0286] In the context of the financial industry, the remotely executed algorithm in payload 19200 may, for example, contain exit conditions that are instructed to take a predetermined action if market conditions change, using algorithmic instructions to assess the direction of market change and then modifying the instructions accordingly to:

[0287] -Cancel

[0288] -reverse

[0289] -strengthen

[0290] - or by changing the instructions in some other way

[0291] Fig. 20 An exemplary embodiment of an information slingshot is shown with respect to ejecting, storing in a PFS, and processing files from a repository. Figure 8 , Figures 12 to 19 The principles established and illustrated in . Adding a file queue manager (FQM) 20810 manages the intervals at which files are pulled in batches, checks the integrity of files, and performs other file-related actions.

[0292] The connection of the FQM to the file 20888 on the PFS 20880 is via path FQM 20386 , and the coordination between the FQM and the retrieval of the file from the queue on the PFS 20380 is via path FQM 20388 .

[0293] This figure also shows each segment of file 20850 in detail, header 20852, payload 20856, and file trailer 20858 to show example fields. In header 20862, payload 20866, file trailer 20868.

[0294] These example fields are shown as key=value pairs of information and are for illustration purposes only. The data may be stored in another data structure and the fields may be different than depicted in this figure.

[0295] Fig.21 An exemplary embodiment of an information slingshot is shown relative to harvesting a file queue to collect subject data for analysis and use. This figure starts at 21000 and shows the file queue manager 21100 pulling a batch of files from the PFS 21800 into the RAM 21900. The integrity of the files pulled by the collection process 21120 is checked 21300. The complete files 21802, 21804, 21806, 21808 that have arrived are listed in 21380, and the incomplete file 21814 is listed in 21340. This is where the process flow of this figure ends at 21990. The implication is that incomplete files 21340 such as file 21814 will be picked up by subsequent processes when completed.

[0296] Fig. 22 An exemplary embodiment of an information slingshot relative to file body data and EOF is shown.

[0297] This picture is Fig. 20 and 21 An algorithmic expression of what is described in .

[0298] The decision gate checks to see if each file is complete 22300, implying that it will look at each file pulled from the PFS 22800 by the file queue manager 22100. If the file 22802, 22804, 22806, 22807 is complete, it will be processed via the path is BFP 22510 and eventually used. If it is not complete 22410, then the file will be ignored 22440 and its processing will end at the end 22990 via BFP 22448.

[0299] End 22990 indicates that the operation of file queue manager 22100 is over.

[0300] Fig.23 An exemplary embodiment of an information slingshot relative to a file process flow is shown. Fig. 22The subsequent description of what happens in 23590 adds elements such as each file on the file list 23590 is pulled via file routing FR 23620 and marked to the synchronization process processing file 23610, processing file 23612 via file routing FR 23622, processing file 23616 via file routing FR 23626, and processing file 23618 via file routing FR 23628. There are also two different endpoints - END 23990 is where the complete file is processed via file paths FP 23630, 23632, 23636, 23638, and the read "flag" indicates that the file has been read, it is complete and used. These files will not be pulled by the subsequent batch file pull 23120.

[0301] The end 23940 implies that the file has been ignored because it is incomplete, and 23400 indicates that it will be read again during a subsequent file batch pull from PFS 23800.

[0302] Fig.24 Illustrates potential issues with file pull queues such as replication processing. This diagram depicts pulling file batches from PFS 24800 over a period of time from start 24000 to end 24990. Time is depicted on two axes, with time intervals at the top and batches along the left.

[0303] Time interval A 24500 and batch pull A 24600 occur at the same time. Use 24602 occurs at the end of this time interval. Time interval B 24510 and batch pull B 24610 occur at the same time. Use 24612 occurs at the end of this time interval. Time interval C 24520 and batch pull C 24620 occur at the same time. Use 24622 occurs at the end of this time interval. Having one time interval start right after another ends causes a problem because there may not be enough time to mark a file as read even though it has been read by a previous batch.

[0304] For example, file 25802 is pulled by both batch pull A 24600 and batch pull B 24610. This also happens again with file 25808, which is read by batch pull B 24610 and also pulled by batch pull C 24620.

[0305] This is a very dangerous flaw, because, for example, a transaction request being executed twice could result in significant financial or other damage due to unintended consequences.

[0306] Marking as read may involve setting a flag on the file record on the storage device, or moving the file to a different folder, or deleting it from the storage device after the move, or other ways of "marking" it so that subsequent batches know that the file has been pulled, read and "used" by a previous batch.

[0307] Fig.25 An exemplary embodiment of synchronization that performs a batch pull of only files that are not marked as being read during a moment is shown. This figure is similar in all respects to Fig.24 Same, but Fig.25 Included is a delay A25505 between interval A 25500 and interval B 25510 (as well as a delay A25605 between batch pull A 25600 and batch pull B 25610).

[0308] There is also a delay B 25515 between interval B 25510 and interval C 21520 (as well as 25615 between batch pull B 25610 and batch pull C 25620).

[0309] The key point is that this delay allows the current batch to evaluate and process all files pulled by the processor, and if it utilizes complete files, mark those files as read. This delay added to the mechanism is completely dynamic and can be extended if the batch requires more processing time, or shortened to complete the batch. The interval time can also be dynamically adjusted based on several factors. The instant granularity description Δt=P+Q+R. t=the difference time from the start of the instant to the end of the instant, P=the time in the cycle for the batch processing of items for this instant. Q=the time used for post-batch calculations. R=a delay time to ensure that there is no overlap between batch items and / or to set an offset between instants.

[0310] Fig.26 An exemplary workflow of a late critical backend process (to write to remote) is shown. This figure describes the algorithmic logic of the write process to the PFS 26860 in the remote region. It begins at start 26000 to prepare for execution 26100 via path CP 26010. The flow then reaches a join point 26120 via path CP 26110. At join point 26120, the flow forks into two streams via paths CP 26130 and CP 26150. These streams run concurrently with each other and are independent.

[0311] Path CP 26150 is from the binding point 26120 to the using process, such as writing a file to the remote PFS 26860. The writing is done by writing a file to the remote PFS 26300 via path CP 26810.

[0312] Another path CP 26130 is for database record keeping and logging. Db records are saved locally in DB_26250 via the process Save in DB_26200. Log record backups or other related files are stored locally in HFS26240 via the process Save in Log 26220. Remote writing ends at end 26998. The management process of database and log saving ends at end 26990.

[0313] Fig. 27 An exemplary workflow for a post-critical (to read from PFS and process) backend process is shown. This example embodiment describes an algorithm that governs batch pulls 27100 and reads files from PFS 27180 for local processing. When a file is read, its integrity is checked 27200. If it is indeed complete, it is marked as pulled 27188 so that the next batch of processes 27100 will not read it repeatedly. See Fig.25 To understand how it can be solved Fig.24 The problem of pulling and copying files described in.

[0314] In parallel and simultaneously with the processing, a comment record is written to the log on HFS 27650 via the log record saved in DB_27320 to Db_1 27350 and saved in the log 27600.

[0315] A file can be marked as read and intact 27188 only if the calculated checksum of the file payload at 27200 is equal to the checksum embedded in the end of the file.

[0316] Using this file 27220 implies that something will be done with the information or instructions or other data sent within the file structure. If market pricing information is displayed or processed to the algorithm for processing. If a trading instruction is issued, then 27220 this will be fed to the market for execution. The response can then be written remotely via the same method. This can be used in the financial world or other applications. A key point is that each batch of files in the list of file processing cycles is pulled as a list at the beginning 27100. Next pull 27100 each file and check its integrity 27200. If the file is complete, "yes" 27222, then use it 27220. The flow forks after this step. Mark file CP 27188 marks the file as read on PFS27180. Path CP 27302 leads to join point 27300. Here, the path forks to CP27322, resulting in saving in database 27350 and saving logs on HFS27650. Another path CP 27312 is to use the file.

[0317] For the sake of simplicity of this presentation, the file pull 27110 is described as a loop pull. The step CP 27102 between pulling the file list from the PFS 27100 and pulling the file from the PFS 27110 can also result in parallel and simultaneous file pulls, each running simultaneously after this process.

[0318] There are other variations and possibilities, and one skilled in the art will be able to make slight changes to this algorithm to achieve a preserved result.

[0319] The key point is that batches of files can be pulled, each file can be checked individually for completeness, and if it is incomplete 27242, the file is not used 27240, but it is still saved 27320 in DB_27350 and recorded 27600 on HFS27650. If the file is complete 27222, it is used 27220, marked 27188, processed 27512, and a record of its pulling, checking, use and other information is saved in database 27350, and the file is also recorded on HFS27650.

[0320] Fig.28 An exemplary system diagram for the Information Slingshot is shown. This diagram depicts various modules running on various devices that can utilize the Information Slingshot described herein. Devices EPD 28000, SRV_AP 28100, SRV_CNTRL 28200, and SRV_BBX 28300 and a junction 28880 between PFS devices 28800, 28810, 28820, and 28830.

[0321] The SRV_BBX servers may also be connected to each other via paths 28738, such as SRV_BBX 28300 and SRV_BBX 28732. This diagram also includes a launch node module 28388 on the SRV_BBX 28300. This launch node may reside on the SRV_BBX 28300, or the launch node may be a separate device / server that works in conjunction with the SRV_BBX 28300.

[0322] Fig.29 A structured combination of layered network fabrics into a network tapestry is shown. This diagram represents various network fabrics illustrated as independent clouds layered at different levels. OTT1 represents the first level on top of the LAN 29002 or Internet 29000 or slingshot mechanism 29020. At OTT1 there is a network type such as GVN 29008.

[0323] OTT2 represents the top second level, these modules are built on top of the GVN 29008, such as the OTT2GVN end-to-end fabric 29010 or the OTT2MPFWM multi-parameter firewall mechanism 29012. OTT1 is the slingshot layer 29020 "below the Internet" or parallel to the Internet 29000. OTT2 is where the slingshot modules 29030 are located, such as bouncing, catapult routing, information beacons and other related technologies.

[0324] At each boundary between one network fabric and another is an entry / exit point (EIP). An EIP such as EIP 29052 implies a bridge between the LAN and the GVN, typically via an endpoint device (EPD) at the edge of the LAN 29002 that builds a secure tunnel with an access point server (SRV_AP) in the GVN 29008. EIP 29058 between the GVN 29008 and the Internet 29000 implies an EIP on the SRV_AP. EIPs between other layers may represent other topologies and involve various other devices.

[0325] These various network fabrics, when linked together by EIPs, form the network tapestry.

[0326] Fig.30 The various network fabrics form sequential layers of a tapestry. This diagram involves Fig.29 The added elements are routing+execution 30115, construction 30225, slingshot 30715, and ejection logic 30815.

[0327] Also new are the GVN Level 2 - Logic 30050, GVN Level 4 - Logic 30150, GVN Sub Level 1 - Logic 30650, and GVN Sub Level 2 - Logic 30750. These logic layers are where support for the EIPs between layers is provided with specific settings, operations, routines, and other methods.

[0328] Fig.31 Conventional technology - CRONS is shown. This example embodiment describes the conventional technology of CRON scheduling tasks. The script / program called 31200 is controlled by the CRON manager CRON manager 31100 process.

[0329] CRON jobs can be set up by manually adding items to the CRON list 31108, or this list can be managed by the software manager layer. There are some issues associated with knowing if this process is running properly. Only one process is running as expected, or there are many resulting in unexpected parallel instances running. The environment project 31318 needs to be up to date and working within expected parameters. The base project 31680 needs to run optimally and within expected parameters.

[0330] The system time / internal clock 31600 must be up to date 31360 and accurate to a very fine time granularity. This update regularity needs to be more frequent than the drift that the system is prone to. The hardware and associated drivers 31620 have some dependencies on available resources.

[0331] Fig.32 Conventional techniques - loops are shown. This example embodiment describes conventional techniques for loops within a script, which are logically governed by a conditional or counting factor 32150. It gives a code structure for a conditional and / or counting loop 32200. When the file is called by a trigger at the start 32000. In this example, the preparation code is run between example lines 004 and 009. In the case of a counter loop, the conditional loop will run while the loop condition 32200 evaluates that the current loop cycle counter $ii is less than the number of times this loop should run. At line 27 of the example script, the counter or (external) condition 32150 will be incremented by one via $ii++ or some equivalent. As long as the condition remains true 32220, this loop will continue to run.

[0332] If the counter $ii is equal to or greater than the number of times the loop should be executed, the loop ends and processing continues. The code to be executed after the loop is between lines 033 and 037. The script / routine / software ends at line 038 and its execution ends at END 32990. There may also be problems when the script is called once from the outside and loops a large number of times predetermined by a counter or condition, for example:

[0333] What happens if / when the script crashes? How can it be recovered?

[0334] ●What if the script runs longer per cycle?

[0335] ●What if the system's base clock drifts too fast or too slow?

[0336] ●What if the script forks and sees more than the expected number of simultaneous instances of itself?

[0337] ●How to fine-tune the instantaneous granularity to ms or even μs.

[0338] Fig.33 shows a conventional technique - recursion - where a script calls itself. This example embodiment is similar to Fig.32 , the main difference is that it is simpler. It does not contain loops within its code, but instead has the command call itself in a constant runtime loop.

[0339] It begins at Start 33000. Code 33100 is listed from lines 001 to 020. Code that is ready to execute is between lines 002 and 004. Code to be run during that one instance is between lines 005 and 014. Post-execution code to be run is between lines 015 and 018. At line 020 of the figure, the script calls itself, for example, it opens a shell CLI> to restart the script 33200. If there is a conditional end to the running of code (logic not shown), then at line 016, the code information can be saved 33800. The script ends at END 33990.

[0340] If there is no condition or logic to end the script run loop, then it will loop indefinitely.

[0341] Fig.34 shows the problem of looping. Fig.34 based on Fig.33 , the main differentiating factors are the issues marked 344##. These issues are as follows:

[0342] 34400 cannot start

[0343] Script 4-100 may not initially run due to a problem with allowing calls to system users or other issues. This may or may not be obvious. Software changes or software updates, missing include files or other factors may prevent running.

[0344] 33402 Too many instances

[0345] If for some reason, script 4-100 is called to run more than once, more than one instance of the script will be running. This is problematic because each instance of the script will call itself and keep running, consuming system resources. There is also the danger of conflicts due to multiple scripts trying to handle or deal with the same subject matter, causing unexpected and potentially destructive or disruptive results. More is not always better.

[0346] 33404 Uncaught exception / crash

[0347] If the script runs and crashes, unless this exception is caught, the next loop will not be triggered and the functionality of the script will not be realized.

[0348] 33406 Execution time > instantaneous allowed time

[0349] If the script takes longer than the time allotted for a moment, the next moment will be delayed too far and will go off schedule.

[0350] 33408 Corrupted status information

[0351] If the state that the script relies on for conditional looping is destroyed, unexpected results may occur.

[0352] 33410 Unable to restart

[0353] If the script is running but cannot restart itself because the process running it has different permissions than the user or process that originally called the script, then it will only run once and will not continue to run.

[0354] 33412 Corrupted script files

[0355] If the script file is corrupted on the storage medium (physical or virtual in a storage device), the next time the CLI calls it to restart the script 4-200, it will crash or may run and produce unexpected results.

[0356] 33414Unknown problem

[0357] There may be other unknown or undetectable problems going on. The launcher that calls the script may think it is running, but the fact is that it is not.

[0358] The above issues are included to highlight potential issues. Not all or just some of these issues may occur at any one time. There may be more issues not described.

[0359] Fig.35 It shows that the difference time = P + Q + R. This figure describes the life cycle of an instant. The time granularity of an instant can be defined as follows.

[0360] Instantaneous granularity

[0361] Δt=P+Q+R

[0362] Δt = the difference in time from the moment 35100 to the moment 35100

[0363] ●Start based on the completion of the previous moment

[0364] or

[0365] ●Start at the scheduled time

[0366] P = the time taken for this instantaneous batch processing item in the cycle.

[0367] Q = time used for post-batch calculations.

[0368] R = Delay time to ensure there is no overlap between batch items and / or to set an offset between moments.

[0369] The period 35000 is the difference time or Δt of the instant duration.

[0370] Fig.36 shows the timing of various cycles, where X≠Y. This diagram is based on Fig.35 , which describes two instants with variable and different times.

[0371] ΔT=x 36100 and ΔT=y 36200 are not equal to each other, as described by x≠y 36010.

[0372] In this example, the delays R 36130 and R 36230 are not equal to each other. The processing times P 36110 and P 36210 do not equally reflect the different amounts of work required for each run. The post-processing times Q 36120 and Q 36220 also differ based on the amount of work to be completed.

[0373] Therefore, the duration of each loop is based on how many items are being processed.

[0374] Fig.37 shows the timing of various cycles where X is consistent. This diagram is based on Fig.35 , when it describes two instantaneous cycles, unlike Fig.36 In that way, the duration of these cycles is equal, x0 = x2 37010. Although Δt = x0 37100 and Δt = x2 37200 are equal, the processing times of each instant P 37100 and P 37210 can either be equal to each other or one can be longer than the other. Each is still based on the amount of work that must be processed. The post-processing times Q 37120 and Q 37220 can also be different from each other or equal, respectively.

[0375] The instants are maintained at equal Δt duration 37010 in at least two ways:

[0376] • By limiting the amount of items to be processed the instant P (P 37110 or P 37210) will never force Δt to exceed its allotted time budget.

[0377] Delays R 37130 and R 37230 are dynamically adjusted to ensure that the delay is long enough before the next cycle runs. The delay can be a sleep timer, or it can be no action, but there must be some gap between the end of the current instant cycle Q and the beginning of the next instant cycle P.

[0378] This fixed time is possible if there is a limit on the number of items to be processed based on how much time each P and Q takes.

[0379] Fig.38The distance thresholds for achieving gain from Slingshot are shown. This figure highlights the algorithmic decision logic that determines whether it is faster and / or more efficient to send packets via Internet path IP 38562 to 38588 or via Slingshot enhanced Internet path 38662 to 38688 (with Slingshot transparent segments 38800 and 38880 between a pair (or more than a pair) of IP addresses 38668 and 36682).

[0380] It does not take into account reliability or loss or related congestion or other issues beyond latency that may have an impact on the IP path.

[0381] Using a slingshot of Δt at each bridgehead 38610 and 38620 of the slingshot transparent jump section 38600 is inefficient compared to the gain 38370 achieved via the use of the slingshot.

[0382] The minimum distance is required to compensate for the necessary processing time of the slingshot mechanism at 38615 and 38625.

[0383] The metric latency of the IP 38100 is measured using the following equation

[0384] Total Internet path time Δt ms = TX 38510 + TW 38500 + TY 34590.

[0385] The metric latency of the SL 38200 is measured using the following equation:

[0386] Total Internet-Slingshot hybrid path time Δt =

[0387] TX 38630+TV 38610+TZ 38600+TV 38620+TY 38640

[0388] Latency comparison between IP and SL using equation

[0389] Δt{TV+TZ+TV}<Δt{TW ms}

[0390] If it is true that the slingshot enhanced path (Δt{TV+TZ+TV}) has lower delay, SL is lower 38370 than the internal path, then “SL is the most efficient transmission, using” 38380 is the optimal path.

[0391] If false and the Internet (Δt{TW ms}) is faster than the slingshot, "IP is lower" 38320, then "IP is the most efficient transmission, using" 38360 is the optimal path.

[0392] The importance of precise timing is not only to receive and process files without duplication, but also to be delay bound, i.e. to process files with minimal delay.

[0393] Fig.39 shows the synchronization of batches relative to the marker file. This figure is based on the above Fig.25 . The use of files is illustrated for use 39602 of files in batch pull A 39600, use 39612 of files in batch pull B 39610, and use 39622 of files in batch pull C 39620. Use of files (batch A) 39702 sends files file 39800 and file 39802 in parallel and ends at batch stop 39902. Use of files (batch B) 39712 sends files file 39804 and file 39806 and file 39808 in parallel and ends at batch stop 39912. Use of files (batch C) 39722 sends files file 39810 and file 39812 and file 39814 in parallel and ends at batch stop 39922. The key point is that although the files are pulled in batches, the pulling of available files is parallel, and therefore the use is also parallel. Files that are not ready in batch pull A 39600, such as file 39804 and file 39806, will not be pulled and used until they are available. Files 39804 and 39806 will be pulled along with file 39808 in batch pull B 39610.

[0394] The ability to pull and process batches in parallel allows the Slingshot mechanism to be massively scaled.

[0395] Fig.40 The instant granularity is shown. This figure shows two instants 40100 and 40200. This example shows two batch file pulls on the backbone exchange server (SRV_BBX) using read queue + process 40510 and read queue + process 40610. They both pull files from the same storage medium parallel file system PFS incoming file 40800. The files pulled in 40510 via path 40512 are processed and then marked in post-processing post P 40520 via path 40522.

[0396] This is a key important point because the next batch of file pull read queue + process 40610 from PFS incoming files 40800 via path 40612 should only include unmarked files or files that were not filled by the previous batch. Then, in post P40620, the pulled and used files are marked via path 40622 so that they will not be disadvantageously pulled by subsequent batch file pulls.

[0397] Fig.41 The instantaneous granularity is shown relative to a variable time interval. This diagram is similar to Fig.40,However, it shows the processing of variable amount files and demonstrates how this affects the processing and post-processing times, as well as indicating the minimum delay before the next cycle should be executed.

[0398] Three periods are indicated here: 41100, 41200 and 41300, each period containing a different corresponding quantity, namely the quantity 58 in 41810, the quantity 22 in 41830 and the quantity 8 in 48860.

[0399] Fig.42 An exemplary algorithm is shown with respect to moment adjustment. This diagram describes the algorithm for different types of moments. It begins at start 42000 and ends at end 42990.

[0400] At join point 42020 , two parallel processes are executed to collect momentary parameters 42030 derived from momentary settings 42050 , check system monitoring 42100 derived from system monitoring data 42150 , data accumulated from system monitoring data 42160 source.

[0401] If everything is ready at the moment, is everything OK? 42140, then the process will continue along path yes 42142. If not ready, the process will proceed to error node 42200 via path no 42144 leading to error response 42400.

[0402] The two types of instants are variable or fixed instants? 42170. Variable time instants 42176 start at script start 42300. Fixed time instants 42172 start at script start 42500 using information from using instant parameters 42175. There are additional logic gates remaining instant time? 42500 to ensure that no time is overrun. Common instants used by fixed and variable processes to run completion processes include preparing for script end 42600 and reporting 42680.

[0403] Fig.43 Artificial intelligence analysis, granularity and cycles are shown. This diagram shows the analysis and interpretation of instantaneous cycle time periods. Artificial intelligence can assist in future calculations to estimate how long the instantaneous processing and post-processing time will take based on the volume of items to be processed, taking into account system resources and other factors.

[0404] Metrics calculated for each period may be plotted on a standard deviation curve based on various periodic trends, with data from current period 43100, short period 43200, and long period 43300 historical data. Annotations of low values, high values, average values, and other analyses may indicate whether current performance is as expected or better, worse, or otherwise different from prior experience.

[0405] This data collection and contextual analysis consists of artificial intelligence (AI) algorithms in decision making. The current period 43100 can be compared with short-term 43200 and long-term 43300 performance data.

[0406] Fig.44 Exemplary components of the instantaneous component are shown. Three devices are shown in this figure. That is, a central control server (SRV_CNTRL) 44200, a general device 44100, and a time server (SRV_TIME) 44600. The device can be any type of endpoint device (EPD), endpoint hub (EPH), access point server (SRV_AP) or other type of device.

[0407] On the central control server (SRV_CNTRL) 44200, system information is stored in a database (Db) 44295 of a central information repository 44295. This information can be used by managers, monitors and other system components associated with the moment. Fig.48 , where some items related to the moment are shown.

[0408] To update the time of its internal clock, the device 44100 may query the NTP server listener 44650 of SRV_TIME 44600 via an API call, or another request-response framework via path REQ 44660, with a request and RESP 44680 obtained in response. Database Db 44155 on the device 44100 and database Db44655 on SRV_TIME 44600 may store information about time requests, time drift of the device, another other instant, time-related information, and other information. The device 44100 stores information about time in its local database Db44155, and also reports its time-related information from its system and time manager 44120 to SRV_CNTRL 44200 via REQ-RESP issued to SRV_CNTRL 44200, via REQ44600 with an acknowledgement returned via RESP 44620.

[0409] The NTP time module 44250 on SRV_CNTRL44200 analyzes information related to the system time.

[0410] Fig.45An exemplary use of moments in beacon flashing is shown. This diagram illustrates the importance of utilizing moments in real-world applications, data beacons. All pulses of information begin at the same starting point 45200, the backbone server (SRV_BB) 45000, and radiate outward. This diagram shows 10 pulses from the starting point to the farthest point, where information files are written to a remote parallel file system (PFS) storage device PFS 45800 for retrieval and use by SRV_BB 45900 in a remote area. SRV_BB can also be a catapult node or other device.

[0411] The pulses are 45300 to 45318 and the duration between each pulse is labeled ts00 45100 to ts18 45118. These pulses should occur at defined intervals so that each flash can be relied upon. These can carry simple to complex data. The key point is that the beacon is reliable.

[0412] The combined timing of multiple beacon flashes is called the pulse rate. The frequency between flashes is Δt. As for tolerance, this is related to the need for delay and information currency. How current the information is when received is a function of the delay. The freshness of the information is a function of the delay plus the delay between pulses.

[0413] For example, if the delay is 50ms from 45000 to 45800, the pulse frequency and the instant granularity is every 1ms, then the information is always fresh. However, if the frequency is 3ms between pulses, and the instant granularity is 1ms, then it is possible that the information may be stale by 2ms.

[0414] The freshness of the information assumes that the pulses are sent at the best possible line rate and are sent and received in the most efficient way possible.

[0415] Fig.46An exemplary instantaneous use of a beacon in multiple pulses is shown. This figure shows instantaneous use in a real-world application, such as a multi-pulse information beacon sent to various regions. The starting point of the information is SRV_BB 46000 at perimeter 46100. There are three beacon pulses sent at exactly the same instant in time to three different locations at different distances from the starting point. The three destination locations are target PFS 46280 at 46200 via path 46282, PFS 46580 via path 46582, and PFS 46380 via path 46382. Assume that the information is transmitted unidirectionally as a file from SRV_BB 46000 at as close to line speed as possible to write to each PFS via remote direct memory access (RDMA). The beacon ensures that the information is transmitted globally and combined with multiple pulses, ensuring that the information is always fresh by combining the known delay between the two points with a consistent instantaneous granularity for sending the beacon information pulse / flash and receiving, and then with the instantaneous granularity for processing and then using the received file next.

[0416] This has many more applications than just illustration.

[0417] Fig.47 An exemplary instantaneous use in beacons at intersections is shown. This figure shows the instantaneous granularity used in real-world applications, such as sending multiple beacon pulses from two locations and the intersection of pulses at a third location.

[0418] The first origin of the information transmitted by the heartbeat information beacon is at NYC 47100. The pulse is sent to LDN 47100 via path 74200. The second origin of the information transmitted by the heartbeat information beacon is at TOK 47120. The pulse is sent to LDN 47100 via path 47208.

[0419] In this example, the pulses from both origins have been synchronized to overlap. This can be to adjust the start time or to align the instant and the transmission rate.

[0420] For the purposes of this example, NYC stands for New York, New York, United States, LDN stands for London, England, United Kingdom, and TOK stands for Tokyo, Japan.

[0421] The network segments between the locations are indicated, as well as the best line rate for one direction of information.

[0422] The distance between New York and London is 3,456 miles, and the best one-way line speed is 47,000, which is equal to 27.3 ms. The distance between Tokyo and London is 5,936 miles, and the best one-way line speed is 47,020, which is equal to 46.8 ms. Not shown in the upper part of the figure but compared below is the distance between New York and Tokyo, which is 6,737 miles, and the best one-way line speed between them is 47,080, which is 53.1 ms.

[0423] For example, in financial markets, pricing data, other information and related news from different global regions are relevant and can affect trading in global markets, and having the latest current date information is mission critical. A supercomputer node (SCN) in London can receive information about New York and Tokyo and can react accordingly. If, for example, pricing in Tokyo begins to fluctuate, London can react faster than New York by a distance advantage of 801 miles 47082, which equals 6.3ms.

[0424] Instantaneous granularity and its impact on timing and consequence are therefore mission critical to traders seeking an edge. There are many other examples of how multiple pulses from multiple locations benefit from reliable instantaneous granularity.

[0425] Fig.48 An exemplary system diagram of instant granularity is shown. This diagram depicts various modules running on various devices that can utilize instant granularity as described herein. General Device All Equipment - With Instant 48900, can be applied to any device for which instant granularity is important. Hardware Monitor + Manager 48916 evaluates available resources, system load, hardware potential and current usage and other factors that may affect timing.

[0426] The OS Monitor+Manager 48918 evaluates the running of processes and other system related items that may have an impact on timing. The Time Updater 48970 interacts with the Time Server (SRV_TIME) 4831800 via the NTP Listener 4831850 to request and receive the most current time.

[0427] Moment software 48950 on all devices - with Moment 48900 working to ensure that the Moment granularity is as efficient as possible. It interacts with the monitor and manager. This can be extended using built-in software and plug-ins and other methods. Resource monitor 48980 and process monitor + manager 48985 can provide information to the Moment Manager 48988.

[0428] The modules, managers, HW components and other components shown in this article are for example only and may be different in actual use. Other devices can also use instant granularity to implement various functions.

[0429] The key unifying thread between them all is the importance of time. Therefore, system time must be accurate and reliable at instant granularity so that events, especially sequential events, can be processed and executed in a predictable manner.

[0430] According to the embodiments of the present application, the following notes are also provided:

[0431] 1. A system for performing file transfer and processing in a network environment, comprising: one or more processors coupled to a first device, configured to:

[0432] retrieving a file from a file queue, wherein the file is stored in a local memory of the first device, wherein the file is transferred from a second remote device via remote direct memory access;

[0433] Determining whether the document is complete;

[0434] If it is determined that the file is complete, the file is removed from the file queue.

[0435] 2. The system according to Note 1, wherein the second remote device is a backbone switching server.

[0436] 3. The system according to Note 1 also includes a dedicated link between the first device and the second remote device.

[0437] 4. A system according to Note 1, wherein the file includes a header, a body and a file trailer.

[0438] 5. A system according to Note 4, wherein the file tail includes a marker indicating the end of the file.

[0439] 6. A system according to Note 4, wherein the file includes a key-value pair.

[0440] 7. A system according to Note 4, wherein the file footer includes a checksum to verify the integrity of the body.

[0441] 8. A system according to Note 1, wherein the remote direct memory access is implemented based on at least one of wireless bandwidth or fiber channel.

[0442] 9. A system according to Note 1, wherein the file includes executable code.

[0443] 10. A system according to Note 1, wherein the one or more processors are further configured to: if the file is determined to be complete, perform at least one of the following operations: set a flag on the file, move the file to a folder, or delete the file from the local memory.

[0444] 11. A system for managing time granularity allocated for processing tasks, comprising: one or more processors, the one or more processors being configured to:

[0445] A duration required to process a task is determined, wherein the duration is calculated based on a first duration for processing the task, a second duration for post-processing associated with the task, and a third duration as a delay.

[0446] 12. A system according to Note 11, wherein the task includes batch processing.

[0447] 13. A system according to Note 11, wherein the duration is in at least one of microseconds or milliseconds.

[0448] 14. A system according to Note 11, wherein the duration is adjusted based on a historical record of runtime data for the duration.

[0449] 15. A system according to Note 11, wherein the second duration includes the time required to update the task status.

Claims

1. A system for file transmission and processing in a network environment, characterized in that: The system comprises: One or more processors coupled to the first device, configured to: retrieving files in batches from a file queue, wherein the files are stored in a local memory of the first device, wherein the files are transferred from a second remote device via remote direct memory access; In any batch retrieval, file pulling occurs at the beginning of a time interval and file usage occurs at the end of the time interval.

2. The system according to claim 1, characterized in that The first device includes a parallel file system.

3. The system according to claim 1, characterized in that The second remote device includes a backbone switching server.

4. The system according to claim 1, characterized in that The one or more processors are configured to: There is a delay between adjacent retrieval batches; The delay allows either batch to evaluate and process all files pulled by that either batch.

5. The system according to claim 4, characterized in that The delay also satisfies: Δt=P+Q+R; Among them, Δt is the difference time from the beginning of the instant to the end of the instant; P is the time for batch processing of items for the instant in the cycle; Q is the time used for post-batch processing calculations; R is the delay time to ensure that there is no overlap between batch items and / or set the offset between instants.

6. The system according to claim 1, characterized in that The one or more processors are further configured to: Determining whether the document is complete; If the file is complete, using the file, saving information about the file in a database in the local storage, and recording the file in a hierarchical file system; If the file is incomplete, the file is not used, the file is saved in a database in the local storage, and the file is recorded in a hierarchical file system.

7. The system according to claim 6, characterized in that The system also includes a launch node module; The launch node module resides on the second remote device.

8. The system according to claim 6, characterized in that The system also includes a launch node module; The ejection node module is a device or server independent of the second remote device.

9. The system according to claim 1 or 6, characterized in that: The file includes a header, a body and a file trailer.

10. The system according to claim 9, characterized in that The file trailer includes a marker indicating the end of the file.

Images