Processing and archiving data from edge nodes across distributed systems

Abstract

According to one method, a computer-implemented method includes detecting new data generated at a first edge node and causing a first copy of the new data to be transmitted in real-time to a central data storage location. A second copy of the new data is also stored in a first location at the first edge node. In response to a predetermined condition being satisfied, it is determined whether any information in the first copy of the transferred new data is missing at the central data storage location. In response to determining that information in the first copy of the new data transmitted is not missing at the central data storage location, the first copy of the new data is removed from a first location at the first edge node.

Description

BACKGROUND

[0001] The present invention relates to distributed storage systems, and more specifically, the present invention relates to processing and archiving data across distributed storage systems.

[0002] As computing power continues to advance and the use of IoT devices becomes more prevalent, the amount of data produced continues to increase. For example, the rise of smart enterprise endpoints has resulted in the generation of large amounts of data at remote locations. As 5G networks grow and the number of connected devices increases, data production will only further increase. This problem has also become more prevalent as the complexity of machine learning models increases. Increasingly complex machine learning models translate to heavier workloads and increased pressure associated with applying the models to received data. As a result, the operations of conventional implementations have been negatively impacted.

[0003] While cloud computing has been implemented in conventional systems in an effort to improve the ability to process this ever-increasing amount of data, the unprecedented scale and complexity of the data being created has outpaced the network and infrastructure capabilities. Sending all of the data generated by the devices to a centralized data center or to a cloud location has resulted in bandwidth and latency issues in conventional systems.

[0004] To overcome this reliance on the network to perform all of the processing at a central location, edge computing has been implemented to extend the computing to the endpoints in the system. For example, applications are moved to the edge location where the data is generated. As a result, both structured and unstructured data is generated at the edge location where the workloads are performed.

[0005] While edge computing has improved how data is processed, conventional systems suffer from inefficiencies due to how the data produced at these edge locations is maintained. These inefficiencies have resulted in both operational and security-based errors. For example, data that is not intended to be produced and processed at the edge is stored there because doing so would involve continuous incremental storage at the edge. This is particularly true for edge workloads involving object detection and / or image classification, which generate large amounts of unstructured data.

[0006] Each edge node also becomes a single point of failure for the entire system, significantly reducing the resiliency of the system. If one of the edge nodes fails, not only are the cached workloads at risk of permanent loss, but the data stored at the edge node is also at risk of permanent loss. The data generated at the edge is also frequently accessed by downstream applications and reporting requests.

[0007] However, allowing downstream applications and reporting to access the data directly from the edge nodes has introduced significant security risks in these conventional systems. As can be seen, conventional systems have suffered from operational and performance-based inefficiencies. Summary of the Invention

[0008] According to a method, a computer-implemented method includes: detecting new data generated at a first edge node, and transmitting a first copy of the new data to a central data storage location in real time. A second copy of the new data is also stored in a first location at the first edge node. In response to satisfying predetermined conditions, determining whether any information in the first copy of the transmitted new data is missing at the central data storage location. In response to determining that no information in the first copy of the transmitted new data is missing at the central data storage location, removing the first copy of the new data from the first location at the first edge node.

[0009] According to another approach, a computer program product includes a computer-readable storage medium having program instructions embodied therein. The program instructions are readable by a processor, executable by a processor, or both readable and executable by a processor to cause the processor to: perform the aforementioned method.

[0010] According to another approach, a system includes: a first edge node connected to a central data storage location, and a processor. Logic is also integrated with the processor and can be executed by the processor, or integrated with and can be executed by the processor. Furthermore, the logic is configured to execute the aforementioned method.

[0011] According to another method, a computer program product includes a computer-readable storage medium having program instructions embodied therein. The program instructions are readable by a processor, executable by a processor, or readable and executable by a processor to cause the processor to: receive a first copy of data from a first edge node. The first copy of the data is stored in a first storage location at a central data storage location. Also, receive a second copy of the data from a second edge node, and the second copy of the data is stored in a second storage location at the central data storage location. A request can be received from the first edge node to determine whether the first storage location includes a mirror copy of the first copy of the data received from the first edge node. In response to determining that the first storage location includes a mirror copy of the first copy of the data, the first copy of the data is released from the first edge node by sending an acknowledgment to the first edge node.

[0012] According to another approach, a system includes a central data storage location connected to a first edge node and a second edge node. The system also includes a processor, and logic integrated with the processor, which can be executed by the processor, or integrated with and executed by the processor. This logic is configured to perform the methods described above.

[0013] Other aspects and implementations of the invention will become apparent from the following detailed description when the principles of the invention are illustrated by way of example in conjunction with the accompanying drawings. Attached Figure Description

[0014] Figure 1 It is a diagram of the computational environment based on a method.

[0015] Figure 2 It is a diagram of a hierarchical data storage system based on a method.

[0016] Figure 3A It is a representational view of a distributed system based on a method.

[0017] Figure 3B yes Figure 3A A partial representational view of the logical file system in a distributed system.

[0018] Figure 4 It is a flowchart based on a method.

[0019] Figure 5 It is a representational operational view of an autonomous program based on an example in use.

[0020] Figure 6 It is a representational view of a distributed system based on an illustrative approach. Detailed Implementation

[0021] The following description is for the purpose of illustrating the general principles of the invention and is not intended to limit the inventive concept claimed herein. Furthermore, the specific features described herein can be used in combination with other described features in a variety of possible combinations and arrangements.

[0022] Unless otherwise specifically defined herein, all terms should be given the broadest possible interpretation, including the meaning inferred from the specification, the meaning understood by those skilled in the art, and / or the meaning as defined in dictionaries, textbooks, etc.

[0023] It must also be noted that, as used in the specification and appended claims, the singular forms “a,” “an,” and “the” include multiple referents unless the context otherwise requires. It will also be further understood that the terms “comprising” and / or “including,” when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0024] The following description discloses several preferred methods for systems, methods, and computer program products for maintaining mirrored copies of data received and / or generated at various edge nodes, thereby enabling efficient and scalable data archiving schemes across distributed systems. Maintaining a complete copy of data at a central and accessible location improves the operational efficiency of the system by providing faster access to the data. Data generated at edge nodes can be requested at different locations downstream in the distributed system, for example, to perform additional processing. Therefore, by sending copies of data from edge nodes to a central data storage location in real time as data is received and / or generated, the central data storage location is able to meet operations and requests related to data from one or more edge nodes, for example, as will be described in further detail below.

[0025] In a general approach, the computer-implemented method involves detecting new data generated at a first edge node. By detecting newly generated data at the edge nodes, the implementation presented in this paper enables real-time changes to the distributed system. This ideally allows for maintaining mirror copies of the data received and / or generated at each edge node.

[0026] The computer-implemented method also includes transmitting a first copy of the new data to a central data storage location in real time. As described above, the implementation in this paper enables real-time changes to distributed systems. This ideally allows for maintaining mirror copies of data received and / or generated at each edge node. Therefore, the implementation in this paper enables an efficient and scalable data archiving scheme across distributed systems in real time. Consequently, this scheme can be applied to various types of implementations, such as enterprise edge node workloads. The method in this paper is also easily scalable, supporting multiple edge nodes and multiple edge workloads even in large enterprise setups.

[0027] The computer-implemented method also includes storing a second copy of the new data in a first location at the first edge node. By maintaining an additional copy of the new data locally at the first edge node, additional operations corresponding to that data can be performed locally at the first edge node without accessing the central data storage location, thereby reducing latency and improving performance throughput.

[0028] The computer-implemented method also includes any information used to determine whether a first copy of the newly transmitted data is missing at the central data storage location in response to predetermined conditions. Examining the copy stored at the central data storage location allows the method to determine whether any portion of the data was lost during transmission to the central data storage location. Data loss during transmission can be caused by communication errors, network shortages, write errors, etc., which can affect some or all of the data sent from the edge nodes. Therefore, these situations can be identified and corrected before deleting the first copy of the new data retained at the edge nodes.

[0029] The computer-implemented method also includes removing a first copy of the new data from a first location at the first edge node in response to determining that no information is missing in the first copy of the new data being transmitted at the central data storage location. Similarly, examining the copy stored at the central data storage location allows the method to determine if any portion of the data was lost during transmission to the central data storage location. Data loss during transmission can be caused by communication errors, network shortages, write errors, etc., which can affect some or all of the data sent from the edge nodes. Therefore, these situations can be identified and corrected before deleting the first copy of the new data retained at the edge nodes. This ideally allows for the real-time maintenance of mirrored copies of data received and / or generated at various edge nodes. Therefore, the implementation in this paper enables an efficient and scalable data archiving scheme across distributed systems. Maintaining a complete copy of the data at a central and accessible location also significantly improves the overall operational efficiency of the system by providing faster access to the data.

[0030] In some implementations, the central data storage location is configured to receive and store one or more additional data copies received from one or more additional edge nodes. As described above, the implementation in this paper enables an efficient and scalable data archiving scheme across distributed systems in real time. Therefore, the approach in this paper is easily scalable, providing support for multiple edge nodes and multiple edge workloads even in large enterprise setups. This further amplifies the improvements of the implementation in this paper by providing faster access to larger amounts of data, increasing the efficiency of the computing system as a whole.

[0031] In a preferred implementation, the central data storage location is configured to maintain a mirror copy of the data received at one or more additional edge nodes. Similarly, the implementation presented in this paper enables an efficient and scalable data archiving scheme that maintains mirror copies of data in real time across a distributed system. Therefore, even in large enterprise setups, the approach presented in this paper can provide support for multiple edge nodes and multiple edge workloads. This reduces latency issues while also improving access to larger volumes of data, thereby increasing the efficiency of the computing system as a whole.

[0032] In some implementations, it is determined that at least some information in the first copy of the new data is actually missing at the central data store, and the missing information is then transferred from the edge nodes to the central data store. Similarly, examining the copy stored at the central data store allows the method to determine whether any part of the data was lost during transmission to the central data store. Therefore, these situations can be identified and corrected before the first copy of the new data held at the edge nodes is deleted. This ideally allows for the real-time maintenance of mirror copies of data received and / or generated at various edge nodes.

[0033] In response to transmitting missing information to the central data storage location, a first copy of the new data can be removed from the first location. Similarly, examining the copy stored at the central data storage location allows the method to determine if any part of the data was lost during transmission to the central data storage location. Therefore, these situations can be identified and corrected before deleting the first copy of the new data retained at the edge nodes. This ideally allows for the real-time maintenance of mirrored copies of data received and / or generated at various edge nodes. Thus, the implementation in this paper enables an efficient and scalable data archiving scheme across distributed systems. Maintaining a complete copy of the data at a central and accessible location also significantly improves the efficiency of the system as a whole by providing faster access to the data.

[0034] Some implementations implement archive processors configured to manage the edge file hierarchy at edge nodes. Archive processors also include a set of observer components and a remover component. System performance is improved because archive processors are implemented that can be easily deployed as containerized edge workloads to a variety of edge nodes. This helps make data available for real-time downstream consumption while protecting edge nodes from unwanted access. Data remains available at the edge nodes as expected, for example, as determined in several different ways as described in this paper. This further minimizes storage costs at each edge node.

[0035] In some implementations, each observer in the observer component set corresponds to a specific folder at the first edge node. Furthermore, new data generated at the first edge node is detected in response to a notification received from the corresponding observer component. Similarly, implementing an archive processor and its corresponding observer allows the implementation in this paper to improve system performance by leveraging an archive processor that can be easily deployed as a containerized edge workload to various edge nodes. This helps make data available for real-time downstream consumption while protecting edge nodes from unnecessarily accessed data.

[0036] In other implementations, one or more instructions are sent to the remover component of the archive processor to remove a first copy of the new data. One or more instructions cause the remover component to remove the first copy of the new data from a first location. As described above, examining the data copy stored at the central data storage location allows the method to determine if any portion of the data has been lost during transmission to the central data storage location. Therefore, these situations can be identified and corrected before deleting the first copy of the new data retained at the edge nodes. This ideally allows for the real-time maintenance of mirror copies of data received and / or generated at various edge nodes.

[0037] In other implementations, meeting predetermined conditions includes maintaining a first copy of the new data at a first location on the first edge node for a first predetermined time period. In other words, after the predetermined time period, the data copy transmitted to the central data storage location can be checked. Furthermore, this predetermined time period is based at least on the type and / or amount of information included in the first copy of the new data. Checking the copy stored at the central data storage location allows the method to determine whether any portion of the data was lost during transmission to the central data storage location. Data loss during transmission can be caused by communication errors, network shortages, write errors, etc., which can affect some or all of the data sent from the edge node. Therefore, these situations can be identified and corrected before deleting the data copy retained at the edge node, thereby improving the efficiency of reflecting new data in real time at the central location.

[0038] According to another general approach, a computer program product includes a computer-readable storage medium having program instructions embodied therein. The program instructions are readable by a processor, executable by a processor, or both readable and executable by a processor, such that the processor: performs the methods described above.

[0039] According to yet another general approach, a system includes: a first edge node connected to a central data storage location, and a processor. The system also includes logic integrated with the processor, processor-executable logic, or logic integrated with and executable by the processor. Furthermore, the logic is configured to perform the aforementioned methods.

[0040] According to another general approach, a computer program product includes a computer-readable storage medium having program instructions embodied therein. The program instructions are readable by a processor, executable by a processor, or both readable and executable by a processor, such that the processor: receives a first copy of data from a first edge node. By receiving newly generated data at the edge nodes, the implementation of this paper enables real-time changes to the distributed system. This ideally allows for maintaining mirror copies of the data received and / or generated at various edge nodes in a central and easily accessible location.

[0041] The program instructions can also be read and / or executed by the processor to cause the processor to: store a first copy of the data in a first storage location at the central data storage location. As described above, the implementation in this paper enables real-time changes to the distributed system. This ideally allows for maintaining mirrored copies of data received and / or generated at various edge nodes. Therefore, the implementation in this paper enables an efficient and scalable data archiving scheme across distributed systems in real time. Consequently, this scheme can be applied to various types of implementations, such as enterprise edge node workloads. The method in this paper is also easily scalable, providing support for multiple edge nodes and multiple edge workloads even in large enterprise settings.

[0042] The program instructions can also be read and / or executed by the processor to cause the processor to: receive a second copy of the data from the second edge node. Similarly, by receiving newly generated data at the edge nodes, the implementation of this paper enables real-time changes to the distributed system. This ideally allows for maintaining mirror copies of the data received and / or generated at various edge nodes in a central and easily accessible location.

[0043] The program instructions can also be read and / or executed by the processor to cause the processor to: store a second copy of the data in a second storage location at the central data storage location. Similarly, the implementation in this paper enables real-time changes to the distributed system. This ideally allows for maintaining mirrored copies of data received and / or generated at various edge nodes. Therefore, the implementation in this paper enables an efficient and scalable data archiving scheme across distributed systems in real time.

[0044] The program instructions can also be read and / or executed by the processor to cause the processor to: receive a request from the first edge node to determine whether the first storage location contains a mirror copy of a first copy of the data received from the first edge node. Checking the copy stored at the central data storage location allows the method to determine whether any portion of the data was lost during transmission to the central data storage location. Data loss during transmission may be caused by communication errors, network shortages, write errors, etc., which can affect some or all of the data sent from the edge node. Therefore, these situations can be identified and corrected before deleting the data copy retained at the edge node.

[0045] The program instructions can also be read and / or executed by the processor to cause the processor to: release the first copy of the data from the first edge node by sending an acknowledgment to the first edge node in response to determining that the first storage location includes a mirror copy of the data. Similarly, checking the copy stored at the central data storage location allows the method to determine whether any part of the data has been lost during transmission to the central data storage location. Data loss during transmission may be caused by communication errors, network shortages, write errors, etc., which can affect some or all of the data sent from the edge nodes. Therefore, these situations can be identified and corrected before deleting the new data copy retained at the edge nodes. This ideally allows for the real-time maintenance of mirror copies of data received and / or generated at various edge nodes. Therefore, the implementation in this paper enables an efficient and scalable data archiving scheme across distributed systems. Maintaining a complete copy of the data at a central and accessible location also significantly improves the efficiency of the system as a whole by providing faster access to the data.

[0046] In some implementations, program instructions can be read and / or executed by the processor to cause the processor to: in response to determining that a mirror copy of the data is not present in the first storage location, issue a request to the first edge node for the missing portion of the first copy of the data. Similarly, examining the copy stored at the central data storage location allows the method to determine whether any portion of the data has been lost during transmission to the central data storage location. Therefore, these situations can be identified and corrected before deleting the data copies retained at the edge nodes. This ideally allows for the real-time maintenance of mirror copies of data received and / or generated at various edge nodes.

[0047] Therefore, the implementation also includes receiving the missing portion of the first copy of the data and using the missing portion to complete a mirror copy of the first copy of the data at the first storage location. Even in large enterprise setups, the method in this paper is able to support multiple edge nodes and multiple edge workloads. This reduces latency issues while improving access to larger amounts of data, thereby increasing the efficiency of the computing system as a whole.

[0048] In some implementations, program instructions can also be read and / or executed by the processor to cause the processor to: receive a request from the second edge node to determine whether the second storage location contains a mirror copy of a second copy of the data received from the second edge node. Similarly, examining the copy stored at the central data storage location allows the method to determine whether any portion of the data has been lost during transmission to the central data storage location. Therefore, these situations can be identified and corrected before deleting the data copies retained at the edge nodes. This ideally allows for the real-time maintenance of mirror copies of data received and / or generated at various edge nodes.

[0049] Therefore, the implementation also includes releasing the second copy of the data from the second edge node by sending an acknowledgment to the second edge node in response to determining that the second storage location contains a second copy of the data. Even in large enterprise setups, the method described in this paper is able to support multiple edge nodes and multiple edge workloads. This reduces latency issues while improving access to larger amounts of data, thereby increasing the efficiency of the computing system as a whole.

[0050] In some implementations, program instructions may also be read and / or executed by the processor to cause the processor to: in response to determining that a mirror copy of the data is not present in the second storage location, issue a request to the second edge node for the missing portion of the second copy of the data. Similarly, examining the copy stored at the central data storage location allows the method to determine whether any portion of the data has been lost during transmission to the central data storage location. Therefore, these situations can be identified and corrected before deleting the data copies retained at the edge nodes. This ideally allows for the real-time maintenance of mirror copies of data received and / or generated at various edge nodes.

[0051] Therefore, the implementation also includes receiving the missing portion of a second copy of the data and using the missing portion to complete a mirror copy of the data at a second storage location. Even in large enterprise setups, the method presented in this paper is able to support multiple edge nodes and multiple edge workloads. This reduces latency issues while improving access to larger amounts of data, thereby increasing the efficiency of the computing system as a whole.

[0052] According to another general approach, a system includes: a central data storage location connected to a first edge node and a second edge node, and a processor. Logic is integrated with the processor, is executable by the processor, or is integrated with and executable by the processor, and is configured to perform the aforementioned operations.

[0053] Various aspects of this disclosure are described by narrative text, flowcharts, block diagrams of computer systems and / or block diagrams of machine logic (including in computer program product (CPP) methods). With respect to any flowchart, depending on the technology involved, operations may be performed in a different order than shown in a given flowchart. For example, also depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0054] Computer program product implementation (“CPP implementation” or “CPP”) is a term used in this disclosure to describe any collection of one or more storage media (also referred to as “media”) collectively included in a collection of one or more storage devices, the collection of one or more storage devices collectively including machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A “storage device” is any tangible device capable of holding and storing instructions for use by a computer processor. Without limitation, a computer-readable storage medium can be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punched cards or pits / bumps formed on the main surface of the disk), or any suitable combination of the foregoing. As used in this disclosure, the term "computer-readable storage medium" should not be construed as storing transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses transmitted through fiber optic cables, electrical signals communicated through wires, and / or other transmission media. As those skilled in the art will understand, data typically moves at some occasional points in time during normal operation of the storage device, such as during access, defraction, or garbage collection, but this does not make the storage device transient, because the data is not transient when it is stored.

[0055] The computing environment 100 includes examples of an environment for performing at least some computer code related to the methods of the invention, such as the improved data archiving code at box 150 for maintaining mirror copies of data received and / or generated at various edge nodes, thus enabling an efficient and scalable data archiving scheme across distributed systems. In addition to box 150, the computing environment 100 also includes, for example, a computer 101, a wide area network (WAN) 102, an end-user equipment (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this method, the computer 101 includes a processor set 110 (including processing circuitry 120 and cache 121), a communication architecture 111, volatile memory 112, persistent storage 113 (including an operating system 122 and box 150, as described above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. Public cloud 105 includes gateway 140, cloud orchestration module 141, host physical machine set 142, virtual machine set 143, and container set 144.

[0056] Computer 101 can take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or to be developed in the future capable of running programs, accessing networks, or querying databases (such as remote database 130). As well understood in the field of computer technology, and depending on the technology, the execution of computer-implemented methods can be distributed among multiple computers and / or across multiple locations. On the other hand, in this introduction to computing environment 100, the detailed discussion focuses on a single computer, specifically computer 101, to keep the introduction as simple as possible. Computer 101 can reside in the cloud, even if it is not physically present in the cloud. Figure 1 The diagram shows that it is located in the cloud. On the other hand, except to a degree that can be definitively indicated, computer 101 is not required to be located in the cloud.

[0057] Processor set 110 includes any type of computer processor now known or to be developed in the future. Processing circuitry 120 may be distributed across multiple packages, for example, multiple coordinated integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is a memory located within the processor chip package and typically used for data or code that should be quickly accessible to the threads or cores running on processor set 110. Cache memory is typically organized into multiple levels based on its relative proximity to the processing circuitry. Alternatively, some or all of the processor set's cache may be located "off-chip". In some computing environments, processor set 110 may be designed to work with qubits and perform quantum computing.

[0058] Computer-readable program instructions are typically loaded onto computer 101 to cause a series of operational steps to be executed by processor set 110 of computer 101, thereby implementing a computer-implemented method, such that the instructions executed thereby instantiate the method specified in the flowcharts and / or narrative descriptions of the computer-implemented method included in this document (collectively, the “inventive method”). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. Processor set 110 accesses the program instructions and associated data to control and direct the execution of the inventive method. In computing environment 100, at least some of the instructions for performing the inventive method may be stored in box 150 of persistent storage 113.

[0059] Communication structure 111 is a signal transmission path that allows various components of computer 101 to communicate with each other. Typically, this structure is made of switches and conductive paths, such as those forming buses, bridges, physical input / output ports, etc. Other types of signal communication paths can be used, such as fiber optic communication paths and / or wireless communication paths.

[0060] Volatile memory 112 is any type of volatile memory now known or to be developed in the future. Examples include dynamically typed random access memory (RAM) or statically typed RAM. Typically, volatile memory 112 is characterized by random access, but this is not necessary unless explicitly indicated. In computer 101, volatile memory 112 is located in a single package and inside computer 101; however, alternatively or additionally, volatile memory may be distributed across multiple packages and / or located externally relative to computer 101.

[0061] Persistent storage 113 is any form of non-volatile storage for a computer, now known or to be developed in the future. The non-volatility of this storage means that the stored data is retained regardless of whether the computer 101 is powered or directly powered. Persistent storage 113 may be read-only memory (ROM), but typically at least a portion of persistent storage allows data to be written, deleted, and rewritten. Some familiar forms of persistent storage include hard disks and solid-state storage devices. Operating system 122 can take many forms, such as various known proprietary operating systems or operating systems employing an open-source portable operating system interface type with a kernel. The code included in box 150 typically includes at least some computer code relating to performing the inventive methods.

[0062] Peripheral device set 114 includes a collection of peripheral devices for computer 101. Data communication connections between peripheral devices and other components of computer 101 can be implemented in various ways, such as Bluetooth connectivity, near field communication (NFC) connectivity, connections made via cables (such as Universal Serial Bus (USB) type cables), plug-in connections (e.g., Secure Digital (SD) cards), connections made via local area communication networks, and even connections made via wide area networks (such as the Internet). In various approaches, UI device set 123 may include components such as displays, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 124 is external storage, such as an external hard drive, or a pluggable storage device, such as an SD card. Storage 124 can be persistent and / or volatile. In some implementations, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In implementations requiring computer 101 to have substantial storage (e.g., where computer 101 locally stores and manages a large database), this storage can be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in IoT applications. For example, one sensor could be a thermometer, and another could be a motion detector.

[0063] Network module 115 is a collection of computer software, hardware, and firmware that allows computer 101 to communicate with other computers via WAN 102. Network module 115 may include hardware (such as a modem or Wi-Fi transceiver), software for packetizing and / or unpacking data for transmission over the communication network, and / or web browser software for communicating data via the Internet. In some implementations, the network control and network forwarding functions of network module 115 are performed on the same physical hardware device. In other implementations (e.g., implementations utilizing Software-Defined Networking (SDN), the control and forwarding functions of network module 115 are performed on physically separate devices, such that the control functions manage several different network hardware devices. Computer-readable program instructions for performing the methods of the invention can typically be downloaded to computer 101 from an external computer or external storage device via a network adapter card or network interface included in network module 115.

[0064] WAN 102 is any wide area network (e.g., the Internet) that can communicate computer data over non-local distances via any technology currently known or to be developed for communicating computer data. In some implementations, WAN 102 can be replaced and / or supplemented by a local area network (LAN) (such as a Wi-Fi network) designed for communication of data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0065] End User Equipment (EUD) 103 is any computer system used and controlled by an end user (e.g., an enterprise customer operating computer 101) and can take any of the forms discussed above regarding computer 101. EUD 103 typically receives helpful and useful data from the operation of computer 101. For example, if computer 101 is designed to provide recommendations to the end user, these recommendations would typically be communicated from computer 101's network module 115 via WAN 102 to EUD 103. In this way, EUD 103 can display or otherwise present recommendations to the end user. In some implementations, EUD 103 can be a client device, such as a thin client, a thick client, a mainframe computer, a desktop computer, etc.

[0066] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 can be controlled and used by the same entity operating computer 101. Remote server 104 represents a machine that collects and stores helpful and useful data for use by other computers, such as computer 101. For example, if computer 101 is designed and programmed to provide recommendations based on historical data, that historical data can be provided to computer 101 from a remote database 130 of remote server 104.

[0067] Public cloud 105 is any computer system that can be used by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities (especially data storage (cloud storage) and computing power) without direct active management by users. Cloud computing typically leverages resource sharing to achieve consistency and economies of scale. Direct and active management of the computing resources of public cloud 105 is performed by the computer hardware and / or software of cloud orchestration module 141. The computing resources provided by public cloud 105 are typically implemented by virtual computing environments running on individual computers in host physical set 142, which is the entire domain of physical computers in and / or available to public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set 143 and / or containers from container set 144. It is understood that these VCEs can be stored as images and can be transferred between individual physical machine hosts as images or after VCE instantiation. Cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instantiations, and manages the instantiation of active VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that allow public cloud 105 to communicate via WAN 102.

[0068] Now, we will provide some further explanation of Virtualized Computing Environments (VCEs). A VCE can be stored as an "image." New active instances of a VCE can be instantiated from an image. Two familiar types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to the operating system feature where the kernel allows the existence of multiple isolated user-space instances (called containers). From the perspective of the programs running within them, these isolated user-space instances typically behave like a real computer. Computer programs running on a regular operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running within a container can only use the contents of the container and the devices allocated to the container; this characteristic is called containerization.

[0069] Private cloud 106 is similar to public cloud 105, except that the computing resources are only available for use by a single enterprise. While private cloud 106 is depicted as communicating with WAN 102, in other implementations, private cloud can be completely disconnected from the internet and accessed only via a local / private network. Hybrid cloud consists of multiple clouds of different types (e.g., private, community, or public cloud types), typically implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple component clouds. In this approach, both public cloud 105 and private cloud 106 are part of a larger hybrid cloud.

[0070] In some aspects, a system, depending on its implementation, may include a processor and logic integrated with and / or executable by the processor, the logic being configured to perform one or more processing steps described herein. The processor may be any configuration as described herein, such as a discrete processor or processing circuitry comprising numerous components such as processing hardware, memory, I / O interfaces, etc. By being integrated with a processor, it means that the processor has logic embedded as hardware logic, such as application-specific integrated circuits (ASICs), FPGAs, etc. By being executable by a processor, it means that the logic is hardware logic; software logic, such as firmware, part of an operating system, part of an application, etc., or a combination of hardware and software logic, which is accessible to the processor and configured to cause the processor to perform some function when executed by the processor. The software logic may be stored in local and / or remote memory of any memory type known in the art. Any processor known in the art may be used, such as software processor modules and / or hardware processors, such as ASICs, FPGAs, central processing units (CPUs), integrated circuits (ICs), graphics processing units (GPUs), etc.

[0071] Of course, depending on the implementation, this logic can be implemented as a method or computer program product on any device and / or system.

[0072] Now for reference Figure 2 The storage system 200 is illustrated according to one method. Note that... Figure 2Some of the components shown can be implemented as hardware and / or software according to various methods. Storage system 200 may include a storage system manager 212 for communicating with multiple media and / or drives on at least one higher storage layer 202 and at least one lower storage layer 206. The higher storage layer 202 preferably includes one or more random access and / or direct access media 204, such as hard disks in hard disk drives (HDDs), non-volatile memory (NVM), solid-state drives (SSDs), flash memory, SSD arrays, flash memory arrays, etc., and / or other media pointed out herein or known in the art. The lower storage layer 206 preferably includes one or more lower-performance storage media 208, including sequential access media, such as magnetic tape and / or optical media in tape drives, slower access HDDs, slower access SSDs, etc., and / or other media pointed out herein or known in the art. One or more additional storage layers 216 may include any combination of storage memory media desired by the designer of system 200. In addition, either the higher storage layer 202 and / or the lower storage layer 206 may include some combination of storage devices and / or storage media.

[0073] Storage system manager 212 can communicate via network 210 (such as storage area network (SAN), etc.) Figure 2 The storage system manager 212 (or other suitable network type) communicates with drives and / or storage media 204, 208 on the higher storage layer 202 and the lower storage layer 206. The storage system manager 212 can also communicate with one or more host systems (not shown) via a host interface 214, which may or may not be part of the storage system manager 212. The storage system manager 212 and / or any other component of the storage system 200 can be implemented in hardware and / or software, and can use a processor (not shown) to execute commands of types known in the art, such as a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Of course, any arrangement of the storage system can be used, which will be apparent to those skilled in the art upon reading this description.

[0074] In other methods, storage system 200 may include any number of data storage tiers, and may include the same or different storage media within each storage tier. For example, each data storage tier may include the same type of storage media, such as HDD, SSD, sequential access media (tape in a tape drive, optical disc in an optical disc drive, etc.), direct access media (CD-ROM, DVD-ROM, etc.), or any combination of media storage types. In one such configuration, higher storage tier 202 may include most SSD storage media for storing data in a higher-performance storage environment, and the remaining storage tiers, including lower storage tier 206 and additional storage tier 216, may include any combination of SSDs, HDDs, tape drives, etc., for storing data in a lower-performance storage environment. In this way, data accessed more frequently, data with higher priority, data requiring faster access, etc., can be stored in higher storage tier 202, while data without one of these attributes can be stored in additional storage tier 216, including lower storage tier 206. Of course, those skilled in the art, after reading this description, can design many other combinations of storage media types based on the methods proposed herein to implement in different storage schemes.

[0075] According to some implementations, the storage system (such as 200) may include logic configured to receive a request to open a dataset, logic configured to determine whether the requested dataset is stored in a lower storage layer 206 of the hierarchical data storage system 200 in multiple associated portions, logic configured to move each associated portion of the requested dataset to a higher storage layer 202 of the hierarchical data storage system 200, and logic configured to assemble the requested dataset from the associated portions on the higher storage layer 202 of the hierarchical data storage system 200.

[0076] Therefore, storage system 200 can use different types of memory to achieve different levels of performance. For example, storage system manager 212 is used to control where data is processed and / or stored in system 200, where each location can achieve a different performance level (e.g., result). Similarly, the following... Figure 3A A distributed system 300 is illustrated, having different edge nodes 304, 306, 308 connected to a central data storage location 302. Therefore, data and other information (e.g., commands, instructions, requests, metadata, etc.) can be sent between the edge nodes 304, 306, 308 and the central data storage location 302, for example, as described in further detail below.

[0077] As mentioned above, data production is constantly increasing with advancements in computing power and the use of IoT devices. For example, the rise of smart enterprise endpoints has led to the generation of massive amounts of data in remote locations. With the growth of 5G networks and the increasing number of connected mobile devices, data production will only increase further. This problem is also becoming more prevalent as the complexity of machine learning models increases. Increasingly complex machine learning models translate into more intensive workloads and increased pressure associated with applying the models to received data. Consequently, conventional implementations are negatively impacted.

[0078] While cloud computing has been implemented in conventional systems in an effort to improve the ability to handle this ever-increasing volume of data, the unprecedented scale and complexity of the data creation has exceeded the capabilities of networks and infrastructure. Sending all the data generated by all devices to a centralized data center or cloud location has already led to bandwidth and latency issues in conventional systems.

[0079] To attempt to overcome this dependency on the network performing all processing at a central location, edge computing has been implemented to extend computation to the endpoints of the system. For example, applications are moved to the edge locations that generate data. Thus, both structured and unstructured data are generated at the edge locations where workloads are executed.

[0080] While edge computing has improved how data is processed, conventional systems have suffered from inefficiencies in how data generated at these edge locations is maintained. These inefficiencies have led to numerous operational and security-based errors. For example, data generated and processed at the edge is not intended to be stored there, as doing so involves continuous incremental storage at the edge, resulting in increased processing overhead. This is particularly true for edge workloads involving object detection and / or image classification, which continuously generate large amounts of unstructured data.

[0081] Each edge node also becomes a single point of failure for the entire system, significantly reducing the resilience of such conventional systems. If one edge node fails, not only is the cached workload at risk of permanent loss, but the data stored at the failed edge node is also at risk of permanent loss. Data generated at the edge is also frequently requested by downstream applications and reporting. However, allowing downstream applications and reporting to access data directly from edge nodes in these conventional systems has introduced significant security risks. These security risks include, but are not limited to, issues related to access control, patching, chain of custody, transport, encryption, integrity, and export controls.

[0082] As can be seen, conventional systems suffer from significant operational and performance-based inefficiencies. In stark contrast to these conventional drawbacks, the implementation presented in this paper ensures that data generated at edge nodes remains available (e.g., for additional processing) in addition to being securely stored. Therefore, the implementation in this paper overcomes the storage- and processing-based problems that plague conventional systems.

[0083] In some implementations, these improvements are made for systems that include (e.g., generate) unstructured data at system edge nodes. However, improvements can also be made in response to applying the methods described herein to systems that include structured data, combinations of structured and unstructured data, etc.

[0084] Looking at it now Figure 3A This illustrates a system 300 with a distributed architecture according to one method. Alternatively, it can be implemented using any other methods listed herein (such as those shown in other figures). Figures 1-2 The features described herein are used in combination to implement the current system 300. However, such a system 300, as well as other systems presented herein, can be used in various applications and / or permutations, which may or may not be specifically described in the illustrative methods or implementations listed herein. Furthermore, the system 300 presented herein can be used in any desired environment. Therefore, Figure 3A (And other accompanying figures) can be considered to include any possible permutation.

[0085] As shown, system 300 includes a central data storage location 302 connected to a first edge node 304, a second edge node 306, and a third edge node 308. Specifically, the central data storage location 302 and edge nodes 304, 306, and 308 are connected to network 310.

[0086] Network 310 can be of any type, depending on the desired method. For example, in some methods, network 310 is a WAN, such as the Internet. However, an illustrative list of other network types that network 310 can implement includes, but is not limited to, LAN, PSTN, SAN, intranet telephone network, etc. Therefore, regardless of the degree of separation between them, such as despite being located in different geographical locations, any desired information, data, commands, instructions, responses, requests, etc., can be transmitted between edge nodes 304, 306, 308 and / or central data storage location 302.

[0087] It should also be noted that two or more of the edge nodes 304, 306, 308 and / or the central data storage location 302 may be connected in different ways. According to examples, two edge computing nodes may be located relatively close to each other and connected via a wired connection (e.g., cable, fiber optic link, wire, etc.), etc., or by any other type of connection that will be obvious to those skilled in the art upon reading this description.

[0088] Continue to refer to Figure 3A Edge nodes 304, 306, and 308 may have different configurations than the central data storage location 302. For example, in some implementations, the central data storage location 302 includes a large (e.g., robust) processor 312 coupled to a cache 309 and a data storage array 314 with relatively high storage capacity. Therefore, the central data storage location 302 is capable of processing and storing relatively large amounts of data, allowing it to connect to and manage multiple different remote edge nodes (also referred to herein as “edge locations”). As described above, the central data storage location 302 can receive data, commands, etc., from any number of locations. Therefore, the components included in the central data storage location 302 preferably have higher storage capacity and throughput than the components included in each of the edge nodes 304, 306, and 308 to accommodate the higher data flow experienced at the central data storage location 302.

[0089] It should be noted that, in this description, "data" can include information of any desired type. For example, in different implementations, data can include raw sensor data, metadata, program commands, instructions, etc. Furthermore, while the implementations herein are described in the context of unstructured data, this is by no means intended to be limiting. Any desired type of data can be applied to the methods described herein. Therefore, processor 312 can use cache 309 and / or memory array 314 to actually cause the execution of one or more data operations. According to the example, processor 312 at central data storage location 302 can be used to perform... Figure 4 Method 400 includes one or more operations.

[0090] The administrator 313 and / or processor 312 of the central data storage location 302 themselves are in Figure 3A The central data storage location 302 is also shown as connected to network 310. In some implementations, the administrator 313 can connect directly to the central data storage location 302, as shown by the dashed arrow. Therefore, the administrator 313 is able to control at least a portion of the central data storage location 302, such as the file system 311 in the data storage array 314.

[0091] For reference only Figure 3BA representative view of file system 311 is shown according to one method. Alternatively, it can be illustrated with any other implementation listed herein (such as referring to other figures, such as...). Figure 3A The current file system 311 is implemented using the features described herein in combination. For example, file system 311 is shown as physical storage corresponding to a specific portion of data storage array 314, as those skilled in the art will understand after reading this description. However, such a file system 311, as well as other file systems presented herein, can be used in various applications and / or arrangements, which may or may not be specifically described in the illustrative implementations listed herein. Furthermore, the file system 311 presented herein can be used in any desired environment. Therefore, Figure 3B (And other accompanying figures) can be considered to include any possible permutation.

[0092] As shown, the file system 311 includes a main folder 350 that can be used as the root folder. The main folder 350 includes subfolders 352 and 354, each of which can correspond to a specific edge node. Therefore, the main folder 350 can include any desired number of subfolders depending on the number of edge nodes connected to the central data storage location in the distributed system. Thus, although in Figure 3B It only includes two subfolders, 352 and 354, but other subfolders corresponding to other edge nodes can also be included under the main folder 350.

[0093] Each of subfolders 352 and 354 also includes a layer pointing to a different workload executed at the corresponding edge node. Thus, subfolder 352 includes a layer 356 corresponding to a first workload executed at the corresponding edge node. Subfolder 352 also includes a layer 358 corresponding to a second workload executed at the corresponding edge node.

[0094] Each of layers 356 and 358 also includes numerous directory locations, each corresponding to a different type of data. As described above, the data included at the edge nodes may have been received from many different sources. For example, different types of sensors generate different types of data, and this data can be used to form different types of files. For instance, data received from a camera can be used to generate video files (e.g., such as .avi, .wmv, .gif, etc.), while data received from a microphone can be used to generate audio files (e.g., such as .mp3, .m4p, .wav, etc.), and data received from a digital still camera can be used to generate image files (e.g., such as .jpg, .tif, .png, etc.).

[0095] Therefore, each of the directory locations 364, 366, and 368 under layer 356 stores different types of data therein. Similarly, directory locations 370, 372, and 374 store different types of data under layer 356. According to the example (which is by no means intended to limit the invention), directory locations 364 and 370 are each configured (e.g., specified) to store video files under the corresponding layer, directory locations 366 and 372 are each configured to store audio files for the corresponding layer, and directory locations 368 and 374 are each configured to store metadata for the corresponding layer for the edge nodes.

[0096] Subfolder 354 comprises layers 360 and 362, each layer corresponding to a workload executed at the respective edge node. Layers 360 and 362 also include numerous directory locations, each corresponding to a different type of data. Therefore, directory locations 376, 378, and 380 under layer 360 store different types of data, while directory locations 382, ​​384, and 386 under layer 362 store different types of data. It should also be noted that directory locations 364, 366, 368, 370, 372, 374, 376, 378, 380, 382, ​​384, and 386 are also referred to as "subfolders" in some methods described herein.

[0097] The configuration of file system 311 as described herein allows for detailed copies of the data to be stored. As mentioned above, the main folder 350 includes subfolders 352 and 354, each of which may correspond to a specific edge node. Therefore, each subfolder of file system 311 can be used to maintain mirror copies of data received and / or generated at multiple edge nodes, for example, as will be described in further detail below.

[0098] Reference Back Figure 3A User 315 is also shown connected to one of the edge nodes 306. In some methods, user 315 connects to edge node 306 via a computing device (e.g., a user's personal computer, mobile phone, locally deployed server, etc.) to allow information to be exchanged between them. However, in other methods, user 315 may access edge node 306 using one or more terminals with a user interface. In some implementations, user 315 may also connect to network 310. Thus, in such an implementation, user 315 can access edge node 306 and / or other locations in system 300 via network 310. In still other implementations, user can access network 310 via a direct connection to edge node 306.

[0099] The first edge node 304 includes a processor 316 coupled to data storage 318. Similarly, edge node 306 includes a processor 320 coupled to data storage 322, and edge node 308 includes a processor 324 coupled to data storage 326. The data storage implemented at each edge node 304, 306, 308 can be used to store data received from one or more sensors (not shown) communicating with the respective edge node, a user 315 communicating with one or more edge nodes, other edge nodes, different systems also connected to network 310, etc.

[0100] System 300 also preferably includes an archive processor configured to manage file hierarchies at edge nodes 304, 306, 308 and / or a mirrored file hierarchy at a central data storage location 302. In some methods, the archive processor includes a software application designed to manage archives (e.g., file hierarchies or file systems) at one of the edge nodes 304, 306, 308 and / or the central data storage location 302. The archive processor may be a distributed microservice deployed in each edge node in some implementations (e.g., an application architecture style where a collection of independent services communicates through a lightweight application programming interface). Thus, the archive processor may be able to select, collect, store, protect, etc., data from various locations for long-term storage. To achieve this functionality, the archive processor may include a set of observer components and / or remover components implemented as distributed microservice code at various locations, for example, as will be described in further detail below.

[0101] At each corresponding edge node 304, 306, and 308, data storage 318, 322, and 326 can implement a corresponding file system, and each file system can be associated with... Figure 3B The file system 311 seen in the diagram shares certain aspects. For example, each data storage 318, 322, 326 may include, for instance, […]. Figure 3B The subfolders, layers, and directory locations seen in the image (e.g., subfolders). Figure 3A The data storage 318, 322, 326 may also include an observer component for each directory location included at each edge node 304, 306, 308. Thus, a new observer component is created in response to identifying a new subfolder introduced into the file system of the corresponding edge node 304, 306, 308.

[0102] Each edge node 304, 306, 308 may also include a remover component. The remover component can be used to determine whether a temporary copy of the data stored at one of the edge nodes 304, 306, 308 has been stored for a predetermined amount of time. In some cases, it is desirable for certain data to remain available at the edge node where the data is received and / or generated. However, the data is eventually removed from the edge node to ensure that storage capacity is freed up over time to accommodate newly received and / or generated data. Therefore, the remover component can monitor whether a copy of the data is still stored at the edge node for at least a predetermined amount of time, for example, as will be explained below. Figure 4 The method is described in further detail in 400.

[0103] While edge nodes 304, 306, and 308 are depicted as including similar components and / or designs, it should be noted that each edge node 304, 306, and 308 may include any desired components, which may be implemented in any desired configuration. In some instances, each edge node in the system may be configured differently to provide different functionality for each location. According to examples that are by no means intended to limit the invention, edge node 304 may include an encryption module (not shown) that allows edge node 304 to generate encrypted data, while edge node 306 includes a data compression module (not shown) that allows edge node 306 to generate compressed data.

[0104] Therefore, different edge nodes (e.g., servers) in system 300 can have different performance capabilities. As mentioned above, the central data storage location 302 can have a higher storage capacity compared to edge nodes 304 and 306. While this allows the central data storage location 302 to store more data than edge nodes 304, 306, and 308, other factors affect performance. For example, traffic on network 310 can limit the amount of data that can be sent from edge nodes 304, 306, and 308 to the central data storage location 302. The workload experienced at a given time also affects latency and limits the achievable performance of system 300.

[0105] While these performance characteristics of the changes have a substantial impact on the efficiency of the system's operation, the implementation presented in this paper can ideally orchestrate data availability, enabling the system to operate more efficiently while improving data retention. For example, different types of data can be used across the system at different locations. As mentioned above, in addition to being requested by downstream applications at the central data storage location, data can also be generated and processed at edge nodes. Therefore, the method presented in this paper ensures that data is available at certain locations while improving system operational efficiency and data security by maintaining backup copies of the data at edge nodes for at least a period of time.

[0106] Some implementations described in this paper can even tune performance by monitoring and responding to changes in bandwidth limitations, high-quality real-time media streaming, excessive latency, network congestion, and so on. Therefore, the implementations can dynamically update system settings to maintain a relatively high level of efficiency.

[0107] Looking at it now Figure 4 A method 400 for implementing an autonomous data archiving procedure across a distributed system is illustrated according to a method. Therefore, the operations of method 400 can be continuously executed in the background of the operating system without requesting input from a user (e.g., a human). Although some information (e.g., warnings, reports, read requests, etc.) may be issued to the user, it should be noted again that various operations of method 400 can be repeated iteratively for each instance of new data detected at any edge node connected to the central data storage location. Therefore, various methods according to the invention can be implemented... Figure 1 Method 400 is performed in any of the environments shown in -3, as well as in other environments. Of course, as those skilled in the art will understand after reading this description, method 400 may include more than Figure 4 More or fewer operations are specifically described in the text.

[0108] Each step of method 400 can be performed by any suitable component of the operating environment. For example, each of nodes 401, 402, 403 shown in the flowchart of method 400 can correspond to one or more processors located at different locations in a multi-tiered data storage system. Furthermore, each of the one or more processors is preferably configured to communicate with each other.

[0109] In another example, at least one of nodes 401, 402, and 403 may include an archive processor configured to manage file hierarchies at edge nodes and / or mirrored file hierarchies at centralized data vaults. In some approaches, the archive processor includes a software application designed to manage archives (e.g., file hierarchies or file systems). Therefore, the archive processor may be able to select, collect, store, protect, etc., data for long-term storage. To achieve this functionality, the archive processor may include a set of observer components and / or remover components, for example, as will be described in further detail below.

[0110] In various implementations, method 400 may be executed, in part or in whole, by a controller, processor, or other device having one or more processors. A processor (e.g., processing circuitry, a chip, and / or a module) implemented in hardware and / or software and preferably having at least one hardware component can be utilized in any device to perform one or more steps of method 400. Illustrative processors include, but are not limited to, central processing units (CPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), combinations thereof, or any other suitable computing device known in the art.

[0111] As mentioned above, Figure 4 This includes different nodes 401, 402, and 403, each representing one or more processors, controllers, computers, etc., located at different locations in the distributed data storage system. For example, node 401 may include one or more processors located at the central data storage location (e.g., the master server) of the distributed computing system (see above). Figure 3A The central data storage location 302). Node 402 may include one or more processors located at the first edge node (e.g., edge server) of the distributed computing system (see above). Figure 3A Edge node 304). Furthermore, node 403 may include one or more processors located at a second edge node of the distributed computing system (see above for example). Figure 3A (Edge node 306). Therefore, depending on the method, commands, data, requests, etc., can be sent between each node 401, 402, 403.

[0112] Furthermore, it should be noted that the various processes included in method 400 are by no means intended to be limiting, for example, as those skilled in the art will understand after reading this description. For instance, in some methods, data sent from node 402 to node 403 may be initiated by a request sent from node 403 to node 402. Additionally, Figure 4 The number of nodes included is by no means intended to be a limitation. For example, in some methods, third edge nodes may be included (see above for example). Figure 3A (Edge node 308). Therefore, any desired number of edge nodes can be connected to the central data storage location, for example, as those skilled in the art will understand after reading this description.

[0113] It should also be noted that each node 402, 403 (e.g., an edge node) preferably includes a remover component and multiple observer components. In this description, "component" is intended to refer to a function or part of software that includes a process that produces the desired result. Thus, the remover component may be a function operating in the background of the corresponding edge node, for example, at predetermined intervals. Similarly, each observer component may be a function operating in the background of the corresponding edge node, for example, continuously operating in an iterative manner.

[0114] look Figure 4 Method 400 includes operation 404 performed at node 402. There, operation 404 includes detecting new data generated at node 402. As described above, node 402 can correspond to an edge node in a distributed system. Node 402 can therefore receive data over time from one or more locations (e.g., sensors). The data received at the edge node can also be processed there (e.g., evaluated) to distribute the data processing overhead to the edge nodes, thereby allowing for increased throughput of the system as a whole. Therefore, as information is processed, node 402 can also generate data over time.

[0115] According to one example, which is by no means intended to limit the invention, node 402 may include multiple observer components. As described above, each observer component may correspond to a different folder (e.g., a subfolder) in the file system of the data storage at the edge node. Thus, any data added to one of the folders in the file system at the edge node can be identified by the corresponding observer component assigned to that folder. Thus, operation 404 can detect new data generated in a subfolder at node 402 in response to receiving a notification from the corresponding one of the observer components.

[0116] Beginning at operation 404, method 400 proceeds to operation 406, which includes transmitting a first copy of the new data to the central data storage location at node 401. The first copy of the new data is preferably sent to the central data storage location in real time. This ideally allows a mirror copy of the data received and / or generated at each edge node connected to the central data storage location to be maintained at the central data storage location. Therefore, any errors experienced at the edge nodes (e.g., data loss, system failure, power outage, etc.) will not impair the overall system functionality. The central copy of the data can be used to dynamically adjust how each edge node operates, thereby overcoming any such errors at the edge nodes.

[0117] In one example, the process of sending a first copy of new data to the central data storage location includes storing the first copy in a "DataOut" directory configured to automatically send any data stored therein to the central data storage location. Therefore, storing a first copy of new data in the "DataOut" directory at node 402 automatically causes the first copy to be sent to node 401. See again operation 406. Thus, in some implementations, operation 406 may also include removing (e.g., deleting) the first copy of the new data from the "DataOut" directory in response to node 401 receiving the copy.

[0118] In response to receiving a first copy of new data from node 402, node 401 performs operation 408. There, operation 408 includes storing the first copy of the data in a first data storage location at the central data storage location. As mentioned above, the type and / or source of the data can influence where or how the data is stored at the central data storage system. For example, the type of data received in operation 406 can determine which file system directory folder the data is stored in at the central data storage location. Therefore, operation 408 may include examining the data details to determine the appropriate storage location.

[0119] Returning to node 402, operation 410 is performed in response to the first copy of the new data being transferred to the central data storage location at node 401. There, operation 410 includes storing a second copy of the new data at node 402. Since operations 408 and 410 are performed by different components at different nodes, at least a portion of operations 408 and 410 can be performed in parallel. Therefore, the first copy of the new data can be stored in the central data storage location at least partially in parallel with the second copy of the new data stored at the source edge node. In one example, the second copy of the new data is stored in a "DataHolding" directory configured to store the second copy of the new data for a predetermined amount of time, as will become apparent, for example, as follows.

[0120] Upon proceeding to operation 412, method 400 includes waiting for a predetermined amount of time before proceeding to operation 414. In other words, operation 412 includes a predetermined timeout, which allows a copy of the data stored at node 402 to remain in the data storage before being removed from node 402. As mentioned above, in some cases, data received and / or generated at edge nodes may be used for additional processing. This data may also be needed to correct one or more errors experienced. For example, a copy of the data sent from node 402 in operation 406 may not have been fully received at node 401 due to transmission errors, data write errors, etc. Therefore, by maintaining a copy of the new data sent in operation 406 at node 402 for at least a predetermined amount of time, method 400 is able to respond in real time to many data management situations without requesting a copy of the new data to be returned from the central data storage location. This allows the implementation of this paper to improve system performance by significantly reducing the latency experienced by conventional systems.

[0121] The amount of time that data is retained in storage at edge nodes varies depending on several factors. For example, the amount of time a copy of data (e.g., a copy of a file) is maintained at an edge node after another copy of the data has been sent to a central data storage location is predetermined, at least in part, based on the type of information included in the data copy. Some types of data may be more difficult to process, and therefore copies of such data can be maintained in storage for a longer period compared to other types. Furthermore, some data types can be handled with more secure procedures than others, resulting in different types of data being maintained at edge nodes for different amounts of time.

[0122] In other implementations, the amount of time a data copy is maintained at the edge node is predetermined, at least in part, based on the amount of information included in the data copy. In other words, the size of the data copy can affect the amount of time it is maintained in storage at the edge node. These relationships can be predefined by the user based on dynamic system conditions such as the amount of data stored at the edge node, the current workload, user input, etc. In other approaches, the data copy can be maintained at the edge node for a predetermined number of data operations, for a predetermined number of data errors experienced, for outputs based on machine learning models, etc.

[0123] Therefore, adaptive functions can be used to determine the amount of time a given copy of data is stored at an edge node. This function can be adaptively applied to different types of data (e.g., files) based on operating parameters, which can include the type and / or amount of data held at the edge node. In some implementations, a discrete function with a set of variables corresponding to the characteristics of the data file (e.g., file type, file size, etc.) can be implemented. Depending on the approach, the function can be implemented as a conditional statement, organized as values ​​in a lookup table, etc.

[0124] In other approaches, the amount of time a given data copy is stored at an edge node is determined by implementing one or more machine learning models. These machine learning models can be trained to generate the expected amount of time a backup copy of data sent to a central remote vault is held at the edge node. For example, the machine learning model (e.g., a neural network) can be trained using labeled and / or unlabeled data corresponding to the past performance of a distributed system implementing any of the processes described herein. Over time, the machine learning model is thus able to recognize when a backup copy of data sent to a central data vault is referenced. This understanding will allow the machine learning model to determine the ideal amount of time to hold each data copy in storage at the corresponding edge node. The machine learning model achieves this by balancing the impact of each copy on the data storage capacity at the edge node with the number of data errors experienced at the central data vault and / or additional processing performed on the data stored at the edge node, as will be understood by those skilled in the art upon reading this description.

[0125] In other implementations, the user specifies the amount of time a data copy is held at the edge node. However, if no user input is received, a predefined value can be implemented as a default value. Operation 414 then determines whether a second copy of the new data should be removed from the edge node at node 402. Although operation 412 includes waiting for the predefined amount of time as described above, some data copies may remain in storage at the edge node. For example, it can be determined that the corresponding data copy has not yet been successfully stored at the central data storage location, for example, due to network traffic, computational congestion, etc., at the central data storage location. Therefore, in response to determining that a second copy of the new data should not be removed from the edge node at node 402, method 400 returns to operation 412.

[0126] In response to determining that a second copy of the new data should be removed from the edge node at node 402, method 400 proceeds from operation 414 to operation 416. There, operation 416 includes determining whether any information is missing from the first copy of the new data delivered to node 401. In other words, operation 416 includes determining whether any data is missing at the central data storage location. In some methods, operation 416 includes sending one or more queries to the central data storage location. For example, this determination may be made by verifying whether any files from the central data storage location are missing. Furthermore, this verification can be performed using information collected by the remover component, for example, as an initial part of executing the removal procedure in the corresponding DataHolding directory.

[0127] Therefore, in some implementations, operation 416 includes sending a request to node 401 to determine whether the central data storage location includes a mirror copy of the first copy of the data sent in operation 406 above. See, for example, dashed line 416a. In response to receiving a request from the edge node to determine whether a mirror copy of the data received from the edge node has been successfully stored, the central data storage location may check one or more directory locations in the storage to determine whether a complete copy of the new data initially sent from the edge node is secure.

[0128] In other words, the task of the central data storage location is to determine whether a complete copy of the new data is retained in the storage there. In some instances, some data may be lost during transmission due to encryption and / or decryption errors, write errors, connectivity problems, etc. Therefore, although a complete copy of the data may be successfully sent from the edge node, it may not be successfully stored in the storage at the central data storage location in every case. Therefore, evaluating the data that has been successfully stored at the central data storage location ensures that a mirror (e.g., intact) copy of the new data sent from the edge node has been successfully written to storage. The result of determining whether a complete copy of the data has been written to the storage at node 401 is then returned to the edge node at node 402, as shown by dashed line 416b.

[0129] If, at operation 416, it is determined that at least a portion of the data initially sent from one of the edge nodes has not been successfully delivered to the central data storage location, method 400 proceeds to operation 418a. In other words, in response to determining that the data stored at the central data storage location is at least partially incomplete, method 400 proceeds to operation 418a. There, operation 418a includes issuing a request to the first edge node for the missing portion of the data. However, in some implementations, the information returned along the dashed line 416b above may also include a request for the portion of data identified as missing from the central data storage location.

[0130] In response to receiving a request for a missing portion of the data, node 402 preferably accesses a copy of the initially sent new data (a second copy of the new data) still stored at node 402. As described above, it is preferable to store the second copy of the data sent from the edge node to the central data storage location at the edge node for a period of time. Therefore, the backup copy of the data can be used if a portion of the data missing from the central data storage location is determined.

[0131] In other words, maintaining a second copy of the new data at the edge node before removing the second copy of the data from the temporary directory location helps ensure that the central data storage location includes a complete copy of the data received and / or processed by the system (e.g., see system 300 in Figure 3).

[0132] Therefore, operation 418b includes returning the missing portion of the requested data to node 401. In response to receiving the missing portion of the data, node 401 also includes using the missing portion to complete a mirror copy of the data. See operation 420. For example, a portion of the data at the central data storage location may be rewritten to produce a complete copy of the new data intended to be sent in operation 406.

[0133] As described above, the central data storage location preferably stores mirrored and real-time copies of data received and / or generated at various edge nodes. This allows the overall system to significantly improve data retention by maintaining a complete and up-to-date copy of the data at a central location. Maintaining a complete copy of the data at a central and accessible location also significantly improves the efficiency of the system as a whole. As mentioned above, data generated at edge nodes can be requested at different downstream locations in the distributed system, for example, to perform additional processing. Therefore, by sending copies of data from edge nodes to the central data storage location in real time while receiving and / or generating data, the central data storage location is able to satisfy operations and requests related to data from one or more edge nodes.

[0134] Continue to refer to Figure 4 In response to sending the missing portion of the data to node 401, method 400 proceeds from operation 418b to operation 422. However, in some methods, edge node 402 may send a verification request to node 401 to verify that the copy of the new data at the central data storage location is now a complete, mirror copy of the data received and / or generated at the corresponding edge node. The verification request can effectively serve as a release verification for the edge node at node 402. Therefore, operation 422 includes removing the copy of the new data still stored in the edge node at node 402 in response to successfully transmitting the missing information to node 402.

[0135] In some methods, causing a copy of new data to be removed from storage at the edge node includes sending one or more instructions to a remover component contained at the edge node. As mentioned above, the remover component can be a function operating in the background of the corresponding edge node, for example, at predetermined intervals, randomly, in response to input received from a user, etc. The one or more instructions are preferably configured to cause the remover component to remove the referenced data copy from the corresponding storage location.

[0136] Depending on the method, data copies can be removed from edge nodes by deleting data from storage, invalidating data stored in a specific part of storage, or marking a portion of storage containing the data copy as garbage collection. Therefore, the steps taken to remove data copies from edge nodes can depend on the data type contained within the edge node.

[0137] After determining that the central data storage location includes a complete copy of the data, temporary copies of the new data stored at edge nodes that receive data from sensors and / or generate data by processing additional information can be removed to accommodate additional copies of the data. Therefore, method 400 ensures the data security and efficient performance of the system as a whole. As mentioned above, maintaining up-to-date and accurate copies of data in a centralized and accessible location allows for reduced data access time, thereby enabling more efficient process execution and more efficient operation of the system as a whole.

[0138] This, combined with maintaining a copy of the data at the source location for at least a period of time, further improves performance. Similarly, this temporary copy of the new data acts as a backup in case any data is lost during transmission to the central data storage location, for example, due to encryption and / or decryption errors, network errors, write errors, etc. Therefore, maintaining a temporary copy of the data increases data security. Additionally, the temporary copy keeps the corresponding data available at the edge location for a period of time. In some cases, even after a copy of the new data has been sent to the central data storage location, supplementary applications, programs, etc., that reference (e.g., use) the data in the temporary copy are executed at the edge nodes. Therefore, maintaining a backup copy of the new data at the edge nodes for a period of time reduces data access time and improves the efficiency of supplementary applications, programs, etc.

[0139] Returning to operation 416, in some cases it is determined that the first storage location does indeed include an updated (real-time) and mirrored (accurate) copy of the data initially sent in operation 406. In such a case, a release confirmation can be sent to the first edge node, causing the additional copy of the new data maintained at node 402 to be released. Therefore, in such a case, method 400 proceeds directly from operation 416 to operation 422.

[0140] As can be seen, Method 400 can maintain mirror copies of data received and / or generated at various edge nodes, thereby achieving an efficient and scalable data archiving scheme. Therefore, this scheme can be applied to various types of implementations, such as enterprise edge node workloads. The method in this paper is also easily scalable, supporting multiple edge nodes and multiple edge workloads even in large enterprise setups.

[0141] Improve system performance by implementing an archive processor that can be easily deployed as containerized edge workloads to a variety of edge nodes. This helps make data available for downstream real-time consumption while protecting edge nodes from unwanted access. Data remains available at the edge nodes as expected, determined in several different ways, as described in this paper. This further minimizes storage costs at each edge node.

[0142] Furthermore, the method presented in this paper significantly improves data retention by maintaining a complete and up-to-date copy of the data from numerous edge nodes at a central location. Maintaining a complete copy of the data at a central and accessible location also significantly improves the efficiency of the system as a whole by providing faster access to the data. As mentioned above, data generated at edge nodes can be requested at different downstream locations in the distributed system, for example, to perform additional processing. Therefore, by sending copies of the data from the edge nodes to the central data storage location in real time as data is received and / or generated, the central data storage location is able to satisfy operations and requests related to data from one or more edge nodes.

[0143] Continue to refer to Figure 4 It should be noted that node 403 may include any one or more operations described above with respect to node 402. Therefore, node 403 is shown as implementing variations of the operations performed in node 402. Thus, the various operations performed at node 403 share the same numbering as the various operations performed at node 402. It is thus evident that any operation performed at node 403 can implement any aspect of the various implementations included herein. In other words, the operations performed at node 403 may be the same as or similar to the operations presented at node 402, as will be understood by those skilled in the art upon reading this description.

[0144] Specifically, node 403 is shown detecting new data generated at node 403. See operation 404'. From operation 404', method 400 proceeds to operation 406', which includes transmitting a first copy of the new data to node 401. In response to receiving the first copy of the new data from node 403, node 401 performs operation 408'. There, operation 408 includes storing the first copy of the data in a second storage location at the central data storage location. As mentioned above, the type and / or source of the data may affect where or how the data is stored at the central data storage system. For example, the type of data received in operation 406' can determine which file system directory folder the data is stored in at the central data storage location. Therefore, operation 408' may include examining the data details to determine the appropriate storage location.

[0145] Returning to node 403, in response to the first copy of the new data being transferred to the central data storage location at node 401, operation 410' is executed. There, operation 410' includes storing a second copy of the new data at node 403. Because operations 408' and 410' are executed by different components at different nodes, at least a portion of operations 408' and 410' can be executed in parallel. Therefore, the first copy of the new data can be stored at the central data storage location, at least partially, in parallel with the second copy of the new data stored at the source edge nodes. In one example, the second copy of the new data is stored in a "DataHolding" directory, which is configured to store the second copy of the new data for a predetermined amount of time, for example, as will soon become apparent.

[0146] Upon proceeding to operation 412', method 400 includes waiting for a predetermined amount of time before proceeding to operation 414'. Operation 414' includes determining whether a second copy of the new data should be removed from the edge node at node 403. Therefore, in response to determining that a second copy of the new data should not be removed from the edge node at node 403, method 400 returns to operation 412'.

[0147] In response to determining that a second copy of the new data should be removed from the edge node at node 402, method 400 proceeds from operation 414' to operation 416'. There, operation 416' includes determining whether any information is missing from the first copy of the new data transmitted to node 401. In some cases, it is determined that node 401 currently includes an updated (real-time) and mirrored (accurate) copy of the data initially sent in operation 406'. In such a case, a release confirmation can be sent to the first edge node, thereby releasing the additional copy of the new data maintained at node 403. Therefore, in such a case, method 400 proceeds directly from operation 416' to operation 422'.

[0148] However, if at operation 416' it is determined that at least a portion of the data initially sent from one of the edge nodes has not been successfully transmitted to the central data storage location, method 400 proceeds to operation 418a'. There, operation 418a' includes issuing a request to the first edge node for the missing portion of the data.

[0149] In response to a request for a missing portion of the data, node 403 preferably accesses a copy of the initially sent new data (a second copy of the new data) still stored at node 403. As described above, the second copy of the data sent from the edge node to the central data storage location is preferably stored at the edge node for a period of time. Therefore, in the event that a missing portion of the data is determined to be in the central data storage location, this backup copy of the new data can be used.

[0150] In other words, maintaining a second copy of the new data at the edge nodes before removing the second copy of the data from the temporary directory location helps ensure that the central data storage location includes a complete copy of the data received and / or processed by the system (e.g., see system 300 in Figure 3).

[0151] Therefore, operation 418b' includes returning the missing portion of the requested data to node 401. In response to receiving the missing portion of the data, node 401 also includes using the missing portion to complete a mirror copy of the data. See operation 420'. In response to sending the missing portion of the data to node 401, method 400 proceeds from operation 418b' to operation 422'. There, operation 422' includes removing a copy of the new data still stored in the edge node of node 402 in response to successfully transmitting the missing information to the central data storage location node 402.

[0152] Looking at it now Figure 5 This illustrates an example of the use of an autonomous program 500 that can be implemented between an edge node and a central data repository. For example, one or more operations included in the autonomous program 500 can be orchestrated by an archive processor, as described in the methods herein. Thus, the autonomous program 500 can be implemented as software running in the background, for example, in response to the detection of new data at an edge node.

[0153] Therefore, operation 502 of procedure 500 includes initiating file synchronization from the edge workload to the central data vault. New data can be received and / or created at the edge nodes in many cases. For example, new data can be received from one or more sensors, sensor readings can be compiled to create new videos represented as data, sensor readings can be evaluated using one or more trained machine learning models to generate new analyses of the sensor readings, etc.

[0154] As described above, new data is preferably transmitted to a central data repository (e.g., a data storage location) in real time to maintain mirror copies of the data received and / or created at multiple edge nodes communicating with the central data repository. However, the new data is also preferably retained at the edge nodes for at least a predetermined amount of time. Therefore, a first copy of the new data is written to the "DataHold" edge node directory, and a second copy of the new data is written to the "DataOut" edge node directory. An acknowledgment is returned to the edge workload to confirm receipt of the instruction and / or successful completion of the data write.

[0155] In one example, which is by no means intended to limit the invention, the file path of the DataOut edge node directory would be:

[0156] DataOUT / [NodeID] / [EdgeWorkLoadID] / [DataCat] / filename,

[0157] The file path for the DataHold edge node directory will be:

[0158] DataHolding / [NodeID] / [EdgeWorkLoadID] / [DataCat] / filename. Here, "NodeID" represents a unique identifier for the edge node, while "EdgeWorkLoadID" represents a unique identifier for the edge workload (e.g., quality inspection). Furthermore, "DataCat" indicates that the edge workload can generate various file types. For example, raw images, annotated images, .json files, etc., from workloads involving computer vision AI object detection can be used. Therefore, DataCat provides separation for different file types as desired. Thus, the lowest-level folder can be considered to include either the [EdgeWorkLoad] level or the [DataCat] level. As described in this article, folders at the lowest level are also referred to as "subfolders" in this document.

[0159] Still referencing Figure 5 The observer component assigned to the "DataOut" edge node directory identifies when new data has been added. In response to the detection of new data in the "DataOut" edge node directory, the new data is automatically sent to the observer component and moved to the central data vault. In other words, new data added to the "DataOut" edge node directory is automatically copied to the central data vault. An acknowledgment is returned from the central data vault, allowing any residual copies of the new data at the "DataOut" edge node directory to be deleted. In other implementations, copies of the data in the "DataOut" edge node directory can be deleted in response to sending a copy to the central data vault. The observer component also writes to a log and returns an acknowledgment (e.g., confirmation).

[0160] In response to the fulfillment of predetermined conditions (e.g., a predetermined amount of time has elapsed since the data was added to the “DataHold” edge node directory), the remover component verifies the integrity of the data stored in the central data vault. As mentioned above, at least a portion of the data sent to the central data vault may fail to be successfully stored in the data store. Therefore, by verifying that a complete (mirror) copy of the data is stored in the central data vault, program 500 is able to reduce data loss and operational errors caused by missing or corrupted data. Maintaining a copy of the data in the central data vault in real time also allows downstream processing (e.g., applications) to operate more efficiently. For example, a real-time mirror copy of the data at the central data vault allows for more efficient additional processing of the data compared to if the data is stored at one or more edge nodes. Thus, computationally intensive operations such as large (complex) machine learning models can be performed on any data in a distributed system much faster than previously possible.

[0161] Furthermore, maintaining temporary copies of new data on edge nodes before removal allows for easy mitigation of data errors experienced at the central data repository. These temporary backup copies can also be used at the respective edge nodes to perform additional processing there. Therefore, edge nodes and the central data repository can process data simultaneously and in parallel, further improving system efficiency. In other words, more operations can be performed in a given amount of time due to the ability to execute operations in parallel and also due to a significant reduction in data access time at edge nodes and the central data repository.

[0162] Therefore, if the central data vault indicates that at least some data is missing, the remover component can send a request for the missing data to the "DataHold" edge node directory. The "DataHold" edge node directory then returns the requested missing data, which is sent to the central data vault. After verifying that the central data vault contains a complete copy of the data, the "DataHold" edge node directory is instructed (e.g., permitted) to delete a backup copy of the new data still stored therein. Finally, an acknowledgment is returned to the remover component, and the log is updated.

[0163] Looking at it now Figure 6 This illustrates a distributed system with edge nodes 600 and a central data repository 650 according to an illustrative method. Alternatively, aspects of this illustrative method can be implemented in combination with features from any other implementation listed herein, such as those described with reference to other figures, for example... Figures 1-5 .However, Figure 6The illustrative methods shown, as well as other methods presented herein, can be used in various applications and / or arrangements, which may or may not be specifically described in the illustrative methods listed herein. Furthermore, the illustrative methods described below can be used in any desired context. Therefore, Figure 6 (And other accompanying figures) can be considered to include any possible permutation.

[0164] like Figure 6 As shown, workloads 1 and 2 are being executed at edge node 600. As workloads 1 and 2, a first copy of the new data is added to the DataHold edge node directory 602, while a second copy of the new data is added to the DataOut edge node directory 604. Specifically, the first copy of the new data generated by workload 1 is sent to workload 1 layer 606 under the DataHold edge node directory 602, and the second copy of the new data generated by workload 1 is sent to workload 1 layer 608 under the DataOut edge node directory 604. Similarly, workload 2 sends copies of the generated new data to the corresponding layers 610 and 612, as illustrated.

[0165] Preferably, data added to different layers 606, 608, 610, and 612 is separated based on type. In other words, data is separated between different layers 606, 608, 610, and 612, such that each layer includes different types of data accumulated within it. Therefore, each of layers 606, 608, 610, and 612 also includes multiple directory locations 614, 616, and 618, each corresponding to a different type of data. It should be noted that although each directory location is numbered the same for each of layers 606, 608, 610, and 612, this is by no means intended to be restrictive. Rather, any layer at an edge node can include data of any desired type. Therefore, the number of directory locations can change and be updated over time as new types of data are received. It should also be noted that directory locations 614, 616, and 618 are also referred to as “subfolders” in some methods described herein.

[0166] Still referencing Figure 6The diagram illustrates how directory locations 614, 616, and 618 under the DataOut edge node directory 604 use network 625 to directly send data stored therein to the corresponding portions of the file system 652 at the remote data vault 650. Specifically, the dashed arrows are shown extending from each directory location 614, 616, and 618 under the DataOut edge node directory 604 to the corresponding observer component of the archive processor 601. Thus, the observer component can be configured to continuously monitor the corresponding directory locations 614, 616, and 618 under the DataOut edge node directory 604 for newly introduced data. In response to identifying new data in one or more of the directory locations, the corresponding observer component preferably pushes the new data automatically and in real time to the file system 652 at the remote data vault 650. Therefore, the remote data vault 650 is able to maintain an accurate copy of the data generated at each edge node of the system.

[0167] Additionally, the archive processor 601 includes a remover component. As described above, the remover component can monitor backup (e.g., temporary) copies of data stored in directory locations 614, 616, and 618 under the DataHold edge node directory 602. In response to a predetermined amount of time that a copy of the data has been maintained in one of the directory locations 614, 616, and 618 under the DataHold edge node directory 602, the remover component sends one or more instructions along the dotted line 620 to remove the copy from the corresponding directory location. However, before removing the copy of the data from the edge node 600, a verification can be sent to the remote data vault 650 to determine whether a complete and up-to-date mirror copy of the data received at the edge node 600 and / or generated by the edge node 600 (and / or other edge nodes connected to the remote data vault 650) is stored at the remote data vault 650.

[0168] Similarly, the method presented in this paper aims to maintain mirrored copies of data received and / or generated at various edge nodes, thereby enabling an efficient and scalable data archiving scheme across distributed systems. Therefore, this scheme can be applied to various types of implementations, such as enterprise edge node workloads. The method is also easily scalable, providing support for multiple edge nodes and multiple edge workloads even in large enterprise setups.

[0169] The system's performance is also improved by implementing an archive processor that can be easily deployed as containerized edge workloads to a variety of edge nodes. This helps make data available for real-time downstream consumption while protecting edge nodes from unwanted access. Data remains available at the edge nodes as expected, as determined in many different ways, as described in this paper. This further minimizes storage costs at each edge node.

[0170] Similarly, the method presented in this paper significantly improves data retention by maintaining a complete and up-to-date copy of data from multiple edge nodes at a central location. Maintaining a complete copy of data at a central and accessible location also significantly improves the efficiency of the system as a whole by providing faster access to the data. As mentioned above, data generated at edge nodes can be requested at different locations downstream of the distributed system, for example, to perform additional processing. Therefore, by sending copies of data from edge nodes to a central data storage location in real time as data is received and / or generated, the central data storage location can satisfy operations and requests related to data from one or more edge nodes.

[0171] By implementing one or more machine learning models, certain aspects of the implementation described in this paper can be further improved. These machine learning models can be trained to generate backup copies of data sent to a central remote vault and kept at the edge nodes (e.g., see above). Figure 6 The desired amount of time for the DataHold edge node catalog (602). For example, a machine learning model (e.g., a neural network) can be trained using labeled and / or unlabeled data corresponding to the past performance of a distributed system implementing any of the processes described herein. Over time, the machine learning model can thus be able to identify when to reference a backup copy of the data sent to a central data vault. This understanding will allow the machine learning model to determine the ideal amount of time to keep a copy of each piece of data in storage at the respective edge node. The machine learning model achieves this by balancing the impact of each copy on the data storage capacity at the edge node with the number of data errors experienced at the central data vault and / or the additional processing performed on the data stored at the edge node, for example, as those skilled in the art will understand after reading this description.

[0172] It should also be noted that the implementation of this paper can use an application programming interface (API) that is configured to collect the outputs (e.g., vectors) of layers to be used as inputs to subsequent layers (e.g., machine learning models) in a process. Therefore, the API enables seamless data transfer between layers in edge servers and central servers. The API can thus be used to help facilitate data movement and the flow of methods, as those skilled in the art will understand after reading this description. The API generalizes the definition of parameters to be passed between servers, allowing it to accommodate tensors of all different sizes based on the size of the output layer of each server.

[0173] It will be clear that the various features of the aforementioned systems and / or methods can be combined in any way, thereby creating a variety of combinations from the description given above.

[0174] It will also be understood that embodiments of the invention may be provided as services deployed on behalf of customers to provide on-demand services.

[0175] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or to limit the embodiments to what is disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, practical applications of technology found in the market, or technical improvements, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: Detect new data generated at the first edge node; This allows a first copy of the new data to be transmitted to the central data storage location in real time. The second copy of the new data is stored at the first location on the first edge node; In response to the fulfillment of predetermined conditions, it is determined whether any information in the first copy of the transmitted new data is missing at the central data storage location; as well as In response to information that there is no missing transmission of the first copy of the new data at the central data storage location, the first copy of the new data is removed from the first location at the first edge node.

2. The computer-implemented method of claim 1, wherein the central data storage location is configured to receive and store one or more additional copies of data received from one or more additional edge nodes.

3. The computer-implemented method of claim 2, wherein the central data storage location is configured to maintain a mirror copy of the data received at one or more corresponding additional edge nodes.

4. The computer-implemented method according to any of the preceding claims, comprising: In response to determining that at least some information in the first copy of the new data is missing at the central data storage location, the missing information is transmitted to the central data storage location; as well as In response to transmitting the missing information to the central data storage location, the first copy of the new data is removed from the first location.

5. The computer-implemented method according to any of the preceding claims, wherein the method is performed by an archive processor configured to manage an edge file hierarchy at the first edge node, wherein the archive processor includes a set of observer components and a remover component.

6. The computer-implemented method of claim 5, wherein each observer in the observer component set corresponds to a corresponding folder at the first edge node, wherein the new data generated at the first edge node is detected in response to receiving a notification from a corresponding observer component in the observer components.

7. The computer-implemented method of claim 5, wherein removing the first copy of the new data from the first location at the first edge node comprises sending one or more instructions to the remover component, the one or more instructions causing the remover component to remove the first copy of the new data from the first location.

8. The computer-implemented method according to any of the preceding claims, wherein satisfying the predetermined condition includes maintaining the first copy of the new data at the first position at the first edge node for a first predetermined amount of time.

9. The computer-implemented method of claim 8, wherein the first predetermined time amount is based at least in part on the type and / or amount of information included in the first copy of the new data.

10. A computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being readable by a processor, executable by a processor, or readable and executable by a processor to cause the processor to: Detect new data generated at the first edge node; This allows a first copy of the new data to be transmitted to the central data storage location in real time. The second copy of the new data is stored at the first location on the first edge node; In response to the fulfillment of predetermined conditions, it is determined whether any information in the first copy of the transmitted new data is missing at the central data storage location; as well as In response to information that there is no missing transmission of the first copy of the new data at the central data storage location, the first copy of the new data is removed from the first location at the first edge node.

11. The computer program product of claim 10, wherein the central data storage location is configured to receive and store one or more additional copies of data received from one or more additional edge nodes.

12. The computer program product of claim 11, wherein the central data storage location is configured to maintain a real-time mirror copy of the data received at one or more corresponding additional edge nodes.

13. The computer program product according to any one of claims 10 to 12, wherein the program instructions are readable and / or executable by the processor to cause the processor to: In response to determining that at least some information in the first copy of the new data is missing at the central data storage location, the missing information is transmitted to the central data storage location; and In response to transmitting the missing information to the central data storage location, the first copy of the new data is removed from the first location.

14. The computer program product according to any one of claims 10 to 13, wherein the method is performed by an archive processor configured to manage an edge file hierarchy at the first edge node, wherein the archive processor includes a set of observer components and a remover component.

15. The computer program product of claim 14, wherein each observer in the observer component set corresponds to a corresponding folder at the first edge node, wherein the new data generated at the first edge node is detected in response to receiving a notification from a corresponding observer component in the observer components.

16. The computer program product of claim 14, wherein removing the first copy of the new data from the first location at the first edge node comprises sending one or more instructions to the remover component, the one or more instructions causing the remover component to remove the first copy from the first location.

17. The computer program product according to any one of claims 10 to 16, wherein satisfying the predetermined condition includes maintaining the first copy of the new data at the first position at the first edge node for a first predetermined amount of time.

18. The computer program product of claim 17, wherein the first predetermined time amount is based at least in part on the type and / or amount of information included in the first copy of the new data.

19. A system comprising: The first edge node connected to the central data storage location; processor; as well as Logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, wherein the logic is configured to: Detect new data generated at the first edge node; This allows a first copy of the new data to be transmitted to the central data storage location in real time. The second copy of the new data is stored at the first location on the first edge node; In response to the fulfillment of predetermined conditions, it is determined whether any information in the first copy of the transmitted new data is missing at the central data storage location; as well as In response to information that there is no missing transmission of the first copy of the new data at the central data storage location, the first copy of the new data is removed from the first location at the first edge node.

20. The system of claim 19, wherein the logic is configured to: In response to determining that at least some information in the first copy of the new data is missing at the central data storage location, the missing information is transmitted to the central data storage location; and In response to transmitting the missing information to the central data storage location, the first copy of the new data is removed from the first location. The central data storage location is configured to receive and store one or more additional copies of data received from one or more additional edge nodes. The central data storage location is configured to maintain a real-time mirror copy of the data received at one or more corresponding additional edge nodes.

21. A computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being readable by a processor, executable by the processor, or readable and executable by the processor to cause the processor to: Receive the first copy of the data from the first edge node; The first copy of the data is stored in a first storage location at the central data storage location; Receive a second copy of the data from the second edge node; The second copy of the data is stored in a second storage location at the central data storage location; Receive a request from the first edge node to determine whether the first storage location includes a mirror copy of the first copy of the data received from the first edge node; as well as In response to determining that the first storage location includes a mirror copy of the first copy of the data, the first copy of the data is released from the first edge node by sending an acknowledgment to the first edge node.

22. The computer program product of claim 21, wherein the program instructions are readable and / or executed by the processor to cause the processor to: In response to determining that the first storage location does not include a mirror copy of the first copy of the data, a request is sent to the first edge node for the missing portion of the first copy of the data. The missing portion of the first copy of the received data; as well as The missing portion is used to complete the mirror copy of the first copy of the data at the first storage location.

23. The computer program product according to any one of claims 21 to 22, wherein the program instructions are readable and / or executed by the processor to cause the processor to: Receive a request from the second edge node to determine whether the second storage location includes a mirror copy of the second copy of the data received from the second edge node; and In response to determining that the second storage location includes a mirror copy of the second copy of the data, the second copy of the data is released from the second edge node by sending an acknowledgment to the second edge node.

24. The computer program product of claim 23, wherein the program instructions are readable and / or executed by the processor to cause the processor to: In response to determining that the second storage location does not include a mirror copy of the second copy of the data, a request is sent to the second edge node for the missing portion of the second copy of the data; The missing portion of the second copy of the received data; as well as The missing portion is used to complete the mirror copy of the second copy of the data at the second storage location.

25. A system comprising: The central data storage location is connected to the first and second edge nodes; processor; as well as Logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, wherein the logic is configured to: Receive the first copy of the data from the first edge node; The first copy of the data is stored in a first storage location at the central data storage location; Receive a second copy of the data from the second edge node; The second copy of the data is stored in a second storage location at the central data storage location; Receive a request from the first edge node to determine whether the first storage location includes a mirror copy of the first copy of the data received from the first edge node; as well as In response to determining that the first storage location includes a mirror copy of the first copy of the data, the first copy of the data is released from the first edge node by sending an acknowledgment to the first edge node.

Images