Learning-based storage reduction in an overlay network

Abstract

An overlay network is augmented to provide more efficient data storage by processing a dataset of high dimension into an equivalent dataset of lower dimension, wherein the data reduction reduces the amount of actual physical data but not necessarily its informational value. Data to be processed (dimensionally-reduced) is received by an ingestion layer and supplied to a learning-based storage reduction application that implements the data reduction technique. The application applies a data reduction algorithm and stores the resulting dimensionally-reduced data sets in the native data storage or third party cloud. To recover the original higher-dimensional data, an associated reverse algorithm is implemented. In general, the application coverts an N dimensional data set to a K dimensional data set, where K<<N. The N dimensional dataset has a high dimension, and the K dimensional dataset has a low dimension. From a data storage perspective, it is cheaper and more efficient to store the low dimensional dataset.

Description

BACKGROUNDTechnical Field[0001]This application relates generally to optimizing data storage in a distributed computing network environment.Brief Description of the Related Art[0002]Distributed computer systems are well-known in the prior art. One such distributed computer system is a “content delivery network” (CDN) or “overlay network” that is operated and managed by a service provider. The service provider typically provides the content delivery service on behalf of third parties (customers) who use the service provider's shared infrastructure. A distributed system of this type typically refers to a collection of autonomous computers linked by a network or networks, together with the software, systems, protocols and techniques designed to facilitate various services, such as content delivery, web application acceleration, or other support of outsourced origin site infrastructure. A CDN service provider typically provides service delivery through digital properties (such as a webs...

AI Technical Summary

Benefits of technology

The patent describes a system that uses a data reduction technique to store high dimension data in a more efficient way. The system uses a learning-based application that processes the data and reduces its size. This reduces the amount of physical data that needs to be stored but does not decrease its informational value. The data is reduced using a technique called Principal Component Analysis (PCA) and stored either in a native data storage or a cloud. The system allows for easy recovery of the original data when requested. Overall, the system provides a way to manage large data sets more efficiently.

Problems solved by technology

Datasets generated by these processes are complex and typically comprise an interrelated set of data items.

Sustaining a large volume of data is costly both in terms of processing (CPU usage) and storage.

Memory and RAM overflow leads to the creation of data backlogs, and the cascading effects of such issues can lead to delays in real-time reporting that is utilized by the provider both for customer billing and internal analytics.

Images