Data retrieval using embedding of data in backup system

Abstract

In general, techniques are described for efficient data retrieval from a backup system. An example computing system includes one or more storage devices and a processing component having access to the one or more storage devices and configured to: process an input to generate a filter, where the input indicates a context of one or more queries; applying the filter to backup data to obtain filtered data from the backup data; generating an embedded index from the filtered data; processing a query based on the embedded index to generate a response to the query; and outputting the response.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Patent Application No. 18 / 618,695 filed on March 27, 2024, U.S. Provisional Patent Application No. 63 / 503,631 filed on May 22, 2023, and Indian Provisional Patent Application No. 202341031783 filed on May 4, 2023, the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to data platforms of computing systems, and more particularly to data retrieval from backup systems. Background Art

[0004] Data platforms that support computing applications rely on primary storage systems to support latency-sensitive applications. They often rely on secondary storage systems to support secondary use cases, such as backup and archiving.

[0005] Backup data is often queried to retrieve specific information or data sets from the storage system, thereby enabling data analysis, data recovery, data mining, forensic analysis, and compliance with regulatory requirements. Many data platform solutions maintain an index or catalog of the backed up data, which facilitates efficient querying of the backup data. The data platform may enable a user to search the backup index based on query criteria, and the data platform may perform queries against the backup index, where the query specifies the search criteria and any additional parameters required. The query may involve searching for specific files, folders, databases, email messages, or other types of data stored in the backup. Based on the query results, which may include metadata information describing the backup data (such as file name, size, timestamp, and backup version), the user may select specific data or data sets to retrieve from the backup. This may involve selecting individual objects or the entire backup. Summary of the invention

[0006] In general, techniques for artificial intelligence (AI)-enhanced and efficient data retrieval from a backup system are described. In some examples, a data platform may generate an embedded index for filtered backup data stored on a backup system. The embedded index may effectively "scope" to a context of one or more queries expected from a user or application, and in some cases, may be generated on demand based on received input. A response generation platform may receive input indicating the context of the query from the response generation platform. A filter generator may process the input to determine the data type associated with the query. For example, a filter generator may analyze the input using a machine learning model to decode the data type (e.g., email data, file sharing data, databases, or other unstructured data) of interest to the user. The filter generator may generate a filter unique to the input based on the decoded data type, and the response generator platform applies the filter to the backed-up data to create an embedded index based on the data filtered using the filter generated based on the input. The embedded index may then be used to drive a retrieval augmented generation (RAG) query for the backup data.

[0007] The technology may provide one or more technical advantages. For example, the technology may allow end users or other users to make stored backup, archive, or other data "AI-Ready" by creating an advanced metadata / embedded index of the stored data and, in some aspects, securing the index through fine-grained role-based access control. End users and other users who store backup or other data on the storage system may use artificial intelligence and machine learning models to repurpose the data to gain additional efficiencies elsewhere in their workflows while keeping the data securely associated with the data platform.

[0008] In some examples, the response generation platform can be a retrieval-enhanced response generation platform that accepts user or application input (such as a question or query). The input can be tokenized using some extracted keywords, which are used to filter a large amount of data including backup data to filter down to a smaller subset of data. The response generation platform can then select the representations from those documents or objects that are most relevant to the user or machine query as embedded indexes. The embedded index can be provided to the language model along with the original query to enable the query processor to provide context-aware responses. Additional one or more queries related to the context indicated by the input can be received, and the query processor can also use the embedded index to generate corresponding responses to the one or more queries. This innovative approach allows the generated responses to be not only knowledgeable but also diverse and relevant to domain-specific content.

[0009] The technology can utilize AI and machine learning (specifically, generative AI) to examine the data managed by the data platform and generate new and original content based on the data. Generative AI tools use complex algorithms to evaluate data and derive new and unique insights to improve decision making and streamline operations. The response generation platform described herein can deliver an end-to-end cloud operations experience that simplifies and transforms information technology (IT) operations using a conversation-centric approach that responds to natural language questions with actionable targeted responses based on the data managed by the data platform.

[0010] In an example, a computing system includes: one or more storage devices; and a processing component having access to the one or more storage devices and configured to: process input to generate a filter, wherein the input indicates a context of one or more queries; apply the filter to backup data to obtain filtered data from the backup data; generate an embedded index from the filtered data; process the query based on the embedded index to generate a response to the query; and output the response.

[0011] In an example, a method includes: processing an input by a computing system to generate a filter, wherein the input indicates a context of one or more queries; applying the filter to backup data to obtain filtered data from the backup data; generating an embedded index from the filtered data; processing the query based on the embedded index to generate a response to the query; and outputting the response.

[0012] In an example, a computer-readable medium includes instructions that, when executed by a processing component, cause the processing component to: process an input to generate a filter, wherein the input indicates a context of one or more queries; apply the filter to backup data to obtain filtered data from the backup data; generate an embedded index from the filtered data; process the query based on the embedded index to generate a response to the query; and output the response.

[0013] The details of one or more examples of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A to Figure 1B is a block diagram illustrating an example system for performing adaptive deduplication on data chunks according to one or more aspects of the present disclosure.

[0015] Figure 2 is a block diagram illustrating an example system in accordance with techniques of this disclosure.

[0016] Figure 3 is a block diagram illustrating an example of a response generation platform and operation in greater detail.

[0017] Figure 4 is a flow diagram illustrating example operation of a computing system in accordance with one or more techniques of this disclosure.

[0018] Figure 5 is a block diagram illustrating an example response generation platform in accordance with one or more techniques of this disclosure.

[0019] Figure 6 is a block diagram illustrating a response generation platform in accordance with one or more techniques of this disclosure.

[0020] Like reference numerals refer to like elements throughout the text and drawings. DETAILED DESCRIPTION

[0021] Figure 1A to Figure 1B is a block diagram illustrating an example system according to one or more aspects of the present disclosure. Figure 1A In the example of , system 100 includes application system 102. Application system 102 can represent a collection of hardware devices, software components, and / or data repositories that can be used to implement one or more applications or services provided to one or more mobile devices 108 and one or more client devices 109 via network 113. Application system 102 can include one or more physical or virtual computing devices that execute workloads 174 for the applications or services. Workload 174 may include one or more virtual machines, containers, Kubernetes pods each including one or more containers, bare metal processes, and / or other types of workloads.

[0022] exist Figure 1AIn the example of , the application system 102 includes application servers 170A-170M (collectively referred to as "application servers 170") connected to a database server 172 implementing a database via a network. Other examples of the application system 102 may include one or more load balancers, web servers, network devices (such as switches or gateways), or other devices for implementing one or more applications or services and delivering one or more applications or services to the mobile device 108 and the client device 109. The application system 102 may include one or more file servers. One or more file servers may implement the primary file system of the application system 102. (In such instances, the file system 153 may be a secondary file system that provides backup, archiving, and / or other services for the primary file system. References to file systems herein may include primary or secondary file systems, for example, the primary file system of the application system 102 or a file system 153 that operates as a primary or secondary file system.) The application system 102 may be located locally and / or in one or more data centers, each of which is part of a public cloud, a private cloud, or a hybrid cloud. The application or service may be a distributed application. The application or service may support enterprise software, financial software, office or other productivity software, data analysis software, end-user relationship management, web services, educational software, database software, multimedia software, information technology, healthcare software, or other types of applications or services. The application or service may be provided in a service (-aaS) format as software as a service, platform as a service, infrastructure as a service, data storage as a service (dSaaS), or other types of services.

[0023] In some examples, application system 102 may represent an enterprise system that includes one or more workstations in the form of desktop computers, laptop computers, mobile devices, enterprise servers, network devices, and other hardware to support enterprise applications. Enterprise applications may include enterprise software, financial software, office or other productivity software, data analysis software, end-user relationship management, web services, educational software, database software, multimedia software, information technology, healthcare software, or other types of applications. Enterprise applications may include applications that generate queries to response generation platform 158, which responds to the queries using backup data stored at storage system 105 or storage system 115. Enterprise applications may be delivered as services from an external cloud service provider or other provider, executed natively on application system 102, or both.

[0024] exist Figure 1AIn the example of , the system 100 includes a data platform 150, which uses the storage system 105 and the separate storage system 115 to provide a file system 153 and a backup function to the application system 102. The data platform 150 implements a distributed file system 153 and a storage architecture to facilitate the access of the application system 102 to the file system data and to facilitate the data transmission between the storage system 105 and the application system 102 via the network 111. In the case of a distributed file system, the data platform 150 enables the device of the application system 102 to access the file system data via the network 111 using a communication protocol, as if such file system data is stored locally (for example, stored to the hard disk of the device of the application system 102). Example communication protocols for accessing files and objects include server message blocks (SMB), network file systems (NFS), or AMAZON simple storage services (S3). The file system 153 can be a primary file system or a secondary file system of the application system 102.

[0025] The file system manager 152 may represent a collection of hardware devices and software components that implement the file system 153 for the data platform 150. Examples of file system functions provided by the file system manager 152 include storage space management, including deduplication, file naming, directory management, metadata management, partitioning, and access control. The file system manager 152 may execute a communication protocol to facilitate access by the application system 102 to files and objects stored in the storage system 105 via the network 111.

[0026] The data platform 150 includes a storage system 105 having one or more storage devices 180A-180N (collectively referred to as "storage devices 180"). Storage devices 180 may represent one or more physical or virtual computing and / or storage devices that include storage media or otherwise have access to storage media. Such storage media may include one or more of a flash drive, a solid-state drive (SSD), a hard disk drive (HDD), an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory form and / or other types of storage media used to support the data platform 150. Different storage devices in the storage devices 180 may have different mixed types of storage media. Each of the storage devices 180 may include system memory. Each of the storage devices 180 may be a storage server, a network attached storage (NAS) device, or may represent disk storage of a computing device. The storage system 105 may include a redundant array of independent disks (RAID) system, storage as a service (STaaS), a network attached storage (NAS) and / or a storage area network (SAN). In some examples, one or more of the storage devices 180 is both a computing device and a storage device that executes software for the data platform 150, such as the file system manager 152 and the backup manager 154 in the example of the system 100, and stores objects and metadata for the data platform 150 to a storage medium. In some examples, a separate computing device (not shown) may execute software for the data platform 150, such as the file system manager 152 and the backup manager 154 in the example of the system 100. Each of the storage devices 180 may be considered and referred to as a "storage node" or simply a "node". The storage device 180 may represent a virtual machine running on a supported hypervisor, a cloud virtual machine, a physical rack server, or a computing model installed in a converged platform.

[0027] In some examples, data platform 150 may run locally on a physical system, virtually, or in the cloud. For example, data platform 150 may be deployed to a physical cluster, a virtual cluster, or a cloud-based cluster that runs in a private cloud deployed by a cloud service provider, an on-premises deployment, a hybrid cloud, or a public cloud. In some examples of system 100, multiple instances of data platform 150 may be deployed, and file system 153 may be replicated between the various instances. In some cases, data platform 150 may be a computing cluster representing a single management domain. The number of storage devices 180 may be expanded to meet performance requirements.

[0028] The data platform 150 may implement and provide multiple storage domains to one or more tenants, or to isolate workloads 174 that require different data policies. A storage domain may be a data policy domain that determines policies for deduplication, compression, encryption, tiering, and other operations performed on objects stored using the storage domain. In this way, the data platform 150 may provide users with the flexibility to choose global data policies or workload-specific data policies. The data platform 150 may support partitioning.

[0029] A view may be a protocol export that resides within a storage domain. A view inherits the data policy of its storage domain, but additional data policies may be specified for the view. A view may be exported via SMB, NFS, S3, and / or another communication protocol. Policies that determine data processing and storage by the data platform 150 may be assigned at the view level. A protection policy may specify backup frequency and retention policies.

[0030] Each of network 113 and network 111 may be the Internet, or may include or represent any public or private communication network or other network. For example, each of network 113 and network 111 may be a cellular, Wi-Fi®, ZigBee®, Bluetooth®, near field communication (NFC), satellite, enterprise, service provider, and / or other types of networks that support data transmission between computing systems, servers, computing devices, and / or storage devices. One or more of such devices may use any suitable communication technology to transmit and receive data, commands, control signals, and / or other information across network 113 or network 111. Each of network 113 or network 111 may include one or more network hubs, network switches, network routers, satellite antenna dishes, or any other network equipment. Such network devices or components may be operatively coupled to each other, thereby providing information exchange between computers, devices, or other components (e.g., between one or more client devices or systems and one or more computers / servers / storage devices or systems). Figure 1A to Figure 1B Each of the devices or systems shown in the diagram may be operatively coupled to network 113 and / or network 111 using one or more network links. The links coupling such devices or systems to network 113 and / or network 111 may be Ethernet, asynchronous transfer mode (ATM), or other types of network connections, and such connections may be wireless and / or wired connections. Figure 1A to Figure 1B One or more of the devices or systems shown or otherwise on network 113 and / or network 111 may be remotely located relative to one or more of the other shown devices or systems.

[0031] The application system 102 may generate objects and other data using the file system 153 provided by the data platform 150, and the file system manager 152 may create, manage and store the objects and data in the storage system 105. For this purpose, the application system 102 may be alternatively referred to as a "source system", and the file system 153 used for the application system 102 may be alternatively referred to as a "source file system". The application system 102 may communicate directly with the storage system 105 via the network 111 to transfer objects for some purposes, and communicate with the file system manager 152 via the network 111 to indirectly obtain objects or metadata from the storage system 105 for some purposes. The file system manager 152 may generate metadata and store the metadata in the storage system 105. The collection of data stored in the storage system 105 and used to implement the file system 153 is referred to as file system data in this article. The file system data may include the aforementioned metadata and objects. The metadata may include file system objects, tables, trees or other data structures; metadata generated to support deduplication; or metadata used to support snapshots. The stored objects may include any of files, virtual machines, databases, applications, pods, containers, workloads 154, system images, directory information, or other types of objects used by the application system 102. These may also be referred to as "backup objects." Objects of different types and objects of the same type may be deduplicated relative to each other.

[0032] Data platform 150 includes backup manager 154 that provides backup for file system data of file system 153. In the example of system 100, backup manager 154 backs up file system data stored by storage system 105 to storage system 115 via network 111.

[0033] The storage system 115 includes one or more storage devices 140A to 140X (collectively referred to as "storage devices 140"). Storage device 140 may represent one or more physical or virtual computing and / or storage devices that include storage media or otherwise have access to storage media. Such storage media may include one or more of a flash drive, a solid state drive (SSD), a hard disk drive (HDD), an optical disk, an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory form and / or other types of storage media. Different storage devices in storage device 140 may have different mixed types of storage media. Each of storage devices 140 may include system memory. Each of storage devices 140 may be a storage server, a network attached storage (NAS) device, or may represent disk storage of a computing device. Storage system 115 may include a redundant array of independent disks (RAID) system. Storage system 115 may be capable of storing much larger amounts of data than storage system 105. Storage device 140 may also be configured for long-term storage of information, more suitable for backup and / or archival purposes. Storage system 115 may alternatively be referred to herein as a "backup storage system" or a "backup system."

[0034] In some examples, storage system 115 may be a storage system deployed and managed by a cloud storage provider and is referred to as a “cloud storage system”. In some examples, storage system 115 may be co-located with storage system 105 in a data center, locally, or in a private, public, or hybrid cloud. When storage system 105 is the primary storage system, storage system 115 may be considered a “backup” or “secondary” storage system for storage system 105. Storage system 115 may be referred to as an “external target” for backups 142A–142K (collectively referred to as “backups 142”). In the case of being deployed and managed by a cloud storage provider, storage system 115 may be referred to as “cloud storage”. Storage system 115 may include one or more interfaces for managing the transmission of data between storage systems 105 and storage systems 115 and / or between application system 102 and storage system 115. The data platform 150 supporting application system 102 may rely on primary storage system 105 to support latency-sensitive applications. However, because storage system 105 is typically more difficult or more expensive to scale, data platform 150 may use secondary storage system 115 to support secondary use cases, such as backup and archiving. In general, a file system backup may be a copy of a file system 153 used to support protection of file system 153 for rapid recovery (typically due to the loss of some data in file system 153), and a file system archive ("archive") may be a copy of a file system 153 used to support long-term retention and review. A "copy" of file system 153 may include only such data as is required to restore or view the state of file system 153 at the time of the backup or archive. Although the technology of the present disclosure is primarily described with respect to retrieving backup data stored to a secondary storage system, the technology may be applied with respect to any data stored in any storage system as backup data. For example, backup data may include archived data, copied data, mirrored data, or snapshots.

[0035] Backup manager 154 can back up the file system data of file system 153 at any time according to a backup policy, which specifies, for example, the periodicity and timing of backups (daily, weekly, etc.), which file system data will be backed up, storage locations, access controls, etc. The backup of the file system data corresponds to the state of the file system data at the time of the backup. Backups 142A-142K (collectively referred to as "backups 142") therefore represent time-series data of file system 153, because each backup stores a representation of file system 153 at a specific time. Because file system 153 changes over time due to the creation of new objects, modification of existing objects, and deletion of objects, backups 142 may be different. Depending on the backup policy, a backup may include a complete backup of file system 153 data, or may include less than a complete backup of file system 153 data. For example, a given backup of backup 142 may include all objects of file system 153 or one or more selected objects of file system 153. A given backup of backup 142 may be a complete backup or an incremental backup.

[0036] Backup 142 may be used to generate views or snapshots. The current view may generally correspond to a (near) real-time backup state of file system 153. A snapshot may represent the backup state of primary storage system 105 at a particular point in time. That is, each snapshot may provide a state of the data of file system 153 that may be restored to primary storage system 105 as needed. Similarly, a snapshot may be exposed to a non-production workload, or a clone of a snapshot may be created if a non-production workload needs to write to the snapshot without disturbing the original snapshot.

[0037] Thus, backup manager 154 may use any of backups 142 to subsequently restore a file system (or portion thereof) to its state at the time the backup was created, or, for example, may use the backup to create or present a new file system (or "view") based on the backup. Backup manager 154 may de-duplicate file system data included in a subsequent backup against file system data included in one or more prior backups. For example, a second object of file system 153 included in a second backup may be de-duplicated against a first object of file system 153 included in a first, earlier backup.

[0038] Backup manager 154 can apply deduplication as part of a write process that writes (i.e., stores) objects of file system 153 to one of backups 142 in storage system 115. Additional description of an example deduplication process is found in U.S. patent application Ser. No. 18 / 183,659, filed Mar. 14, 2023, entitled “Adaptive Deduplication of Data Chunks,” which is incorporated herein by reference in its entirety. A user or application associated with application system 102 can access (e.g., read or write) backup data stored in storage system 115 via data platform 150.

[0039] Figure 1B The system 190 is Figure 1A 100, because data platform 150 stores backup 142 in storage system 105 instead of storage system 115. For example, system 190 can be deployed where operators of application system 102 rely on local archives.

[0040] The backup system may contain a large amount of information for the enterprise, but the backup 142 may have high access latency because it is stored on slower storage media. In addition, in modern distributed architectures, collecting, collating, and utilizing data from workflows across the data infrastructure (data estate) of an organization may be complex. The data platform 150 may operate in numerous locations across private data centers, single or multiple clouds, SaaS applications hosted by other organizations, and edge locations such as repositories, Internet of Things (IoT) devices, and many other applications. Conventional data platforms may store petabytes (or more) of data without classifying, indexing, or tracking the data. This is often referred to as "dark data," and it is often unknown to the organization and is often unstructured and / or difficult to access. The main challenge with dark data is that dark data represents missed opportunities for organizations to gain insights and make informed decisions, significantly reduce their data costs, and make data secure and protect data.

[0041] With advanced backup systems, backup data is readily available for analysis and use by machine learning / artificial intelligence applications to bring added value to users and businesses. Such advanced backup systems can create embedded indexes to use data in a systematic way. However, this approach can result in requiring double (or more) storage space compared to conventional backups.

[0042] According to the technology disclosed in this disclosure, Figure 1A to Figure 1BThe data platform 150 can generate an index of embeddings 164 (or “embeddings index 164”) of filtered backup data present in backups 142 on a backup system (e.g., storage system 115 or storage system 105), without having to store full embeddings for all backups 142. The generated embeddings 164 index can effectively be “scoped” to the context of a set of one or more queries anticipated from a user or application, and in some cases can be generated on demand based on received input.

[0043] The response generation platform 158 may receive an input indicating the context of the query of the response generation platform 158. The input itself may be a query. The filter generator 160 may process the input to determine the data type associated with the query. For example, the filter generator 160 may use a machine learning model to analyze the input to decode the data type (e.g., email data, file sharing data, databases, or other unstructured data) of interest to the user. The filter generator 160 may generate a filter unique to the input based on the decoded data type, and the response generator platform 158 may apply the filter to the data of the backup 142 to create an embedding 164 index based on the data filtered using the filter generated based on the input. In general, embedding is a simplified representation of more complex data and is useful for similarity searches. The embedding 164 index generated for the filtered data from the backup 142 can be used to drive a subsequent retrieval enhancement generation (RAG) query for the data in the backup 142.

[0044] Computing embeddings may include data conversion to text to obtain text-based embeddings. Therefore, data access may include such data conversions, for example, PDF to text, image to text using character recognition, etc. The response generation platform 158 may create an embedding 164 index by applying a machine learning model (not shown) based on artificial intelligence or other machine learning techniques. For example, the response generation platform 158 may use Word2Vec or global word vector representation (GloVe), a recurrent neural network (RNN) such as a long short-term memory (LSTM) or a gated recurrent unit (GRU) architecture, a transformer model, a convolutional neural network (CNN), a graph neural network (GNN), an autoencoder, a gradient boosting machine (GBM), a deep neural network (DNN), or other artificial neural networks for generating embeddings of text data. The response generation platform 158 may apply different techniques for embedding generation for different types of backup data, such as text, images, sequential data, tabular data, graph data, databases, time series data, email data, file sharing data, or unstructured data. Figure 6 An example machine learning model 520 for embedding generation is shown in FIG.

[0045] A machine learning system separate from the data platform 150 may be used to train a machine learning model for the response generation platform 158. The machine learning system may be executed by a computing system. The machine learning system may apply other types of machine learning to train the machine learning model to generate an embedding 164 index from the backup data. For example, the machine learning system may apply one or more of the following to train the machine learning model: nearest neighbor, naive Bayes, decision tree, linear regression, support vector machine, neural network, k-means clustering, Q-learning, temporal difference, deep adversarial network, or other supervised learning algorithm, unsupervised learning algorithm, semi-supervised learning algorithm, or reinforcement learning algorithm.

[0046] The technology may provide one or more technical advantages. For example, generating an embedded 164 index for data filtered according to a dynamically generated filter may provide a better balance between (1) storage consumption / access speed and (2) having to create a temporary embedded index each time a user or application wants to query or contact the associated backup data (a time-consuming process). Given that conventional systems continuously update embedded indexes for all data, requiring a large amount of storage space for the index, the technology may facilitate accurately determining the intent of a user or application and generating unique filters based on that intent, thereby ensuring that most relevant data is processed and included in the embedded 164 index for query response generation. Because unique filters may in some cases be generated based on decoded data types, the filters facilitate efficiently screening large amounts of backup data, focusing on specific data types identified by machine learning analysis, and ensuring that only relevant data is processed and included in the embedded 164 index.

[0047] Figure 2 is a block diagram illustrating an example system 200 in accordance with techniques of this disclosure. Figure 2 The system 200 can be described as Figure 1A System 100 or Figure 1B Examples or alternative implementations of the system 190. Figure 1A and Figure 1B Description of the field Figure 2 one or more aspects of .

[0048] exist Figure 2 In the example of , system 200 includes network 111, data platform 150 implemented by computing system 202, and storage system 115. Figure 2 In the example, the network 111, the data platform 150 and the storage system 115 can be connected with Figure 1A 1, corresponds to the network 111, the data platform 150, and the storage system 115. Although only one storage system 115 is depicted, the data platform 150 can apply the technology according to the present disclosure using multiple instances of the storage system 115. Different instances of the storage system 115 can be deployed by different cloud storage providers, the same cloud storage provider, an enterprise, or other entities.

[0049] The computing system 202 may be implemented as any suitable computing system, such as one or more server computers, workstations, mainframes, electrical appliances, cloud computing systems, and / or other computing systems that may be capable of performing operations and / or functions described according to one or more aspects of the present disclosure. In some examples, the computing system 202 may represent a cloud computing system, a server farm, and / or a server cluster (or a portion thereof) that provides services to other devices or systems. In other examples, the computing system 202 may represent or be implemented by one or more virtualized computing instances (e.g., virtual machines, containers) of a cloud computing system, a server farm, a data center, and / or a server cluster.

[0050] exist Figure 2 In the example of , computing system 202 may include one or more communication units 215, one or more input devices 217, one or more output devices 218, and one or more storage devices of storage system 105. Storage system 105 includes interface module 226, file system manager 152, backup manager 154, policy 159, backup metadata 222, block metadata 220, response generation platform 158, and filter generator 160. One or more of the devices, modules, storage areas, or other components of computing system 202 may be interconnected to enable inter-component communication (physically, communicatively, and / or operationally). In some examples, such connectivity may be provided by a communication channel (e.g., communication channel 212), which may represent one or more of a system bus, a network connection, an inter-process communication data structure, or any other method for transferring data.

[0051] The computing system 202 includes processing components. Figure 2 In the example of the present invention, the processing component includes one or more processors 213, which are configured to implement functionality and / or execute instructions associated with the computing system 202 or associated with one or more modules shown and / or described below, including the interface module 226, the file system manager 152, the backup manager 154, the response generation platform 158, and the filter generator 160. The one or more processors 213 can be a processing circuit system, can be a part of a processing circuit system, and / or can include a processing circuit system, which performs operations according to one or more aspects of the present disclosure. Examples of processors 213 include microprocessors, application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device. The computing system 202 can use one or more processors 213 to perform operations according to one or more aspects of the present disclosure using software, hardware, firmware, or a mixture of hardware, software, and firmware resident in the computing system 202 and / or executed at the computing system.

[0052] One or more communication units 215 of computing system 202 can communicate with devices outside computing system 202 by transmitting and / or receiving data, and in some aspects, can operate as both input devices and output devices. In some examples, communication unit 215 can communicate with other devices through a network. In other examples, communication unit 215 can send and / or receive radio signals on a radio network (such as a cellular radio network). In other examples, communication unit 215 of computing system 202 can transmit and / or receive satellite signals on a satellite network. Examples of communication unit 215 include network interface cards (e.g., Ethernet cards), optical transceivers, radio frequency transceivers, GPS receivers, or any other type of device that can send and / or receive information. Other examples of communication unit 215 may include devices that can communicate through Bluetooth®, GPS, NFC, ZigBee®, and cellular networks (e.g., 3G, 4G, 5G) and Wi-Fi® radios in mobile devices and universal serial bus (USB) controllers, etc. Such communications may adhere to, implement or comply with appropriate protocols, including Transmission Control Protocol / Internet Protocol (TCP / IP), Ethernet, Bluetooth®, NFC or other technologies or protocols.

[0053] The one or more input devices 217 may represent any input device of the computing system 202 that is not otherwise separately described herein. The input device 217 may generate, receive, and / or process input. For example, the one or more input devices 217 may generate or receive input from a network, a user input device, or any other type of device for detecting input from a person or a machine.

[0054] One or more output devices 218 may represent any output device of computing system 202 that is not otherwise described separately herein. Output device 218 may generate, present, and / or process output. For example, one or more output devices 218 may generate, present, and / or process output in any form. Output device 218 may include one or more USB interfaces, video and / or audio output interfaces, or any other type of device capable of generating tactile, audio, visual, video, electrical, or other output. Some devices may be used as both input devices and output devices. For example, a communication device may send data to other systems or devices and receive data from other systems or devices over a network.

[0055] One or more storage devices of the storage system 105 within the computing system 202 may store information for processing during operation of the computing system 202. The storage device may store program instructions and / or data associated with one or more of the modules described according to one or more aspects of the present disclosure. One or more processors 213 and one or more storage devices may provide an operating environment or platform for such modules, which may be implemented as software, but may include any combination of hardware, firmware, and software in some examples. One or more processors 213 may execute instructions, and one or more storage devices of the storage system 105 may store instructions and / or data of one or more modules. The combination of the processor 213 and the storage system 105 may retrieve, store, and / or execute instructions and / or data of one or more applications, modules, or software. The processor 213 and / or the storage device of the storage system 105 may also be operably coupled to one or more other software and / or hardware components, including but not limited to one or more of the components of the computing system 202 and / or one or more devices or systems shown as connected to the computing system 202.

[0056] The file system manager 152 may perform functions related to providing the file system 153, as described above with respect to Figure 1A The file system manager 152 may generate and manage file system metadata 232 for constructing file system data 230 of the file system 153, and store the file system metadata 232 and the file system data 230 to the storage system 105. The file system metadata 232 may include one or more trees that describe objects within the file system 153 and the file system 153 hierarchy and may be used to write or retrieve objects within the file system 153. The file system metadata 232 may reference either the block metadata 220 or the backup metadata 222, and vice versa.

[0057] File system manager 152 may interact and / or operate in coordination with one or more modules of computing system 202 , including interface module 226 and backup manager 154 .

[0058] The backup manager 154 can perform backup functions related to backing up the file system 153, as described above with respect to Figure 1AThe backup manager 154 may generate one or more backups 142 and cause file system data 230 to be stored in the backups 142. The backup manager 154 may generate and manage backup metadata 222 for generating, viewing, retrieving, or restoring any of the backups 142. The backup manager 154 may generate and manage block metadata 220 for objects stored as blocks (and references thereto) within block files for any of the backups 142. Logical files may be used to represent and manipulate stored objects to identify blocks of objects. The block metadata 220 may include a block table describing the blocks. The block table may include a corresponding block ID for the block and may contain a pointer to a block file and an offset within the block file for retrieving the block from the storage system 115. The blocks are written to the block file at different offsets. By comparing the new block ID with the block table, the backup manager 154 may determine whether the data already exists on the system. If the block already exists, the data may be discarded and the metadata of the object may be updated to reference the existing block. The backup manager 154 may use the block table to find the block file identifier of the block file containing the block.

[0059] Storing and retrieving backup objects as part of backup 142 is described in U.S. Patent 10,996,875, issued on May 4, 2021, entitled “Making More Active Use of a Secondary Storage System,” which is incorporated herein by reference in its entirety.

[0060] The block metadata 220 may include a block file table that describes the corresponding physical or virtual location of the block file on the storage system 115, as well as other metadata about the block file, such as checksums, encryption data, compression data, etc. Figure 2 142, backup manager 154 causes backup metadata 222 and block metadata 220 to be stored to storage system 105. In some examples, backup manager 154 causes some or all of backup metadata 222 and block metadata 220 to be stored to storage system 115. Backup manager 154, optionally in conjunction with file system manager 152, can use backup metadata 222, block metadata 220, and / or file system metadata 232 to restore any of backups 142 to a file system implemented by data platform 150, which can be presented by file system manager 152 to other systems.

[0061] The interface module 226 may implement an interface through which other systems or devices may determine the operation of the file system manager 152 or the backup manager 154. Another system or device may communicate via the interface of the interface module 226 to specify one or more policies 159.

[0062] System 200 can be modified to implement Figure 1B In the modified system 200 , the storage system 105 stores the backup 142 .

[0063] The interface module 240 of the storage system 105 can execute an interface through which other systems or devices can create, modify, delete and / or delete backup data of the backup 142. The interface module 240 can execute and present an API. The interface presented by the interface module 240 can be a gRPC, HTTP, RESTful, command line, graphical user, web, or other interface.

[0064] Although the techniques described in this disclosure are primarily described with respect to the backup functions performed by the backup manager of the data platform, similar techniques may be additionally or alternatively applied to the archive, replica, mirror / clone, or snapshot functions performed by the data platform. In such cases, the backup 142 will be an archive, replica, mirror / clone, or snapshot, respectively.

[0065] Figure 3 is a block diagram illustrating an example of the response generation platform 158 and operation in more detail. The response generation platform 158 includes an interface module 302, a filter generator 160, an embedding generator 162, and a query processor 321. The interface module 302 may implement an interface through which other systems or devices may communicate with the response generation platform 158. The interface presented by the interface module 302 may be a gRPC, HTTP, RESTful, command line, graphical user, web, or other interface.

[0066] Interface 302 receives input 300 . Input 300 may include contextual text or other data indicative of one or more queries from a user or application. The one or more queries may include input 300 .

[0067] The filter generator 160 processes the input 300 to determine the data type associated with one or more queries. For example, the filter generator 160 may analyze the input using a machine learning model to decode the data type that the user is interested in. The data type may be based on file type (e.g., email data, file sharing data, databases, or other unstructured data), based on association with certain entities (individuals, organizations, etc.), based on time or date, based on similar topics or semantics, based on context, or based on other dimensions used to classify or characterize data (such as text data). In this example, the filter generator 160 applies a machine learning model 306 to analyze the input 300 to decode the data type associated with one or more queries. This analysis allows the system to accurately determine the user's intent and customize the filter 304 accordingly, thereby ensuring that most of the relevant data is processed and included in the embedding 164 index. The machine learning model 306 may include, for example, natural language processing (NLP) and deep learning algorithms.

[0068] The filter generator 160 generates filters 304 based on the decoded data types. The filters 304 can be designed to enable the response generation platform 158 to efficiently sift through the large amounts of backup data in the backup 142, focus on specific data types identified by the machine learning analysis applied by the filter generator 160, and ensure that only relevant data is processed and included in the embedding 164 index.

[0069] Response generator platform 158 applies filter 304 to backup data in backups 142 to obtain filtered data that may be relevant to one or more queries as indicated by input 300 and determined by filter generator 160. In some cases, input 300 may specify a particular backup in backups 142 as the context of the one or more queries (i.e., all filtered data is included in the specified backup).

[0070] In some examples, the filtering applied by the response generator platform 158 can utilize three key concepts: preprocessing, CountVectorizer, and Naive Bayes classifier. First, the data index for the data backup system 142 can be preprocessed. The preprocessing step helps to clean the backup data that can be used for filtering and processing, remove unnecessary noise, and make it suitable for further analysis. Preprocessing may include tokenization, stop word removal, and other text normalization techniques. These improve the overall performance of the machine learning model 306. CountVectorizer is a technique for transforming text data into a token count matrix in natural language processing. It effectively creates a bag-of-words representation of text data, in which the occurrence of each word in the data set is tracked. This structured representation of the text data is then used as the input of the machine learning model 306. Naive Bayes classifier is a popular probabilistic algorithm for text classification problems. It is based on Bayes' theorem and assumes independence between features. In the context of filtering applications, the algorithm calculates the probability of different categories or labels for given text data. It assigns the category with the highest probability to the text, which can be considered a filtering decision. In case the category / label matches the filter 304 generated by the filter generator 160 for the input 300 , the classified text data is relevant according to the input 300 .

[0071] In summary, in this example, the filtering process begins by preprocessing the text data and creating a structured representation using CountVectorizer. This representation is used as input to a Naive Bayes classifier, which then calculates the probabilities of different categories for the text data. The category with the highest probability can be selected as the final label, thereby effectively filtering the text data from backup 142 based on its content.

[0072] Data access: The data platform 150 may provide role-based access control (RBAC) to the backup data and prevent users from accessing data they do not have permissions for, such as sensitive data (patient data / PII, trade secrets, financial, and more). In some examples, the response generation platform 158 may incorporate RBAC, where the filter generator 160 generates filters 304 for filtering out data that is inconsistent with the user's permissions in order to provide responses that are consistent with the user's permissions.

[0073] The embedding generator 162 processes the obtained filtered text data matched with the generated filter 304 to generate an embedding 164 index. An embedding is a numerical value, usually a vector, representation of a piece of information (e.g., text, document, image, audio, etc.). Embedding is a way to represent data as points in an n-dimensional space so that similar data points are clustered together. In NLP and other forms of artificial intelligence, embedding can represent text data and is used for text analysis because embedding is dense in semantic meaning. The embedding generator 162 obtains items of filtered text data (e.g., files, emails, text objects, etc.), encodes the items as embeddings, and indexes the items to generate an embedding 164 index (sometimes referred to as an embedding database). Based on the embedding 164 index, the query processor 321 processes the query to generate a response. For example, the query processor 321 can use the embedding 164 index to, for example, perform a semantic search of the query against the embedding 164 index and generate a response to the query based on the semantic search results. The embedding 164 index can follow the RBAC model, thereby allowing access control to the reading, writing, updating and deleting of the index at the role level. The query may be one of one or more queries for which the input 300 indicates a context. In another example, the query may be one of one or more additional queries 330 that may be obtained before, while, or after receiving the input.

[0074] Two embeddings are shown as being generated from data objects (hash loops) in backup 142, but embedding 164 index may have millions of embeddings. Any of backups 142 may be presented to response generation platform 158 as a view or snapshot of file system 153 at a point in time. Each of the data objects may be a "backup object," as described in U.S. Patent 10,996,875, referenced elsewhere in this disclosure.

[0075] By employing embeddings in the data retrieval process, the response generation platform 158 can efficiently and accurately identify relevant data points based on the embedded semantic meanings and relationships rather than relying solely on traditional keyword-based search methods. This allows for a more nuanced and comprehensive understanding of the query, resulting in improved data retrieval results and overall system performance.

[0076] In a further aspect of the invention, the embedding 164 index may be used to drive RAG queries and other such AI / ML applications from users of the application. RAG is a framework that combines pre-trained sequence-to-sequence (seq2seq) models with dense retrieval mechanisms, allowing for more intelligent and contextually relevant outputs to be generated. This allows users and applications to retrieve data in a secure and efficient manner without compromising the integrity of the system or the data itself. RAG queries may also be customized to specific data types identified by machine learning analysis, ensuring that users and applications can quickly and easily access desired information.

[0077] In the era of artificial intelligence, readily available trained large language models (LLMs) have become a powerful tool for generating human-like responses in a variety of applications. However, most existing knowledge-based conversational models rely on outdated material that can be individual documents related to the topic of the conversation, limiting the ability of LLMs to generate diverse and knowledgeable responses that can be more proprietary or domain-specific. To overcome this challenge, the concept of RAGs has been introduced, which combines the strength of LLMs with the ability to retrieve information from multiple documents. RAGs not only enable LLMs to generate more knowledgeable, diverse, and relevant responses, but also provide a more efficient way to fine-tune these models. By using RAGs to determine what to respond with and fine-tuning to guide how to respond, LLMs can deliver a more engaging and informative conversational experience.

[0078] The data platform 150 can provide a robust and domain-specific context to the RAG-driven AI system. By leveraging a robust file system 153, the data platform 150 incorporating a response generation platform 158 can be "AI-ready" for RAG-assisted large language models (LLMs) by providing on-demand embedded 164 indexes to applications requesting data in a timely manner. Data can be made secure through an RBAC control model.

[0079] In some examples, the response generation platform 158 is a retrieval-enhanced response generation platform that can accept user or application input 300 (such as a question or query). As described above, the input 300 can then be tokenized using some of the extracted keywords, which are used to filter a large amount of data including the enterprise's backup data (e.g., backup 142 - such backup data can be tens of petabytes) to filter down to a smaller subset of data. The response generation platform 158 can then select the representations from those documents or objects that are most relevant to the user or machine query as the embedding 164 index. The embedding 164 index can be provided to a language model (such as GPT4) (e.g., model 322) along with the original query to enable the query processor 321 to provide a context-aware response 332. Additional one or more queries 330 related to the context indicated by the input 300 can be received, and the query processor 321 can also use the embedding 164 index to generate corresponding responses 332 to the one or more queries 330. This innovative approach allows the generated responses to be not only knowledgeable but also diverse and relevant to the domain-specific content of the enterprise. The query processor 321 and the model 322 may be third-party applications. The query processor 321 and the model 322 may be executed using a public cloud, private cloud, or hybrid cloud infrastructure separate from other components of the response generation platform 500 and separate from the storage system 115 .

[0080] Multiple embedding 164 indexes may be generated and stored concurrently for different inputs 300 and corresponding sets of additional one or more queries 330. For example, a first user may provide a first instance of input 300, for which response generation platform 158 generates a first instance of embedding 164 indexes, and a second user may provide a different second instance of input 300, for which response generation platform 158 generates a different second instance of embedding 164 indexes. Response generation platform 158 may generate responses to respective queries from the first and second users using respective embedding 164 indexes.

[0081] By leveraging RAG on the enterprise's own dataset, end users will not need to perform expensive fine-tuning or initial training to teach the language model (e.g., model 322) what to say. This saves time, money, and reduces environmental impact as the enterprise's datasets are constantly changing and evolving. Utilizing RAG provides the latest and relevant context for any query. The technology can also enable responses based on any point in time (if reflected in a given backup in backup 142), thereby enabling versioned analysis of objects "in-flight."

[0082] Both automatic and human evaluation results using large-scale datasets show that the response generation platform 158 can generate more knowledgeable, diverse, and relevant responses than off-the-shelf LLMs without duplicating or significantly increasing data storage requirements. This breakthrough has significant implications for the future application of enterprise conversational question answering and search and discovery models in a variety of industries.

[0083] The introduction of RAG-driven AI systems presents a unique opportunity to leverage data-driven insights and enhance the quality of conversations across various platforms. Using a data platform 150 enhanced by a response generation platform 158, organizations can unlock new levels of efficiency, innovation, and growth.

[0084] The response generation platform 158 represents a significant leap forward in the field of knowledge-based conversations. By leveraging the power of multiple documents and incorporating both the topic and local context of the conversation, the approach can generate responses that are more knowledgeable, diverse, and relevant than ever before. As a result, companies and technology executives can harness the potential of RAG-driven AI systems to transform the way end users, partners, and employees interact, driving innovation and growth in the process.

[0085] In other words, the techniques of the present disclosure can implement RAG from backup data. Data retrieval techniques are very important for extracting relevant information from large data sets, making these methods crucial in data management and recovery processes. Traditional methods of data retrieval are insufficient to generate accurate and context-rich responses. RAG is a seq2seq model that incorporates problematic encoders and generators. This advanced model pairs a pre-trained seq2seq model (e.g., model 322) with an efficient retrieval mechanism to enable the generation of context-relevant and informative outputs.

[0086] The input 300 is encoded by the question encoder (e.g., filter generator 160) during the forward pass and is then passed to the retriever (described above with respect to the response generation platform 158) which extracts relevant context documents. These documents are added to the input before it is processed by the generator. RAG can work in conjunction with RagRetriever for end-to-end generation, or be combined with the output of the retriever in several steps.

[0087] RAG can be applied to backup data from data protection solutions. By incorporating RAG into the data management and recovery process, more accurate context-aware responses can be generated, improving the overall efficiency and effectiveness of the system.

[0088] advantage

[0089] RAG offers numerous advantages, including:

[0090] a. Improved data retrieval accuracy: RAG generates more relevant and accurate responses based on retrieved context documents.

[0091] b. Context-aware response: By leveraging the retriever mechanism, RAG provides more context-rich output.

[0092] c. Efficient Data Management: RAG provides a more efficient data management process by integrating advanced sequence-to-sequence models and data retrieval mechanisms.

[0093] The disclosed techniques may provide one or more advantages over traditional data retrieval methods in backup systems and artificial intelligence applications. These advantages may include:

[0094] 1. Improved efficiency: By generating unique filters 304 and creating embedded 164 indexes on demand or “on the fly,” the system can quickly and efficiently process large amounts of backup data for use in RAG queries or other AI / ML applications, ensuring that users can access the information they need without significant delays.

[0095] 2. Enhanced security: RBAC is implemented in a way that ensures that users and applications can access only the data for which they have appropriate permissions, thereby protecting sensitive information from unauthorized access.

[0096] 3. Scalability: The system is designed to handle large amounts of backup data, making it suitable for a wide range of advanced file systems and artificial intelligence applications.

[0097] 4. Flexibility: The on-demand nature of the embedded 164 index allows users and applications to access the data they need when they need it, without having to wait for long processing times or rely on pre-generated indexes that require large amounts of storage.

[0098] 5. Automatic cleanup: The system can automatically clean up embedded 164 indexes after a certain amount of time has passed or the number of times the index is used drops below a threshold, ensuring that the system remains efficient and does not get bogged down by outdated or unused indexes.

[0099] In this way, the technology of the present disclosure can provide efficient and secure data retrieval in advanced file systems and artificial intelligence applications. By generating unique filters and creating embedded indexes on the fly, the system allows users to quickly and securely access the data they need while also ensuring that the system remains efficient and scalable.

[0100] Figure 4 is a flowchart illustrating an example operation of a computing system according to one or more techniques of this disclosure. Figure 4As can be seen in the example of , computing system 202 may initially process input to generate (or otherwise obtain) a filter, where the input indicates a context for one or more queries (400). Next, computing system 202 may apply the filter to the backup data to obtain filtered data from the backup data (405). Next, computing system 202 may generate an embedded index from the filtered data (410). Next, computing system 202 may process the query based on the embedded index to generate a response to the query (415). Next, computing system 202 may output the response (420).

[0101] Figure 5 5 is a block diagram illustrating an example response generation platform according to one or more techniques of the present disclosure. Response generation platform 500 is another example of a response generation platform and may be used in conjunction with Figure 2 202 in a manner similar to response generation platform 158. Similar to response generation platform 158, response generation platform 500 includes embedding generator 162, embedding 164, and query processor 321.

[0102] The response generation platform 500 includes a data access layer 504, a dynamic filter service 560, a database layer 320, an embedding generator 162, and an API access layer 502. The API access layer 502 can be used to generate filters 505 to identify data to be vectorized and / or take an input query 330 and return relevant context in the embedding 164 from the data stored by the database layer 320. In order to interact with AI applications that can use private / enterprise data sets, the response generation platform 500 (and other response generation platforms described herein) can deliver additional context to the query 330. This context retrieval utilizes the data access layer 504, the dynamic filter service 560, the database layer 320, the embedding generator 162, and the API access layer 502. The database layer 320 can be used in conjunction with the data access layer 504, which accesses individual files, folders, and entire data sets as a result of the filters 505 applied to the data in the storage system 115. The filters 505 can be generated by an end user or an automated system (e.g., the dynamic filter service 560). Dynamic filter service 560 may be an example instance of filter generator 160. Filtered data is retrieved from data access layer 504, for example, as a view or snapshot of file system 153 or a filtered portion thereof at a particular time. The filtered data is presented to database layer 300. Embedding generator 162 uses a vector database (not shown) together with machine learning model 520 to compute embeddings 164 for the data. Machine learning model 520 may be a language model. These embeddings 164 may be stored in the vector database of database layer 320 along with the data and metadata. Examples of machine learning models (including language models) are described elsewhere in this document.

[0103] The data access layer 504 may be configured to receive input and return data from the storage system 115 based on the data set / filter 505 requested as input to the data access layer.

[0104] Dynamic filter service 560 can be configured to receive input when requesting data from data access layer 504 and convert the input into a filter (e.g., filter 505). Filter 505 can be manually specified (e.g., a user manually selects a workload or view to be backed up) or derived dynamically based on intent (e.g., "I need all emails for the last 90 days."). For example, Figure 5 Filter selection 550 is depicted, which may indicate a manual specification of desired data or intent. Filter selection 550 may include filter 505.

[0105] The embedding generator 162 may receive data as input and return an array of floating point numbers as a vector representation of the data. The embedding of the computed data is described elsewhere in this document. The embedding 164 index may be per tenant, per account, or per user, where the storage system 115 stores data for multiple tenants / accounts / users. The data access layer 504 may perform RBAC on the data set stored in the storage system 115 to provide responses consistent with the tenant / account / user permissions. The embedding generator 162 may be extended to additional workloads for new filter selections 550. For example, the response generation platform 500 may spawn additional instances of the embedding generator 162 to compute the embedding of the newly retrieved filtered data set retrieved by the data access layer 504. The database layer 320 (including the embedding 164) may be shared across multiple instances. In some cases, the additional instance may be executed at a third-party computing node (e.g., in a public cloud).

[0106] The database layer 320 may be configured to insert new entries of the vector / embedding into the embedding 164 or to perform queries using the embedding / vector to return the n entries of the approximate nearest neighbors from the submitted embedding.

[0107] The API access layer 502 may be configured to receive an input query 330 ( Figure 6 ) to match the context from embedding 164 and data access layer 504. API access layer 502 may also be configured to receive input for creating dynamic filters 505, as described above.

[0108] Figure 5The process of filter configuration / generation and embedding generation is depicted. A user / administrator or application (via API access layer 502) can use the response generation platform 500 to provide or generate (via dynamic filter service 560) filters 505 by sending filter selections 550, which can be applied to a data set (530) stored in the storage system 115. The filter 505 can be interpreted by the dynamic filter service 560 and sent to the data access layer 504 (531) to retrieve data from the data set that satisfies the filter 505 (532). The data set can be, for example, one of the backups 142 or another data set. Once filtered by applying the filter 505 and retrieved through the data access layer 504 (533), the embedding generator 162 of the application model 520 (534) can process the filtered data set through a vectorization engine to catalog the filtered data set and store the resulting vectors along with subsequent metadata in a vector database (536). In Figure 5 , and is described elsewhere in this document as having an embedded embedding 164 index. An entry in the vector database may contain a complete file, a portion of a file, or the location of a file along with the embedding itself. Additional metadata may be added, such as file location (for reference purposes to support responses to queries) and access control list (ACL) information. The created embedding 164 index can now be used and interacted with post-processing interactions such as answering user questions ("queries"). In some examples, the data access layer 504 is configured to remove personally identifiable information from data retrieved from the storage system 115.

[0109] At step 536 , in some examples, the database layer 320 may save the data / text block into the embed 164 or save a reference link to the data for retrieval.

[0110] Figure 6 is a block diagram illustrating a response generation platform in accordance with one or more techniques described herein. Figure 6The query / response process involving the response generation platform 500 is depicted. A user or application may send a query 330 to the API access layer 502. In general, the query 330 may indicate a question that needs to be answered, a request for information, a guide, or other information to be answered using data stored to the storage system 115. The API access layer 502 computes an embedding for the query 330, and the API access layer 502 computes the approximate nearest neighbors of the query 330 using the database layer 320 indexed by the query embedding 164 (540), and the database layer 320 returns the indexed relevant data set (542). The returned indexed data set may indicate the nearest neighbor embedding mapped in the embedding 164 index and metadata about the embedding, such as the file name, location, and access information (user / group ownership). This metadata may then be passed to the data access layer 504, which will retrieve the reference files and / or objects and return them to the API access layer 502. Once the collection of files and / or objects is obtained, the relevant context "{context}" is extracted from them and the relevant context is used to construct hints for the model 322. Once the lookup occurs, the system can prepare the payload (hint) and send the payload to the query processor 321 (580).

[0111] An example prompt for a payload sent to the query processor 321 is as follows:

[0112] 1) user_msg = "Answer the user-supplied question based only on the context in which the question was provided: {question}context: {context}"

[0113] 2) Input {question}.

[0114] 3) {context} obtained from the embedding 164 index search. This is the top n data / text chunks that match the ANN query.

[0115] The lookup by the database layer 320 from the embedding 164 index may return the relevant paragraph / data block (if stored in the embedding 164 index) or a pointer to the location of the data / offset of the location of the data (which the data access layer 540 may obtain and append to the prompt). In this way, this collaboration between the database layer 320 and the data access layer 502 provides on-demand data payloads from the backup or file storage system 115 for use by AI-driven applications without having to create embeddings in advance for all data stored to the storage system 115. The query processor 321 applies the model 322 to the prompt and returns an evaluated answer (582). The API access layer 504 includes the answer in the response 332.

[0116] In some examples, the data access layer 504 may be configured to cache all or some portion of a data set retrieved from the storage system 115. Cache 561 may represent a storage cache within the data access layer 504 that stores data from the storage system 115 for future queries. The data access layer 504 optionally stores all or selected data from the retrieved filtered data set. In some examples, the filter selection 550 may include an indication for persisting all or selected data from the retrieved data set that has been filtered based on the filter 505 corresponding to the filter selection 550. The data access layer 504 may use the cache 561 to persist the data in response to the indication. The indication may specify a time range for persistence, for example, one day, or until the user completes the session with the API access layer 502. In some examples, the cache 561 may be included in an embedded 164 index database or other data structure. In addition, the embedded 164 index may be updated to reference the cached data instead of (or in addition to) the location in the storage system 115. In some examples, data access layer 504 may store selected data in cache 561 based on historical data indicating the frequency of retrieval of various files and objects from storage system 115. For example, more frequently used data to answer queries may be cached more frequently and / or for longer periods of time.

[0117] In some examples, the user can provide data to be queried in association with the query. Response generation platform 500 can provide the data to the user via API access layer 502.

[0118] For the processes, devices, and other examples or illustrations described herein, including in any flowchart or flowchart format, certain operations, actions, steps, or events included in any of the techniques described herein may be performed in a different order, may be added, merged, or omitted entirely (e.g., not all described actions or events are necessary for the practice of the technique). In addition, in some examples, operations, actions, steps, or events may be performed simultaneously, such as by multithreading, interrupt processing, or multiple processors, rather than sequentially. In addition, certain operations, actions, steps, or events may also be automatically performed even if not explicitly identified as being automatically performed. Moreover, certain operations, actions, steps, or events described as being automatically performed may alternatively not be automatically performed, but, in some examples, such operations, actions, steps, or events may be performed in response to an input or another event.

[0119] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a comprehensive understanding of the various concepts. However, it is apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, in order to avoid confusion of such concepts, well-known structures and components are shown in block diagram form.

[0120] According to one or more aspects of the present disclosure, the term "or" may be interpreted as "and / or" if the context does not dictate otherwise. In addition, while phrases such as "one or more" or "at least one" may have been used in some instances, they are not used in other instances; those instances in which such language is not used may be interpreted as having this meaning implied if the context does not dictate otherwise.

[0121] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on and / or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium (such as a data storage medium), or a communication medium (including any medium that facilitates the transfer of a computer program (e.g., according to a communication protocol) from one place to another). In this manner, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0122] For example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, flash memory or any other medium that can be used to store the desired instructions or program code in the form of data structures and can be accessed by the computer. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit instructions from a website, server or other remote source, then coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave) is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but rather relate to non-temporary tangible storage media. As used, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, where disks usually reproduce data magnetically, and optical disks reproduce data optically using lasers. The above combination should also be included in the scope of computer-readable media.

[0123] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the terms "processor" or "processing circuit system" as used herein may each refer to any of the aforementioned structures or any other structure suitable for implementing the described techniques. In addition, in some examples, the described functionality may be provided within dedicated hardware and / or software modules. Moreover, the described techniques may be implemented entirely in one or more circuits or logic elements.

[0124] As used herein, a processing component may include a processing circuit system as has been described above. In some examples, the processing component may include at least one processor and at least one memory having a computer code, the computer code including an instruction set, which when executed by the at least one processor causes the at least one processor to perform any of the functions described herein. In some examples, the processing component may receive the computer code including the instruction set from at least one memory coupled to the processing component.

[0125] The technology of the present disclosure may be implemented in a variety of devices or equipment, including wireless handsets, mobile or non-mobile computing devices, wearable or non-wearable computing devices, integrated circuits (ICs) or IC sets (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize the functional aspects of devices configured to perform the disclosed technology, but they do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in a hardware unit, or may be provided by a collection of interoperable hardware units (including one or more processors as described above) in conjunction with suitable software and / or firmware.

[0126] Aspects of the present invention are provided with reference to the following clauses:

[0127] Clause 1. A computing system comprising: one or more storage devices; and a processing circuit system having access to the one or more storage devices and configured to: process input to generate a filter, wherein the input indicates a context of one or more queries; apply the filter to backup data to obtain filtered data from the backup data; generate an embedded index from the filtered data; process a query based on the embedded index to generate a response to the query; and output the response.

[0128] Clause 2. The computing system of Clause 1, wherein the processing circuitry is configured to: encode the filtered data to generate an embedding for each item of the filtered data; and generate the embedding index from the generated embedding.

[0129] Clause 3. The computing system of Clause 1, wherein the filter specifies one or more of: a file type, an association with an entity, a date, a time, or a topic.

[0130] Clause 4. A computing system as described in Clause 1, wherein, in order to generate the filter, the processing circuit system is configured to: preprocess the backup data to generate preprocessed text data; apply CountVectorizer to the preprocessed text data to calculate a word count matrix; and process the word count matrix using a machine learning model to classify items of the preprocessed text data into any of a plurality of categories.

[0131] Clause 5. The computing system of Clause 4, wherein to apply the filter, the processing circuitry is configured to include in the filtered data items of the pre-processed text data that are assigned a category that matches the filter.

[0132] Clause 6. The computing system of Clause 1, wherein to generate the filter, the processing circuitry is configured to: apply role-based access control to at least one of the queries or the filter generation.

[0133] Clause 7. The computing system of Clause 1, wherein to generate the filter, the processing circuitry is configured to: delete the embedding of the index based on a determination that the embedded index is no longer needed to satisfy a query.

[0134] Clause 8. The computing system of Clause 1, wherein to process the query, the processing circuitry is configured to: apply retrieval enhancement generation using the query and the embedded index to generate the response to the query.

[0135] Clause 9. The computing system of Clause 1, wherein the input comprises the query.

[0136] Clause 10. The computing system of Clause 1, wherein the processing circuitry is configured to receive the query after generating the embedded index.

[0137] Clause 11. The computing system of Clause 1, wherein to process the query, the processing circuitry is configured to process the query using a language model and the embedding index to perform retrieval enhancement generation to generate the response to the query.

[0138] Clause 12. The computing system of Clause 1, wherein the processing circuitry is configured to store at least a portion of the backup data to a cache.

[0139] Clause 13. The computing system of Clause 12, wherein the processing circuitry is configured to generate or modify the embedding with reference to corresponding backup data of the embedding of the embedding index stored to the cache.

[0140] Clause 14. The computing system of Clause 12, wherein the processing circuitry is configured to process a subsequent query based on the embedded index and the cache to generate a response to the subsequent query.

[0141] Clause 15. A method comprising: processing an input by a computing system to generate a filter, wherein the input indicates a context of one or more queries; applying the filter to backup data to obtain filtered data from the backup data; generating an embedded index from the filtered data; processing a query based on the embedded index to generate a response to the query; and outputting the response.

[0142] Clause 16. The method of Clause 15, further comprising: encoding the filtered data to generate an embedding for each item of the filtered data; and generating the embedding index from the generated embedding.

[0143] Clause 17. The method of Clause 15, wherein processing the query comprises processing the query using a language model and the embedding index to perform retrieval enhancement generation to generate the response to the query.

[0144] Clause 18. The method of Clause 15, wherein processing the query comprises applying retrieval enhancement generation using the query and the embedded index to generate the response to the query.

[0145] Clause 19. The method as described in Clause 15 further includes: storing at least a portion of the backup data in a cache; generating or modifying the embedding with reference to the corresponding backup data of the embedding of the embedded index stored in the cache; and processing subsequent queries based on the embedded index and the cache to generate a response to the subsequent query.

[0146] Clause 20. A non-transitory computer-readable medium comprising instructions that, when executed by a processing circuit system, cause the processing circuit system to: process an input to generate a filter, wherein the input indicates a context for one or more queries; apply the filter to backup data to obtain filtered data from the backup data; generate an embedded index from the filtered data; process a query based on the embedded index to generate a response to the query; and output the response.

Claims

1. A computing system comprising: one or more storage devices; and a processing component having access to the one or more storage devices and configured to: processing input to generate a filter, wherein the input indicates context of one or more queries; applying the filter to the backup data to obtain filtered data from the backup data; generating an embedding index from the filtered data; processing a query based on the embedded index to generate a response to the query; and The response is output.

2. The computing system of claim 1, wherein: The processing component is configured to: encoding the filtered data to generate an embedding for each item of the filtered data; as well as The embedding index is generated from the generated embedding.

3. A computing system as claimed in claim 1 or claim 2, wherein: The filters specify one or more of: file type, association with an entity, date, time, and topic.

4. A computing system as claimed in any preceding claim, wherein: To generate the filter, the processing component is configured to: preprocessing the backup data to generate preprocessed text data; Applying CountVectorizer to the preprocessed text data to calculate a word unit count matrix; The word-gram count matrix is ​​processed using a machine learning model to classify items of the pre-processed text data into any of a plurality of categories.

5. The computing system of claim 4, wherein: To apply the filter, the processing component is configured to: Items of the pre-processed text data that are assigned a category that matches the filter are included in the filtered data.

6. A computing system as claimed in any preceding claim, wherein: To generate the filter, the processing component is configured to: Role-based access control is applied to at least one of the queries or the filter generation.

7. A computing system as claimed in any preceding claim, wherein: To generate the filter, the processing component is configured to: Based on a determination that the embedded index is no longer needed to satisfy the query, the embedded index is deleted.

8. A computing system as claimed in any preceding claim, wherein: In order to process the query, the processing component is configured to: Retrieval enhancement generation is applied using the query and the embedded index to generate the response to the query.

9. A computing system as claimed in any preceding claim, wherein: The one or more queries include the query.

10. A computing system as claimed in any preceding claim, wherein: The query includes the input.

11. The computing system of any one of claims 1 to 8, wherein: The processing component is configured to receive the query after generating the embedded index.

12. A computing system as claimed in any preceding claim, wherein: To process the query, the processing component is configured to process the query using a language model and the embedding index to perform retrieval enhancement generation to generate the response to the query.

13. A computing system as claimed in any preceding claim, wherein: The processing component is configured to store at least a portion of the backup data in a cache.

14. The computing system of claim 13, wherein: The processing means is configured to generate or modify the embedding with reference to corresponding backup data of the embedding of the embedding index stored to the cache.

15. A computing system as claimed in claim 13 or claim 14, wherein: The processing component is configured to process a subsequent query based on the embedded index and the cache to generate a response to the subsequent query.

16. A method comprising: processing, by a computing system, an input to generate a filter, wherein the input indicates a context of one or more queries; applying the filter to the backup data to obtain filtered data from the backup data; generating an embedding index from the filtered data; processing a query based on the embedded index to generate a response to the query; and The response is output.

17. The method of claim 16, further comprising: encoding the filtered data to generate an embedding for each item of the filtered data; as well as The embedding index is generated from the generated embedding.

18. A method according to claim 16 or claim 17, wherein: Processing the query includes processing the query using a language model and the embedding index to perform retrieval enhancement generation to generate the response to the query.

19. The method according to any one of claims 16 to 18, wherein: Processing the query includes applying retrieval enhancement generation using the query and the embedded index to generate the response to the query.

20. The method of any one of claims 16 to 19, further comprising: storing at least a portion of the backup data in a cache; generating or modifying the embedding with reference to corresponding backup data of the embedding of the embedding index stored to the cache; as well as A subsequent query is processed based on the embedded index and the cache to generate a response to the subsequent query.

21. A computer readable medium comprising instructions which, when executed by a processing component, cause the processing component to perform the method of any one of claims 16 to 20.

Images