Dynamic embedded process based on system resource utilization

Abstract

Embodiments of the present disclosure relate to dynamic embedded processes based on system resource utilization. Systems and methods are provided for managing dynamic embedded processes based on current system resource utilization. In this manner, system resource utilization values can correspond to memory utilization, CPU or GPU consumption, or other usage of system components, and the techniques of the present disclosure can adjust embedded processes based on these values.

Description

Background Technology

[0001] In computational search, "embedding" refers to the technique of converting text or other data into a mathematical representation that captures the semantic meaning of the transformed data. Search engines can use embeddings to perform searches more efficiently. In this sense, relevant search results are generated based on the potential or semantic meaning of the search, rather than simply matching keywords, thus enabling the system to achieve more accurate search capabilities. Attached Figure Description

[0002] This disclosure is described in detail with reference to the following accompanying drawings, based on one or more various examples. These drawings are provided for illustrative purposes only and depict only typical, non-limiting aspects of such examples.

[0003] Figure 1 The illustration shows a set of computing components in a distributed system based on some examples of the disclosed technology, including an object client, a data intelligence engine, a workload computing component, and a search component.

[0004] Figure 2 The illustration shows an example of a data intelligence engine that can implement the disclosed technology, including a data path component, an embedded computing component, a workload optimizer engine, and a vector data repository.

[0005] Figure 3 It is an example message flow between a data path component and an embedded computing component, based on some examples of the disclosed technology.

[0006] Figure 4 The diagram illustrates an embedded profile data structure based on some examples of the disclosed techniques.

[0007] Figure 5 This is an example message flow between a workload optimizer engine, a processor with embedded computing components, and a vector data repository, based on some examples of the disclosed technologies.

[0008] Figure 6 This is an example message flow between a search component, an embedded computing component, a processor, and a vector data repository, based on some examples of the disclosed technology.

[0009] Figure 7 It is an example message flow between a data path component and an embedded computing component, based on some examples of the disclosed technology.

[0010] Figure 8 This is an example method for implementing a dynamic embedded process based on system resource utilization, based on examples of the disclosed technology.

[0011] Figure 9The illustration shows a computing component that can be used to implement dynamic embedded processes based on system resource utilization, according to an example of the disclosed technology.

[0012] Figure 10 A block diagram depicts an example computer system that can implement various examples of the techniques disclosed herein.

[0013] The accompanying drawings are not exhaustive and do not limit this disclosure to the precise form disclosed. Detailed Implementation

[0014] Traditional systems can generate embeddings of data objects to speed up and efficiently search the data later. However, the process of generating embeddings can be computationally expensive. Additionally, near real-time embedding generation for latency-sensitive storage systems can impact the quality of service of input / output (I / O), processors, and subsystems. For example, the embedding process in a traditional system can use virtually unlimited resources when identifying data objects to generate embeddings.

[0015] In some examples, the embedding process (and the use of the resulting embedding) is implemented through a distributed system. For example, the distributed system may include a data intelligence engine that communicates with object clients, a workload computation component, and a search component. Object clients may generate / store data in an object storage system (e.g., in data buckets / data sets) and provide the data or data objects to the data intelligence engine. The data intelligence engine may initiate an embedding process to generate an embedding of the received data object. In some examples, the embedding process is implemented using a third-party entity (called a workload computation component) that provides service-based or function-based resources to help accelerate the embedding process (this can be very resource-intensive). The workload computation component may be decoupled from the data intelligence engine and object clients. The data intelligence engine may invoke the services / functions provided by the workload computation component to help generate the embedding during the embedding process.

[0016] Examples of the disclosed techniques include systems, methods, and computer-readable media that can manage dynamically embedded processes implemented in a distributed system based on current system resource utilization measured by a workload computing component. Current system resource utilization may correspond to, for example, memory utilization, central processing unit (CPU) or graphics processing unit (GPU) consumption, I / O throughput, or other system operations measured by the workload computing component, which may be utilized by service / function calls from a data intelligence engine. The data intelligence engine can adapt the embedded process by taking into account the resources currently being used by the data path component, the workload computing component, and other system components, as calculated by the workload computing component. The embedded process can be adapted so as not to degrade computational or memory performance when the workload computing component processes services / functions from the data intelligence engine and other services / functions across the distributed system. The workload computing component can calculate / measure system utilization when resource consumption comes from both the data path component and the embedded component. That is, if system resources are currently under pressure from the data path component or the embedded component, examples of the disclosed techniques can employ an embedded process to minimize / limit any further impact on these system resources. On the other hand, if system resources are currently idle or otherwise lightly used, examples of the disclosed technology may employ more resource-intensive embedded processes and utilize more system resources provided by the workload computing components.

[0017] In some examples, the data intelligence engine can select an embedding profile from a set of embedding profiles, based on which an artificial intelligence (AI) model can generate embeddings of data objects. The various embedding profiles in the set can generate embeddings at different levels of accuracy. For example, a first embedding profile (e.g., an approximate embedding profile) can be associated with a first AI model that generates an "approximate embedding" of the data object, and a second embedding profile (e.g., an accurate embedding profile) can be associated with a second AI model that generates an "accurate embedding" of the data object. When comparing these two embeddings, approximate embedding can generate an embedding using fewer system resources than accurate embedding, and it can create an embedding that is smaller and has a lower level of accuracy than accurate embedding. Using fewer system resources allows workload computing components and the data intelligence engine to perform other operations, such as I / O processes, memory utilization, and processing functions, while still creating embeddings without waiting for resources to become available for embedding creation. Overall, workload computing components and the data intelligence engine can create approximate embeddings while leaving sufficient headroom for other tasks to utilize available resources, thereby improving the efficiency of the entire system.

[0018] In some examples, approximate embedding and exact embedding are two points on the spectrum of creating data vector representations, allowing for the selection of an embedding process to weigh the trade-offs between accuracy and efficiency with respect to the second embedding process. An "exact" embedding can correspond to a high-fidelity representation of the original data and involves more complex computations and higher dimensionality than an approximate embedding. An "approximate" embedding can correspond to low-dimensional encoding that prioritizes efficiency, speed, and scalability over the complex computations used in the exact embedding process.

[0019] In some examples, when the current system resource utilization value of the workload computing component exceeds a resource utilization threshold, an approximate embedding profile can be dynamically selected. This ensures that the embedding process executed according to the approximate embedding profile does not interfere with or restrict the use of other system resources while system resources execute instructions unrelated to the embedding process. When the current system resource utilization value drops below the resource utilization threshold, the processor of the data intelligence engine can switch to a second embedding process / profile to generate a more accurate embedding. This can correspond to a two-pass embedding approach, where the first pass generates an approximate embedding (e.g., using low CPU / GPU utilization and high throughput), and the second pass generates an accurate embedding (e.g., using high CPU / GPU utilization and low throughput). Generating an embedding based on an accurate embedding profile can use previously unavailable or system resources limited by embedding generation constraints to generate an embedding with a larger file size than that generated based on an approximate embedding profile.

[0020] The data intelligence engine can dynamically switch between two embedding profiles, for example, based on the resource utilization or availability of workload computing components. In some examples, the data intelligence engine may include a workload optimizer engine for initiating the switch between two embedding profiles and a processor for executing the embedding process. For example, the workload optimizer engine may identify which embedding profile to use when generating embeddings of data objects. When an embedding profile is assigned to the processor of the data intelligence engine, internal resources may also be assigned to execute the embedding process. Based on the applied embedding profile, the embedding process may utilize the maximum values ​​of in-memory queue limits, concurrent processing task limits, memory buffers, processors (e.g., CPU cores or GPUs), or other system components.

[0021] Once an embedding is created, the search component can utilize that embedding (e.g., to generate a response to the search using the corresponding data object). Search components, including but not limited to Retrieval Augmentation (RAG), Large Language Model (LLM), Semantic Search components, or other search components, can use exact or approximate embeddings, as described herein.

[0022] Throughout this disclosure, technical improvements are described. For example, a data intelligence engine can generate fast and efficient embeddings based on the state of the system workload (e.g., at the workload computing component) without significantly interfering with other processing. When the workload is idle or the system is less affected, the embeddings can be rerun, or more accurate embeddings can be generated than before. These later-generated embeddings can be more accurate and utilize more of the system's available processing power.

[0023] Figure 1 The illustration depicts a set of computing components 100 in a distributed system according to some examples of the disclosed techniques, including an object client 110 (e.g., a data provider), a data intelligence engine 120 (e.g., an embedding provider), a workload computing component 130 (e.g., a workload service provider), and a search component 140 (e.g., a search provider). In this example, the search component 140 may use or implement other applications such as chatbots, RAG, LLM, generative artificial intelligence (GenAI), semantic clustering, or searching data / objects. These distributed components may be implemented via one or more machines without departing from the spirit of this disclosure.

[0024] Object client 110 can be configured to provide data associated with the embedding process to data intelligence engine 120. For example, object client 110 can upload data from an object repository to data intelligence engine 120 (e.g., via a data path). The upload process can be implemented using a "PUT" call, and the download process can be implemented using a "GET" call.

[0025] Data from object client 110 can be sent to data intelligence engine 120 via application programming interface (API) 112. API 112 can process the data from object client 110, format the data for export, and send it to data intelligence engine 120.

[0026] Object client 110 can also be configured to provide data associated with an embedding to search component 140. For example, search component 140 may access the embedding (e.g., during a search process targeting data used to generate a response to a query / suggestion). The embedding may correspond to a data object stored at object client 110. Object client 110 may retrieve the data and provide it to search component 140, or in some examples, may provide the data in response to a request received from search component 140 for data corresponding to an embedding.

[0027] Data intelligence engine 120 can be configured to receive data from object client 110 (e.g., via data path component 122) and transform / store it as data objects. Data objects may include structured or unstructured information (e.g., text, images, audio, video, etc.). Data intelligence engine 120 may also include data path component 122, embedded computing component 124, and workload optimizer engine 126. Data path component 122 is configured to receive data and identify its communication path or initiation location. In some examples, data path component 122 may expose protocol endpoints, such as Amazon. ® S3 and Network File System (NFS) are used to receive data. Data path component 122 can receive data via protocol APIs such as OBJECT PUT or FILESYSTEM WRITE.

[0028] Embedding computation component 124 can be configured to generate embeddings using an embedding profile set at that point in time. If the embedding profile is set to approximate embedding, the data object can be added to a delayed processing queue maintained by the processor. Embeddings can be stored in a vector data repository 128 in an appropriate set associated with a bucket or namespace. Approximate embeddings and exact embeddings can be stored in separate sets.

[0029] Embedded computing component 124 can also be configured to use system resource utilization calculations to determine system resource utilization values. In some examples, system resources reside at workload computing component 130, which may be located remotely from data intelligence engine 120 and object client 110. System resource utilization values ​​can identify the amount of system resources utilized by data intelligence engine 120 when workload computing component 130 remotely provides services / functions. System resource utilization values ​​may include the amount of memory utilized, CPU or GPU consumption values, or utilization values ​​of other resources in workload computing component 130. Multiple utilization values ​​can be determined, including a first system resource utilization value and a second system resource utilization value.

[0030] In some examples, system resource utilization values ​​are calculated remotely from the data intelligence engine 120 and object client 110. This calculation can be performed by workload computing component 130, which can be located remotely from the data intelligence engine 120 and object client 110. In other examples, workload computing component 130 can be integrated with the data intelligence engine 120, and the calculations of workload computing component 130 are performed within the data intelligence engine 120 (e.g., as a component separate from workload optimizer engine 126).

[0031] System resource utilization values ​​can be determined in association with specific components of workload computing component 130 and the usage of those components (e.g., throughput, used / remaining storage). For example, embedded computing component 124 can identify the resources utilized by data path component 122 when performing services or other actions. Embedded computing component 124 and data path component 122 can be two components that can compete for system resources within data intelligence engine 120. In this sense, the resource utilization of embedded computing component 124 can be limited / adjusted so that it does not consume more and degrade the quality of services provided by data path component 122. The functionality provided by workload computing component 130 can be accessed by data intelligence engine 120 via API 132 of workload computing component 130.

[0032] In some examples, components of the data intelligence engine 120 (e.g., data path component 122, embedded computing component 124, or workload optimizer engine 126) may execute various processes that can be used as a calculation of the current system resource utilization. When the current system resource utilization exceeds a resource utilization threshold, the data intelligence engine 120 (as a whole or its components) may fail to execute these processes correctly. In this example, the workload computing component 130 may remotely calculate the workload of these components to determine whether the current system resource utilization exceeds the resource utilization threshold or is below / within that threshold.

[0033] Embedded computing component 124 can also be configured to dynamically switch between embedding profiles in association with system resource utilization values. For example, based on the resource utilization or availability of workload computing component 130, embedded computing component 124 may select a first embedding profile for use in generating the embedding of data objects. When an embedding profile is applied, data intelligence engine 120 may assign its internal resources to the embedding process and initiate the embedding process in light of the current embedding profile. Resources may include memory queues / buffers, concurrent processing capabilities, and processors (e.g., CPU cores or GPUs) based on the applied embedding profile. In some examples, data intelligence engine 120 may access these or other additional system resources (e.g., processors, memory / buffers, I / O, etc.) to generate an accurate embedding compared to those accessed during the generation of an approximate embedding.

[0034] When the current system resource utilization exceeds a resource utilization threshold, an approximate embedding can be dynamically selected so that the embedding process does not interfere with other processes. When the system resource utilization drops below the resource utilization threshold, the current system can switch to a second embedding profile (e.g., an accurate embedding process), which generates a more accurate embedding of the data object and uses the system resources currently available to generate the embedding.

[0035] When the current system resource utilization does not exceed a resource utilization threshold, accurate embedding can be dynamically selected. In this example, the embedding computation component 124 can assume that system resources are available to implement the embedding process. When the system resource utilization exceeds the resource utilization threshold, the current system can switch to a second embedding profile (e.g., an approximate embedding process), which generates an approximate embedding of the data object and delays the use of some system resources while performing other processing (e.g., generating embeddings using an AI model associated with the approximate embedding profile).

[0036] The embedded computing component 124 can also be configured to store embeddings in a vector data repository 128 or in other data storage formats (e.g., a Milvus vector database). The vector data repository 128 can refer to a database or storage facility that facilitates the storage and retrieval of unstructured data by embedding unstructured data (converting unstructured data into a numerical array expressing the original meaning of the data) and storing the resulting embedding vectors (unique fingerprints / values ​​for a piece of data). The vector data repository 128 can be a cloud-based database included as part of the data intelligence engine 120. The vector data repository 128 can store embeddings that encode related data into embedding vectors.

[0037] The workload optimizer engine 126 can be configured to monitor system utilization metrics, such as CPU, memory, and I / O, during the iterative process. When system utilization drops below a resource utilization threshold or other configurable value, the workload optimizer engine 126 can dynamically change the embedding profile from generating approximate embeddings to generating accurate embeddings, and initiate the generation of new embeddings (via the embedding computation component 124).

[0038] Search component 140 is configured to generate embeddings for queries / suggestions. In some examples, both approximate embeddings (as defined in the approximate embedding profile) and exact embeddings (as defined in the approximate embedding profile) can be generated. Both embedding types can be generated regardless of the current embedding profile.

[0039] Search component 140 is also configured to receive queries / prompts from an end-user device (not shown). In some examples, queries may be received via an interface to search component 140 (e.g., an LLM or a corresponding RAG agent). Search component 140 may be trained to initiate various processes, including finding relevant documents with information used by an AI model to generate responses. In some examples, search component 140 may implement an LLM or a corresponding RAG agent without departing from the spirit of this disclosure.

[0040] Search component 140 can determine one or more embeddings in the latent space that are near the semantic similarity value of a query / hint. In some examples, the embedding associated with the query / hint can be compared with stored embeddings to determine semantic similarity. For example, search component 140 may receive embeddings from vector data repository 128 or directly from data intelligence engine 120 and use these embeddings for search workload computation component 130 (e.g., retrieving related data). When accurate embeddings are available, they can be used to perform searches on data; otherwise, approximate embeddings can be used. Embeddings can be retrieved if they are relevant to the response to the query / hint (e.g., within the latent space).

[0041] In some examples, search component 140 can be configured to execute an information retrieval algorithm that compares the numerical representation of each item in the search query and the embeddings, and retrieves data that is probabilistically similar to the embedding representation of the query. In examples, search component 140 can perform cosine similarity to calculate a similarity score between the numerical representation of the query and the numerical representation of the embeddings. The numerical representations of the embeddings can be sorted according to the similarity score (e.g., from highest to lowest similarity), and the most similar can be identified as probabilistically relevant to the search query. In some examples, the embeddings are transformed back into text / image documents that can be processed by search component 140, and output is generated in response to the query.

[0042] Figure 2 The illustration shows a data intelligence engine 200 that can implement an example of the disclosed technology. The data intelligence engine 200 may include... Figure 2 The data path component 210, the embedded computing component 220, the workload optimizer engine 230, and the vector data repository 240 can respectively represent Figure 1 Examples of the data intelligence engine 120, data path component 122, embedded computing component 124, and workload optimizer engine 126 are provided, with additional details about the interactions and characteristics of each component presented herein.

[0043] Data path component 210 can be configured to receive data objects and store them at object repository 212. Data objects may include structured or unstructured information (e.g., text, images, audio, video, etc.). Data path component 210 can provide the data objects to embedded computing component 220 for additional processing.

[0044] In some examples, data path component 210 is configured as a public protocol endpoint, such as Amazon. ®S3 and NFS are used to receive data. In response to public endpoints, data path component 210 can receive data via protocol APIs such as OBJECT PUT or FILESYSTEM WRITE.

[0045] For a data object received by data path component 210, data path component 210 may notify receiver 224 of the embedded computing component 220 of the received data object. Receiver 224 may add the data object to its internal persistent queue (e.g., receiver queue). Based on the availability of free segments in shared memory 222, receiver 224 may request data with a memory location (e.g., in object repository 212).

[0046] In some examples, data path component 210 can transfer data from write buffer 216 (or from disk) to shared memory 222. If free memory is unavailable, the data object can be requested at a future point in time when the memory slot becomes available.

[0047] Upon receiving data, receiver 224 can notify processor 226 that a new data object has been received and can store it in the receiver queue. Processor 226 can compute the embedding using the embedding profile determined at that point in time. If the embedding profile is set to an approximate embedding, the data object can be added to the deferred processing queue 228 maintained by processor 226. In this example, the data object can be stored in the receiver queue until it is served, and the data object can be stored in the deferred processing queue during the delay period. The data object can be stored in both the receiver queue and the deferred processing queue simultaneously. In some examples, the embedding can be stored in a vector data repository 240 in a suitable set associated with a bucket or namespace. In some examples, approximate embeddings and exact embeddings can be stored in separate sets, buckets, or namespaces with links to each type of embedding. This link can correspond to the identifier or location of the embedding stored in the vector data repository 240, the deferred processing queue 228, or the receiver queue.

[0048] In some examples, the links associated with the embeddings can also be stored with the data object. For instance, a data object can be associated with an exact embedding accessible via a first link / identifier, and also with an approximate embedding accessible via a second link / identifier. Both links / identifiers that identify the corresponding embedding of the data object can be stored with the data object.

[0049] In some examples, processor 226 can identify object identifiers (IDs) from delayed processing queue 228 and check the data availability of shared memory 222. If the data is unavailable, processor 226 can send a request to data path 214 for the object ID of the data object in object repository 212. Once the data is delivered to shared memory 222, processor 226 can generate an embedding using a suitable embedding model / profile for accurate embedding.

[0050] In some examples, processor 226 may access one or more embedding profiles, each of which performs an embedding process (e.g., using different AI models) to generate embeddings with different levels of accuracy. For example, a first embedding profile may be associated with a first embedding process that generates an “approximate embedding” of a data object, and a second embedding profile may be associated with a second embedding process that generates an “accurate embedding” of a data object.

[0051] An embedding profile defines the rules used to initiate the embedding process to generate embeddings. An embedding can transform a data object into an embedding that represents the data object as points in a high-dimensional vector space. In some examples, embeddings can be generated using an imported rule base that identifies semantic relationships between words (e.g., Word2Vec, GloVe). An embedding can also be a programmatic transformation of a data object into a numerical array that expresses the original meaning of the data.

[0052] The workload optimizer engine 230 can be configured to determine system resource utilization values ​​for system resources at remote computing components (e.g., workload computing components). System resource utilization values ​​can identify the amount of system resources being utilized. This can include the amount of memory utilized, CPU or GPU consumption values, or utilization values ​​for other resources in the system. System resource utilization values ​​can be determined in association with system resources (e.g., throughput, used / remaining storage). Multiple utilization values ​​can be determined, including a first system resource utilization value and a second system resource utilization value.

[0053] The workload optimizer engine 230 can also be configured to monitor system utilization metrics, such as CPU, memory, and I / O, during the iterative process. When system utilization drops below a resource utilization threshold or other configurable value, the workload optimizer engine 230 can dynamically change the embedding profile from generating an approximate embedding to generating an accurate embedding, and initiate the generation of a new embedding (via processor 226).

[0054] During the monitoring process, the workload optimizer engine 230 can assign the processor 226 to different embedding profiles. For example, in accurate embedding mode, the processor 226 can actively servic both the receiver queue at receiver 224 and the delayed processing queue 228 simultaneously. Actively servicing these two components can help clear any backlog of remaining processes. The workload optimizer engine 230 can continue to monitor the resource consumption of the processor 226, and also monitor the total resource consumption of the system while performing servicing and other functionalities (e.g., outside of the embedding process). If the resource consumption exceeds a first / high watermark threshold, the processor 226 can switch back to approximate embedding mode.

[0055] Figure 3 This is an example message flow 300 between a data path component and an embedded computing component, based on some examples of the disclosed technology. In this example, components of the embedded computing component 303 can communicate with various components of the data path component 302, including a receiver 304, a processor 306, and a shared memory 308.

[0056] At operation 310, data path component 302 may expose a protocol endpoint to embedded computing component 303. The protocol endpoint may include, for example, s3 and NFS, allowing data path component 302 to receive data via a protocol API (e.g., OBJECT PUT or FILESYSTEM WRITE).

[0057] At operation 312, data path component 302 can receive data (e.g., via API).

[0058] At operation 314, data path component 302 can notify receiver 304 of embedded computing component 303 that a data object has been received. For example, for each data object received by data path component 302, receiver 304 can receive a notification that the data object has been received.

[0059] At operation 316, the receiver 304 of the embedded computing component 303 can add objects to the internal persistent queue of the embedded computing component 303.

[0060] At operation 318, receiver 304 of embedded computing component 303 can request data from data path component 302. In some examples, receiver 304 can identify whether a segment in shared memory 308 is available. Based on the availability of a free segment, receiver 304 can request data with a memory location. If free memory is unavailable, the receiver can request the data object at a future point in time when the memory slot becomes available.

[0061] At operation 320, data path component 302 can transfer data to shared memory 308 of embedded computing component 303. In some examples, data path component 302 can transfer data from its internal data repository (e.g., from a write buffer or from disk) to shared memory 308.

[0062] At operation 330, data path component 302 can notify receiver 304 of embedded computing component 303 that data has been sent to shared memory 308.

[0063] At operation 332, the receiver 304 of the embedded computing component 303 can notify the processor 306 of the embedded computing component 303 that data has been received / stored by the shared memory 308.

[0064] At operation 334, the processor 306 of the embedded computing component 303 can initiate an embedding process for new data using the current embedding profile. For example, the processor 306 can use the embedding profile currently set at this point in time to initiate the embedding process.

[0065] In some examples, if the embedding profile is set to approximate embedding, the data object can also be added to a persistent, delayed queue maintained by the processor. Embeddings can be stored in a vector data repository in a suitable collection associated with a bucket or namespace, with approximate and exact embeddings stored in separate collections.

[0066] At operation 336, the processor 306 of the embedded computing component 303 can store the embedding in a vector data repository.

[0067] Figure 4 The diagram illustrates an embedding profile data structure according to some examples of the disclosed technology. In some examples, the processor 400 of the embedding computing component may store multiple embedding profiles 402 that can be applied during a dynamic embedding process. The embedding profiles may be accessible via a link or a profile ID 410. In some examples, the profile ID 410 may uniquely identify an embedding profile accessible to the embedding computing component in the system. The profile ID 410 may be used by the embedding component to search for embedding profiles in a vector data repository.

[0068] The embedded profile can also define system resource utilization type 420. System resource utilization type 420 can define the maximum permissible system resource utilization value for components in the system or the entire system (e.g., including data paths), at which point intervention can be taken.

[0069] In some examples, embedding profile 402 can also define a maximum allowed system resource utilization value. For example, using the defined maximum value, the embedding process can adapt to compute embeddings at a capacity less than or equal to the maximum system resource utilization value defined in the profile. System resource utilization can be limited for generating embeddings when it processes services / functions from the data intelligence engine and other services / functions across the distributed system. The maximum allowed system resource utilization value under each profile can correspond to various system resources, including memory, number of threads, percentage of CPU or GPU utilization, and other system characteristics.

[0070] The embedding profile can also define an embedding model 430. The embedding model can correspond to a specific embedding model used when generating embeddings under the embedding profile.

[0071] At box 440, a first embedding profile is provided with a profile ID 410 (e.g., "profile A100"), a utilization type 420 (e.g., "exceeds a first threshold"), and an embedding model 430 (e.g., "model B100"). For example, the embedding computation component can access the embedding profile corresponding to the profile ID in a vector data repository to determine the embedding model to use. The embedding computation component can adjust the embedding process based on the utilization type defined in the profile (e.g., allowing embeddings to be generated if the system resource utilization value exceeds a first threshold).

[0072] At box 442, a second embedding profile is provided with profile ID 410 (e.g., “profile A200”), utilization type 420 (e.g., “exceeding the second threshold”) and embedding model 430 (e.g., “model B200”).

[0073] At box 444, a third embedding profile is provided with profile ID 410 (e.g., “profile A300”), utilization type 420 (e.g., “between the first threshold and the second threshold”) and embedding model 430 (e.g., “model B300”).

[0074] In some examples, the embedding profile can include thresholds or watermarks with different configurations (which can be used interchangeably) to define the embedding process for a specific embedding profile. Various thresholds or watermarks can be defined. For example, the embedding profile can define a low watermark corresponding to minimum embedding throughput, a low watermark corresponding to maximum CPU utilization, a low watermark corresponding to maximum GPU utilization, a low watermark corresponding to maximum memory utilization, a high watermark corresponding to idle time, a high watermark corresponding to minimum embedding throughput, a high watermark corresponding to maximum CPU utilization, a high watermark corresponding to maximum GPU utilization, a high watermark corresponding to maximum memory utilization, and so on.

[0075] In some examples, an approximate embedding profile may correspond to low CPU / GPU utilization and high embedding throughput, while an accurate embedding profile may correspond to high CPU / GPU utilization and low embedding throughput. Each of these embedding profiles may correspond to an identifier that identifies the data object in the object data repository and the location of the embedding.

[0076] In some examples, configurable resource utilization thresholds can help specify the resource utilization of various components, including embedded compute components and data path components. For example, processor configurable settings can define the number of threads, the number of concurrent tasks, the maximum / minimum size of internal memory buffers, the number of CPUs / GPUs assigned for utilization, and so on.

[0077] In some examples, the exact embedding model and the approximate embedding model can be different parts of the same model (e.g., 2d-matryoshka). For example, the approximate embedding model may contain the first few initial layers of the embedding model, while the exact embedding model may contain the complete set of layers of the same embedding model.

[0078] In some examples, the exact embedding model and the approximate embedding model can be different models, defined by different parameters, sizes, dimensions, and other features of the embedding model. As an illustrative example, an approximate embedding model may include 35 million parameters, 0.1 GB of memory, and 512 dimensions, while an exact embedding model may include 7 billion parameters, 261.49 GB of memory, and 4096 dimensions.

[0079] Figure 5 This is an example message flow 500 between a workload optimizer engine 502, a processor 504 with embedded computing components, and a vector data repository 506, based on some examples of the disclosed technology. In the vector data repository 506, multiple buckets, namespaces, or sets can be maintained for different embedding models, including approximate embedding buckets / namespaces 508 and exact embedding buckets / namespaces 510.

[0080] At operation 520, the workload optimizer engine 502 can monitor system utilization metrics. Depending on the determined frequency, the process of monitoring system utilization metrics can be iterative and repeatable. For example, the workload optimizer engine 502 can monitor system utilization metrics such as CPU, memory, and I / O.

[0081] At operation 522, workload optimizer engine 502 can assign an embedding profile to processor 504. The assignment of the embedding profile can be based on a system resource utilization metric identified by workload optimizer engine 502.

[0082] For example, when system resource utilization drops below a resource utilization threshold or other configurable value, workload optimizer engine 502 can dynamically change the embedding profile from generating approximate embeddings to generating accurate embeddings. In accurate embedding mode, processor 504 can actively serve both the receiver queue and the delayed processing queue simultaneously and clear the backlog of processing commands (read / write). Workload optimizer engine 502 can continue to monitor the total system resource consumption and processor resource consumption. If resource consumption exceeds a high-watermark threshold, workload optimizer engine 502 can assign processor 504 to switch back to approximate mode.

[0083] At operation 524, processor 504 can use the identified embedding profile to generate an embedding of a data object. For example, if the embedding profile corresponds to an approximate embedding profile, processor 504 can generate an approximate embedding, and if the embedding profile corresponds to an exact embedding profile, processor 504 can generate an exact embedding.

[0084] At operation 526, if the embedding profile corresponds to the approximate embedding profile, the processor 504 may store the approximate embedding in the approximate embedding bucket / namespace 508 of the vector data repository 506.

[0085] At operation 528, if the embedding profile corresponds to the exact embedding profile, the processor 504 may store the exact embedding in the exact embedding bucket / namespace 510 of the vector data repository 506.

[0086] At operation 530, the workload optimizer engine 502 can continue to monitor system utilization metrics. The process of monitoring system utilization metrics can continue to be executed iteratively, and this iteration can be repeated at a defined frequency.

[0087] At operation 532, the system utilization metric can be adjusted and compared with a threshold in the embedding profile. Based on the comparison between the system utilization metric and the threshold, workload optimizer engine 502 can switch / assign an embedding profile to processor 504. The embedding profile can be selected based on the updated system utilization metric.

[0088] At operation 534, processor 504 can use the identified embedding profile to generate an embedding of a data object. For example, if the embedding profile corresponds to an approximate embedding profile, processor 504 can generate an approximate embedding, and if the embedding profile corresponds to an exact embedding profile, processor 504 can generate an exact embedding.

[0089] At operation 536, processor 504 can initiate an embedded garbage collection process. For example, if system storage falls below a threshold, the garbage collector can reclaim space by deleting approximate embeddings of objects that have already been accurately embedded. In some examples, space can only be reclaimed if an accurate embedding profile exists.

[0090] Figure 6 This is an example message flow 600 between a data path component 601 and an embedded computing component 610, according to some examples of the disclosed technology. In the data path component 601, various components can communicate with each other, including a write buffer or disk 602 and a data path 604. In the embedded computing component 610, various additional components can communicate with each other, including a receiver 612, a processor 614, shared memory 616, and a delayed processing queue 618.

[0091] At operation 620, receiver 612 can request data from data path 604. For example, receiver 612 can identify whether a segment in shared memory is available. Based on the availability of a free segment, receiver 612 can request data with a memory location. If free memory is unavailable, receiver 612 can request the data object at a future point in time when the memory slot becomes available.

[0092] At operation 622, data path 604 can read data objects that were previously stored in write buffer or disk 602.

[0093] At operation 624, data path 604 can write a data object to shared memory 616 of the embedded computing component 610. For example, data path 604 can transfer data from a write buffer or disk to shared memory 616. If free memory is unavailable, the data object can be requested at a future time when a memory slot becomes available.

[0094] At operation 626, processor 614 can read data objects previously stored in shared memory 616. For example, after receiving data, receiver can notify processor 614 that a new data object has been received.

[0095] At operation 628, processor 614 can use the current embedding profile to generate an embedding of the data object. For example, processor 614 can use an embedding profile determined at that point in time to generate an embedding. If the embedding profile is set to an approximate embedding, the data object can be added to a delayed processing queue maintained by processor 614.

[0096] At operation 630, when the embedding profile is an approximate embedding profile, a series of operations can be initiated. For example, in response to generating an approximate embedding corresponding to the approximate embedding profile, processor 614 can write a data object to the deferred processing queue 618 of the embedding computing component 610 (operation 632), so that the deferred processing queue 618 can store the embedding for processing at a later time (operation 634).

[0097] At operation 640, when the embedding profile is an exact embedding profile, a series of operations can be initiated. For example, in response to generating an exact embedding corresponding to the exact embedding profile, processor 614 can actively serve receiver 612 (operation 642) and simultaneously actively serve delayed processing queue 618 (operation 644). In this example, serving both receiver 612 and delayed processing queue 618 can improve the processing / execution efficiency of processor 614 (e.g., when the system resource utilization value is below a threshold).

[0098] Figure 7 This is an example message flow 700 between a search component 702, an embedded computing component processor 704, and a vector data repository 706, based on some examples of the disclosed technology. In the vector data repository 706, multiple buckets, namespaces, or sets can be maintained for different embedding models, including approximate embedding buckets / namespaces 708 and exact embedding buckets / namespaces 710.

[0099] At operation 720, search component 702 can generate a query or receive a query / hint from an end-user device. In some examples, the query can be received via an interface of search component 702 (e.g., an LLM or corresponding RAG agent). The search component can be trained to initiate various processes, including finding relevant documents with information used by the model to generate a response. Search component 702 can determine one or more embeddings in the latent space near the semantic similarity value of the query / hint. In some examples, the embeddings associated with the query / hint can be compared with stored embeddings to determine semantic similarity values. If an embedding is relevant to the response of the query / hint (e.g., within the latent space), the embedding can be retrieved.

[0100] In some examples, search component 702 can be configured to execute an information retrieval algorithm that compares the numerical representation of each item in the search query and the embeddings, and retrieves data that is probabilistically similar to the embedding representation of the query. In examples, search component 702 can perform cosine similarity to calculate a similarity score between the numerical representation of the query and the numerical representation of the embeddings. The numerical representations of the embeddings can be sorted according to the similarity score (e.g., from highest to lowest similarity), and the most similar can be identified as probabilistically relevant to the search query. In some examples, the embeddings are used to obtain associated text / images, which can be used to generate answers. Search component 702 can retrieve the corresponding data (e.g., text / image documents) and process it to generate output in response to the query.

[0101] At operation 722, search component 702 can send a query to processor 704 of the embedding computation component. The embedding computation component can use an embedding profile to transform the query into an embedding. A single query can be transformed into one or more embeddings (approximate embeddings and exact embeddings) and can be compared with embeddings (approximate embeddings and exact embeddings) in a vector data repository.

[0102] At operation 724, the processor 704 of the embedded computing component can generate an embedding of the query for each available embedding profile. For example, an embedding process can be used to embed the query to generate both an exact embedding and an approximate embedding of the query.

[0103] At operation 726, processor 704 can assign weights to embeddings. For example, processor 704 can assign a first weight to the exact embedding of the query, which is greater than a second weight assigned to the approximate embedding of the query.

[0104] At operation 730, processor 704 can use the exact embedding of the query to search the exact embedding bucket / namespace 710 of vector data repository 706.

[0105] At operation 732, processor 704 can use the approximate embedding of the query to search the approximate embedding bucket / namespace 708 of vector data repository 706.

[0106] At operation 734, processor 704 can select an embedding. For example, the selected embedding may correspond to an exact embedding retrieved from exact embedding bucket / namespace 710 or an approximate embedding retrieved from approximate embedding bucket / namespace 708.

[0107] At operation 736, processor 704 can send the selected embedding and its associated text back to search component 702.

[0108] At operation 738, search component 702 can use the sent embedding to generate a response to the query.

[0109] Figure 8 This is an example method 800 for implementing a dynamic embedded process based on system resource utilization, according to examples of the disclosed technology. In some examples, the data intelligence engine can execute machine-readable instructions to perform the operations described throughout the disclosure.

[0110] At operation 806, the process may determine a first system resource utilization value for the system resources. The system resource utilization value may include the amount of memory utilized, CPU or GPU consumption, or the utilization value of other resources within the workload computing component used to perform a specific task or set of processes using system resources. In some examples, the system resource utilization value may be determined in association with a specific component of the workload computing component and the amount of that component utilized (e.g., throughput, used / remaining storage).

[0111] At operation 808, the process can determine a first embedding profile associated with a first system resource utilization value. For example, the first embedding profile (e.g., an approximate embedding profile) can be associated with a first AI model that generates an approximate embedding of the data object, and a second embedding profile (e.g., an accurate embedding profile) can be associated with a second AI model that generates an accurate embedding of the data object. When comparing the two embeddings, the approximate embedding process can generate an embedding using fewer system resources than the accurate embedding process, which can create an embedding that is smaller in size and has a lower level of accuracy than the accurate embedding. Using fewer system resources allows workload computing components and the data intelligence engine to perform other operations, such as I / O processes, memory utilization, and processing functions, while still creating embeddings without waiting for resources to become available for embedding creation.

[0112] In some examples, when the current system resource utilization value of the workload computing component exceeds a resource utilization threshold, an approximate embedding profile can be dynamically selected. This ensures that while system resources are executing instructions unrelated to the embedding process, the embedding process executed according to the approximate embedding profile does not interfere with or restrict the use of other system resources. When the current system resource utilization value drops below the resource utilization threshold, the data intelligence engine's processor can switch to a second embedding process / profile to generate a more accurate embedding. This can correspond to a two-pass embedding approach, where the first pass generates an approximate embedding (e.g., using low CPU / GPU utilization and high throughput), and the second pass generates an accurate embedding (e.g., using high CPU / GPU utilization and low throughput).

[0113] At operation 810, the process can initiate a first embedding process defined by a first embedding profile to generate an approximate embedding. For example, the workload optimizer engine can identify which embedding profile to use when generating embeddings of data objects. Internal resources can also be assigned to execute the embedding process when the embedding profile is assigned to the processor of the data intelligence engine. Based on the applied embedding profile, the embedding process can utilize the maximum values ​​of in-memory queue limits, concurrent processing task limits, memory buffers, processors (e.g., CPU cores or GPUs), or other system components.

[0114] At operation 812, this process can allow the use of system resources when resource consumption thresholds are exceeded. For example, when the current system resource utilization value of a workload computing component exceeds the resource utilization threshold, the embedding process, executed according to an approximate embedding profile, can avoid interfering with and restricting the use of other system resources while system resources execute instructions independent of the embedding process. In some examples, when the current system resource utilization value drops below the resource utilization threshold, the process can switch to a second embedding process / profile to generate a more accurate embedding.

[0115] At operation 814, the process may determine a second system resource utilization value. In some examples, the second system resource utilization value may be a resource utilization value of the same type as the first system resource utilization value. Thus, the second system resource utilization value may include the amount of memory utilized, CPU or GPU consumption values, or utilization values ​​for other resources in the workload computing components used to perform a specific task or set of processes using system resources.

[0116] At operation 816, the process can determine that the second system resource utilization value has dropped below a resource consumption threshold. For example, the process can determine the system resource utilization value and compare it with a resource utilization threshold to indicate that the resource utilization has decreased relative to a previous determination.

[0117] At operation 818, the process can switch to a second embedding profile. For example, the process can dynamically switch between two embedding profiles. The data intelligence engine may include a workload optimizer engine for initiating the switch between the two embedding profiles and a processor for executing the embedding process. For example, in response to a second system resource utilization value dropping below a resource utilization threshold, the process can switch to a second embedding profile that utilizes increased system resource usage compared to the first embedding profile to generate accurate embeddings of data objects.

[0118] In some examples, the embedding process can correspond to a two-pass embedding method, utilizing a switch from a first embedding profile to a second embedding profile. The first pass could be generating an approximate embedding (e.g., using low CPU / GPU utilization and high throughput), and the second pass could be generating an accurate embedding (e.g., using high CPU / GPU utilization and low throughput). Generating an embedding based on an accurate embedding profile can utilize previously unavailable or limited system resources for generating embeddings to generate an embedding with a larger file size than that generated based on an approximate embedding profile.

[0119] At operation 820, the process can initiate a second embedding process defined by a second embedding profile to generate an accurate embedding of the data object. For example, the workload optimizer engine can identify which embedding profile to use when generating the embedding of the data object. Internal resources can also be assigned to execute the embedding process when the embedding profile is assigned to the processor of the data intelligence engine. Based on the applied embedding profile, the embedding process can utilize the maximum values ​​of in-memory queue limits, concurrent processing task limits, memory buffers, processors (e.g., CPU cores or GPUs), or other system components.

[0120] In some examples, generating an embedding based on an accurate embedding profile can use system resources that were previously unavailable or limited by constraints on generating embeddings to generate an embedding with a larger file size than that generated based on an approximate embedding profile.

[0121] Figure 9 The illustrations depict various examples of computing components that can be used to implement dynamic embedded processes based on system resource utilization, according to the disclosed technology. Reference is now made. Figure 9 The computing component 900 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 9 In the example implementation, computing component 900 includes hardware processor 902 and machine-readable storage medium 904.

[0122] Hardware processor 902 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in machine-readable storage medium 904. Hardware processor 902 may fetch, decode, and execute instructions, such as instructions 906 to 914, to control processes or operations for burst preloading for available bandwidth estimation. Alternatively to, or in addition to, retrieving and executing instructions, hardware processor 902 may include one or more electronic circuits, including functional electronic components for executing one or more instructions, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other electronic circuits.

[0123] Machine-readable storage media (such as machine-readable storage media 904) can be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Therefore, machine-readable storage media 904 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), storage devices, optical discs, etc. In some examples, machine-readable storage media 904 can be a non-transient storage medium, where the term "non-transient" does not cover transient propagation signals. As described in detail below, machine-readable storage media 904 can be encoded with executable instructions, such as instructions 906 to 914.

[0124] Hardware processor 902 can execute instructions 906 to determine a first system resource utilization value for system resources. The current system resource utilization can correspond to, for example, memory utilization, CPU or GPU consumption, I / O throughput, or other system operations measured by workload computing components, which may be utilized by service / function calls from the data intelligence engine. The first system resource utilization value can be determined in response to a received data object.

[0125] The hardware processor 902 can execute instructions 908 to determine a first embedded profile associated with a first system resource utilization value.

[0126] Hardware processor 902 can execute instruction 910 to initiate a first embedding process defined by a first embedding profile. In some examples, the first embedding process can generate an approximate embedding of a data object. This instruction can allow system resources to be utilized when resource consumption thresholds are exceeded.

[0127] The hardware processor 902 can execute instruction 912 to determine a second system resource utilization value for system resources.

[0128] Hardware processor 902 can execute instruction 914 to switch to a second embedding profile. In response to switching to the second embedding profile, the instruction can initiate a second embedding process defined by the second embedding profile. The second embedding process can generate an accurate embedding of the data object.

[0129] Figure 10 A block diagram of an example computer system 1100 is depicted, illustrating various examples of the techniques disclosed herein that can implement the technologies described herein. The computer system 1000 includes a bus 1002 or other communication mechanism for communicating information, and one or more hardware processors 1004 coupled to the bus 1002 for processing information. The hardware processors 1004 may be, for example, one or more general-purpose microprocessors.

[0130] Computer system 1000 also includes main memory 1006, such as random access memory (RAM), registers, and / or other dynamic storage devices, coupled to bus 1002, for storing information and instructions to be executed by processor 1004. Main memory 1006 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 1004. When stored in a storage medium accessible to processor 1004, such instructions make computer system 1000 a special-purpose machine customized to perform the operations specified in the instructions.

[0131] The computer system 1000 also includes a read-only memory (ROM) 1008 or other static storage device coupled to the bus 1002 for storing static information and instructions for the processor 1004. Storage devices 1010, such as disks, optical discs, or USB thumb drives (flash drives), are provided and coupled to the bus 1002 for storing information and instructions.

[0132] The computing system 1000 may include a user interface module for implementing a GUI, which may be stored as executable software code executed by (multiple) computing devices in a mass storage device. By way of example, this module and other modules may include components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuit systems, data, databases, data structures, tables, arrays, and variables.

[0133] Generally, as used herein, terms such as “component,” “engine,” “system,” “database,” and “data repository” can refer to logic implemented in hardware or firmware, or to a set of software instructions that may have entry and exit points and are written in programming languages ​​such as, for example, Java, C, or C++. Software components can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be understood that software components can be called from other components or from themselves, and / or can be invoked in response to detected events or interrupts. Software components configured to execute on a computing device can be provided on computer-readable media, such as compressed disks, digital video disks, flash drives, magnetic disks, or any other tangible media, or provided as digital downloads (and may initially be stored in a compressed or installable format that requires installation, decompression, or decryption prior to execution). Such software code can be stored, partially or entirely, on a memory device executing the computing device for execution by the computing device. Software instructions can be embedded in firmware, such as EPROM. It will also be understood that the hardware components may include connected logic units, such as gates and flip-flops, and / or may include programmable units, such as programmable gate arrays or processors.

[0134] Computer system 1000 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, when combined with the computer system, makes computer system 1000 a special-purpose machine, or is programmed to be a special-purpose machine. According to one example of the disclosed techniques, the techniques herein are executed by computer system 1000 in response to processor(s) 1004 executing one or more sequences of one or more instructions contained in main memory 1006. Such instructions may be read into main memory 1006 from another storage medium, such as storage device 1010. Execution of the instruction sequence contained in main memory 1006 causes processor(s) 1004 to execute the process steps described herein. In alternative examples, hardwired circuitry may be used in place of or in combination with software instructions.

[0135] As used herein, the term "non-transient medium" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such non-transient media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 1010. Volatile media include dynamic memory, such as main memory 1006. Common forms of non-transient media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or tape cartridges and their networking versions.

[0136] Non-transient media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the transfer of information between non-transient media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including the wires that constitute bus 1002. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0137] Computer system 1000 also includes an interface 1018 coupled to bus 1002. Interface 1018 provides bidirectional data communication coupling to one or more network links connected to one or more local networks. For example, interface 1018 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for providing data communication connectivity to a corresponding type of telephone line. As another example, interface 1018 may be a Local Area Network (LAN) card to provide data communication connectivity to a LAN-compatible network (or a WAN component communicating with a WAN). Wireless links may also be implemented. In any such implementation, interface 1018 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0138] Network links typically provide data communication to other data devices over one or more networks. For example, a network link can provide a connection to a host computer or a data device operated by an Internet Service Provider (ISP) via a local network. The ISP, in turn, provides data communication services through a global packet data communication network now commonly referred to as the "Internet." Both local networks and the Internet use electrical, electromagnetic, or optical signals that carry digital data streams. Signals over various networks, as well as signals on network links and through interface 1018 (carrying digital data to and from computer system 1000), are example forms of transmission media.

[0139] Computer system 1000 can send messages and receive data including program code via (multiple) networks, network links, and interface 1018. In the Internet example, the server can send requested code for an application via the Internet, an ISP, a local network, and interface 1018.

[0140] The received code may be executed by processor 1004 upon receipt and / or stored in storage device 1010 or other non-volatile storage device for later execution.

[0141] Each of the processes, methods, and algorithms described in the preceding sections can be implemented in code components executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. One or more computer systems or computer processors can also operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). Processes and algorithms can be implemented partially or wholly in application-specific circuit systems. The various features and processes described above can be used independently of each other or can be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of this disclosure, and certain method or process boxes may be omitted in some implementations. The methods and processes described herein are not limited to any particular sequence, and the boxes or states associated with them can be executed in other suitable sequences, or can be executed in parallel, or in some other way. Boxes or states can be added to or removed from the disclosed examples. The execution of certain operations or processes can be distributed among computer systems or computer processors, residing not only within a single machine but also deployed across multiple machines.

[0142] As used herein, the circuit can be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to form the circuit. In implementations, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be shared partially or wholly among one or more circuits. Although various features or functional elements can be described or claimed separately as separate circuits, these features and functionalities can be shared among one or more common circuits, and this description should not require or imply a requirement for separate circuits to implement such features or functionalities. Where the circuit is implemented wholly or partially using software, such software can be implemented to operate with a computing or processing system capable of implementing the functionalities described herein, such as computer system 1000.

[0143] As used herein, the term “or” can be interpreted in an inclusive or exclusive sense. Furthermore, descriptions of resources, operations, or structures in the singular form should not be interpreted as excluding the plural form. Unless otherwise specifically stated or otherwise understood within the context in which they are used, conditional language (such as “can / could / might / may”, etc.) is generally intended to convey that certain examples include certain features, elements, and / or steps while others do not.

[0144] Unless otherwise expressly stated, the terms and phrases used in this document, and their variations, should be interpreted as open-ended rather than restrictive. Adjectives such as “regular,” “traditional,” “normal,” “standard,” “known,” and similar terms should not be interpreted as limiting the described items to a given time period or to items available at a given time, but rather should be interpreted as encompassing regular, traditional, normal, or standard techniques that may be available or known at any time now or in the future. The presence of extended words and phrases (such as “one or more,” “at least,” “but not limited to,” or other similar phrases in some instances) should not be interpreted as implying a narrower intent or requirement in instances where such extended phrases may not be present.

Claims

1. A data intelligence engine, comprising: The memory stores instructions; as well as A processor, communicatively coupled to the memory, and configured to execute the instructions to: Determine the first system resource utilization value for system resources; In response to the first system resource utilization value exceeding the resource utilization threshold, a first embedding profile is selected. The first embedding profile uses the system resources to generate an approximate embedding of the data object. The first embedding profile restricts the use of the system resources during the generation of the approximate embedding. The approximate embedding is generated based on the first embedding profile; Determine the second system resource utilization value of the system resources; as well as In response to the second system resource utilization value dropping below the resource utilization threshold, a second embedding profile is switched to, which utilizes the increased use of system resources compared to the first embedding profile to generate an accurate embedding of the data object.

2. The data intelligence engine according to claim 1, wherein the system resource utilization value defines the memory utilization of the workload computing component, the consumption of the central processing unit (CPU) or the graphics processing unit (GPU), or the I / O throughput.

3. The data intelligence engine according to claim 1, wherein the first system resource utilization value is calculated remotely from the data intelligence engine at the workload computing component, and the processor is further configured to: The first system resource utilization value is received from the workload computing component before the first system resource utilization value is determined.

4. The data intelligence engine according to claim 3, wherein the processor is further configured to: When the exact embedding is available, the exact embedding is provided to the search component for accessing the data at the object client corresponding to the data object; otherwise, the approximate embedding is provided.

5. The data intelligence engine according to claim 4, wherein the search component uses a large language model (LLM), a retrieval enhancement generation (RAG), or semantic search.

6. The data intelligence engine of claim 1, wherein limiting the use of the system resources causes the processor to: Simultaneously serving the receiver queue of the data object and the delayed processing queue of the data object.

7. The data intelligence engine according to claim 1, wherein the processor is further configured to: Initiate an embedded garbage collection process; and In response to generating the accurate embedding, the approximate embedding is deleted using the embedding garbage collection process.

8. The data intelligence engine of claim 1, wherein the approximate embedding and the accurate embedding are generated using different models defined in an embedding profile, the embedding profile including the first embedding profile and the second embedding profile, and the approximate embedding and the accurate embedding are stored in different embedding sets in a vector data repository.

9. The data intelligence engine according to claim 1, wherein the processor is further configured to: Generate a first link to the approximate embedding; and Store the first link together with the data object.

10. The data intelligence engine of claim 1, wherein the processor is further configured to: Generate a second link to the accurately embedded link; and Store the second link together with the data object.

11. The data intelligence engine according to claim 1, further comprising: Embedded computing components; as well as Workload optimizer engine The processor is further configured to: In response to selecting the first embedding profile or the second embedding profile, the embedding computation component and the workload optimizer engine are used to generate the approximate embedding or the accurate embedding, respectively.

12. The data intelligence engine according to claim 1, further comprising: Data path storage engine; The processor is further configured to: The protocol endpoint is exposed to receive the data object via the protocol application programming interface (API); as well as The accurate embedding is stored in a first vector database set for accurate embedding, and the approximate embedding is stored in a second vector database set for approximate embedding.

13. The data intelligence engine according to claim 12, further comprising: Receiver; The processor is further configured to: Receive the notification received by the data path storage engine from the data object; as well as In response to the notification, based on the availability of free segments in the shared memory, the data object with a position in the internal persistent queue is requested, and the data object is added to the position in the internal persistent queue by the receiver.

14. A method comprising: The data intelligence engine determines the first system resource utilization value. In response to the first system resource utilization value exceeding a resource utilization threshold, an approximate embedding of a data object is generated using a first model defined by a first embedding profile, the first embedding profile restricting the use of system resources during the generation of the approximate embedding; The approximate embedding is generated by the data intelligence engine based on the first embedding profile; The data intelligence engine determines a second system resource utilization value for the system resources; as well as In response to the second system resource utilization value falling below the resource utilization threshold, an accurate embedding of the data object is generated by a second model defined by a second embedding profile, which generates the accurate embedding by utilizing an increased use of the system resources compared to the first embedding profile.

15. The method of claim 14, wherein the system resource utilization value defines the memory utilization of the workload computing component, the CPU or GPU consumption, or the I / O throughput.

16. The method of claim 14, wherein the approximate embedding and the accurate embedding are generated using different sets of embeddings stored in a vector data repository.

17. The method of claim 14, further comprising: In response to selecting the first embedding profile or the second embedding profile, the data intelligence engine utilizes the embedding computing component and the workload optimizer engine to generate the approximate embedding or the accurate embedding, respectively.

18. A non-transitory computer-readable storage medium storing a plurality of instructions executable by a processor, the plurality of instructions causing the processor, when executed by the processor, to: Determine the first system resource utilization value for system resources; In response to the first system resource utilization value being lower than the resource utilization threshold, a first embedding profile is selected. Compared with a second embedding profile, the first embedding profile increases the use of system resources during the generation of accurate embedding. The accurate embedding of the data object is generated based on the first embedding profile; Determine the second system resource utilization value of the system resources; as well as In response to the second system resource utilization value exceeding the resource utilization threshold, switch to the second embedding profile, which restricts the use of system resources during the generation of the approximate embedding of the data object.

19. The non-transient computer-readable storage medium of claim 18, wherein limiting the use of the system resources causes the processor to: Simultaneously serving the receiver queue of the data object and the delayed processing queue of the data object.

20. The non-transient computer-readable storage medium of claim 18, wherein the processor further comprises: Expose protocol endpoints to receive the data object via the protocol application programming interface (API); and The accurate embedding is stored in a first vector database set for accurate embedding, and the approximate embedding is stored in a second vector database set for approximate embedding.

Images