Domain-specific retrieval language model

Abstract

The invention relates to a domain-specific retrieval language model. Various examples, systems, and methods are disclosed that relate to domain-specific document retrieval that combines custom vocabulary integration and embedded model update. A computing system may extract a plurality of segments from a collection of documents and generate a query corresponding to at least one segment. The computing system may identify terms that meet uniqueness criteria, and input these terms into a markup machine to create a vocabulary dataset. The vocabulary datasets, document segments, and queries can be used to update the embedding model to support retrieval and semantic alignment within private documents.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to International Application No. PCT / CN2024 / 129231, filed on November 1, 2024, the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0003] Improving the accuracy and performance of document retrieval in text-based information retrieval systems faces challenges. Some traditional methods rely on general text retrieval models that do not support specialized vocabulary or internal document-specific terms, leading to inefficiency and limited retrieval performance in proprietary environments. This approach can result in insufficient retrieval accuracy. Current systems are not configured and / or trained to identify associations between proprietary terms and relevant document content, leading to inconsistent query responses when handling domain-specific language. Furthermore, traditional methods rely on static, pre-trained tokenizers whose vocabulary is limited to publicly available terms, resulting in inefficiency and degraded retrieval performance due to the inability to identify proprietary terms. This approach can lead to redundant processing and ineffective handling of document-specific terms. Current methods are insufficient to handle the updating of proprietary terms over time, increasing the complexity of maintaining retrieval relevance in an ever-evolving document set. The challenges of implementing neural networks in embedding-based retrieval models contribute to inefficiency, impacting the accuracy and computational efficiency of text retrieval in domain-specific environments. Summary of the Invention

[0004] Implementations of this disclosure relate to systems and methods for improving text retrieval in document collections using embedding models trained with domain-specific vocabulary. The disclosed systems and methods can improve retrieval accuracy across private documents by combining machine learning models (e.g., large language models (LLMs)) with automatic term extraction and query generation. For example, the systems and methods according to this disclosure can extract domain-specific terms from documents and utilize these terms to generate representative queries. This approach can output vectors to capture semantic relationships between private terms, thereby aligning the retrieval output with the internal document vocabulary. Furthermore, the system and methods can update retrieval criteria at least based on parameters such as term frequency, relevance, or other probabilistic measures, thereby improving retrieval alignment with private vocabulary. By updating the embeddings based on these parameters to include newly identified terms, the system and methods improve retrieval performance without significant human intervention. This updating allows the retrieval model to maintain accuracy and efficiency when updating the private document collection.

[0005] Some implementations involve one or more processors including processing circuitry. The processing circuitry inputs one or more terms that satisfy a uniqueness criterion (property criterion) into a tokenizer, causing the tokenizer to tokenize the one or more terms into a lexical dataset. In some implementations, one or more terms corresponding to a certain domain are extracted from multiple documents. The processing circuitry extracts multiple sections from multiple documents, each containing one or more terms corresponding to the domain. The processing circuitry generates multiple queries corresponding to the multiple sections based at least on the multiple sections. The processing circuitry updates the embedding model based at least on the multiple queries, the multiple sections, and the lexical dataset.

[0006] In some implementations, multiple parts of multiple documents are extracted by segmenting the content of multiple documents into multiple parts based on at least one token. In some implementations, multiple queries corresponding to the multiple parts are generated by a large language model (LLM), which is trained to generate multiple queries based on the extracted content of the multiple parts and context. In some implementations, the content includes textual information within the multiple parts. In some implementations, the context includes the association between the multiple queries and the multiple parts.

[0007] In some implementations, generating multiple queries includes prompting a large language model (LLM) with multiple instructions based on the content and context of multiple parts, the multiple instructions corresponding to at least one parameter. In some implementations, one or more terms from multiple documents are extracted by the large language model (LLM), which is trained to identify multiple data segments based on uniqueness criteria. In some implementations, extracting one or more terms includes prompting the large language model (LLM) with multiple instructions to identify multiple terms in multiple documents that correspond to uniqueness criteria.

[0008] In some implementations, the uniqueness criteria include multiple frequencies of one or more terms below a threshold frequency in the tokenizer's vocabulary. In some implementations, the threshold frequency corresponds to occurrence frequency or co-occurrence frequency. In some implementations, the threshold frequency is set based on the multiple occurrences of multiple domain-specific terms across multiple documents. In some implementations, the embedding model includes a transformer model trained to transform multiple text inputs into multiple continuous vector representations based on processing multiple tokens through multiple multi-layer attention mechanisms to encode multiple semantic relationships between one or more terms.

[0009] Some implementations relate to a system. The system may include one or more processors for performing operations including extracting multiple parts of multiple documents containing one or more terms corresponding to a domain from multiple documents. The system may include one or more processors for performing operations including generating multiple queries corresponding to the multiple parts based at least on the multiple parts. The system may include one or more processors for performing operations including inputting one or more terms satisfying a uniqueness criterion to a tokenizer so that the tokenizer tokenizes one or more terms into a lexical dataset. In some implementations, one or more terms are extracted from multiple documents. The system may include one or more processors for performing operations including updating an embedding model based at least on multiple queries, multiple parts, and a lexical dataset.

[0010] In some implementations, multiple parts of multiple documents are extracted by segmenting the content of multiple documents into multiple parts based on at least one token. In some implementations, multiple queries corresponding to the multiple parts are generated by a large language model (LLM), which is trained to generate multiple queries based on the content of the extracted multiple parts and context. In some implementations, the content includes textual information within the multiple parts. In some implementations, the context includes the association between the multiple queries and the multiple parts.

[0011] In some implementations, generating multiple queries includes prompting a large language model (LLM) with multiple instructions based on the content and context of multiple parts, the multiple instructions corresponding to at least one parameter. In some implementations, one or more terms from multiple documents are extracted by the large language model (LLM), which is trained to identify multiple data segments based on uniqueness criteria. In some implementations, extracting one or more terms includes prompting the large language model (LLM) with multiple instructions to identify multiple terms in the multiple documents that correspond to uniqueness criteria. In some implementations, uniqueness criteria include multiple frequencies of one or more terms below a threshold frequency in the tokenizer's vocabulary.

[0012] Some implementations relate to a method. This method includes using one or more processors as input to one or more terms that satisfy a uniqueness criterion, such that the one or more processors tokenize the one or more terms into a lexical dataset. In some implementations, the one or more terms correspond to a domain and are extracted from multiple documents. The method includes using one or more processors to extract multiple parts of multiple documents containing the one or more terms corresponding to the domain. The method includes using one or more processors to generate multiple queries corresponding to the multiple parts, at least based on the multiple parts. The method includes using one or more processors to update an embedding model, at least based on the multiple queries, the multiple parts, and the lexical dataset.

[0013] The processors, systems, and / or methods described herein may be implemented by or included in at least one system. This system may include a system implementing one or more Large Language Models (LLMs). This system may include a system implementing one or more Small Language Models (SLMs). This system may include a system implementing one or more Visual Language Models (VLMs). This system may include a system for generating synthetic data. This system may include a system for generating synthetic data using AI. This system may include a control system for autonomous or semi-autonomous machines. This system may include a perception system for autonomous or semi-autonomous machines. This system may include a system for performing simulated operations. This system may include a system for performing digital twin operations. This system may include a system for performing optical transmission simulation. This system may include a system for performing collaborative content creation of 3D assets. This system may include a system for performing deep learning operations. This system may include a system for performing remote operations. This system may include a system for performing real-time streaming processing. This system may include a system for generating or rendering one or more of augmented reality content, virtual reality content, or mixed reality content. This system may include a system implemented using edge devices. This system may include a system implemented using robots. This system may include a system for performing conversational AI operations. The system may include a system that implements one or more multi-model language models. The system may include a system containing one or more virtual machines (VMs). The system may include a system that is at least partially implemented in a data center. The system may include a system that is at least partially implemented using cloud computing resources. Attached Figure Description

[0014] The following will describe in detail, with reference to the accompanying drawings, the system and method for improving the accuracy and efficiency of document retrieval in an embedding-based text retrieval pipeline, wherein:

[0015] Figure 1 This is a block diagram illustrating an example of a system according to some embodiments of the present disclosure;

[0016] Figure 2 This is a flowchart illustrating an example of a method for performing embedding-based document retrieval in a text retrieval pipeline according to some embodiments of the present disclosure;

[0017] Figure 3A This is a block diagram of an example generative language model system for implementing at least some embodiments of this disclosure;

[0018] Figure 3B It is a block diagram of an example generative language model including a transformer encoder-decoder for implementing at least some embodiments of this disclosure;

[0019] Figure 3C It is a block diagram of an example generative language model including a decoder-only transformer architecture for implementing at least some embodiments of this disclosure;

[0020] Figure 4 This is a block diagram of an example computing device for implementing at least some embodiments of this disclosure; and

[0021] Figure 5 This is a block diagram of an example data center used to implement at least some of the embodiments of this disclosure. Detailed Implementation

[0022] This disclosure relates to systems and methods for domain-specific retrieval language models, such as retrieval models applicable to document retrieval in specific and / or private domains, for example, domains associated with unique or private vocabulary (e.g., proprietary technical terms, domain-specific abbreviations, industry-specific terms, internal project codes, confidential product names, and / or any professional terminology). Some Large Language Modeling (LLM) techniques (e.g., embedding models) perform well on general text, such as retrieving data from textual information and / or generating responses to queries on textual information. However, such models may fail to retrieve data accurately when required to operate on documents containing vocabulary outside of general or publicly available data sources. For example, retrieval-based methods may chunk (e.g., form subsections of a document) documents from which data is to be retrieved and may embed these chunks. A user's query can be embedded, and comparisons and / or similarities between the embedded query and the embedded chunks are evaluated to select the document or chunks to retrieve. However, the tokenizer used by the model may not be trained on the semantics or syntax of terms in the external vocabulary (e.g., technical terms, proprietary terms), which may lead to degraded retrieval performance.

[0023] The systems and methods according to this disclosure can apply language models to domain-specific documents to detect terms in the documents that meet uniqueness criteria (e.g., not present in a predetermined vocabulary of the language model and / or occur at a frequency below a threshold frequency). The system can input the detected terms as tokens into a tokenizer and / or an embedding model. The system can generate a training dataset from the domain-specific documents, for example, by chunking the documents and having the language model generate queries (e.g., questions) corresponding to the document chunks. The system can determine training data samples containing a given chunk of a document and the corresponding queries generated for that given chunk. The system can update the embedding model (e.g., train the embedding model, fine-tune the embedding model, perform transfer learning on the embedding model, configure the embedding model) at least based on the training data samples.

[0024] For example, the system can input one or more terms that meet uniqueness criteria into the tokenizer, which are extracted from multiple documents. The system can also extract multiple parts of multiple documents (e.g., text fragments containing domain-specific terms). That is, the system can use a Large Language Model (LLM) to identify and extract these parts based on terms that meet uniqueness criteria (e.g., terms with frequencies below a threshold in the vocabulary). The system can generate multiple queries corresponding to these parts based on at least several parts (e.g., generating queries customized for proprietary terms). Based on the generated multiple queries, the system can use the tokenizer and update the embedding model based on at least the multiple queries and the multiple parts. Therefore, the system can improve retrieval performance by using domain-specific vocabulary, thereby improving the retrieval accuracy of the transformer-based model.

[0025] In some implementations, the system can use an LLM trained to identify multiple data segments based on uniqueness criteria (e.g., domain-specific). That is, the LLM can be trained to extract document fragments that correspond to domain-specific applications or documents and meet predefined criteria. For example, these fragments may include technical terms or proprietary vocabulary uncommon in general language datasets. The system can also generate multiple queries corresponding to the extracted fragments, where at least one of the fragments satisfies the uniqueness criteria. For example, the LLM can be used to generate multiple queries based on the content and context of the extracted fragments (e.g., relationships between terms in the extracted data).

[0026] In some implementations, the textual information in multiple sections may include multiple domain-specific terms (e.g., related to internal terms) that satisfy uniqueness criteria. That is, the context may include associations between multiple queries and multiple sections based on the frequency of occurrence or co-occurrence (e.g., the frequency of multiple terms appearing together) of one or more terms that satisfy the uniqueness criteria in multiple sections. Furthermore, the system can refine the uniqueness criteria by updating (or setting) the criteria based on threshold frequencies of one or more terms that are lower than the frequency of words in the tokenizer (e.g., technical terms), where the threshold frequencies can be adjusted according to the occurrence of domain-specific terms in the document. For example, the system may dynamically adjust the threshold frequencies to improve search results.

[0027] The system can also prompt the LLM with multiple instructions to identify multiple terms in multiple documents that correspond to at least one parameter (e.g., domain-specific terminology and / or topic). That is, the system can configure the LLM to respond to specific prompts designed to highlight terms relevant to the domain context. For example, the system can provide parameters such as domain-specific keywords or context to guide the LLM in identifying terms that can improve comprehension of specialized vocabulary retrieval. Furthermore, the embedding model can include a transformer model trained to transform multiple text inputs into multiple consecutive vector representations (e.g., vector embeddings representing relationships between technical terms). For example, multiple tokens can be processed through a multi-layer attention mechanism to encode multiple semantic relationships between one or more terms.

[0028] In some examples, one or more machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural render field (NERF) models, binarization models, transcription models, etc.) can be packaged as microservices, such as inference microservices (e.g., NVIDIA NIM), which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice may include the container itself and one or more models (e.g., weights and biases). In some cases, such as when one or more machine learning models are small enough (e.g., have a small enough number of parameters), one or more models may be contained within the container itself. In other examples, such as when one or more models are large, one or more models may be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In such implementations, the model can be accessed via one or more APIs (e.g., REST APIs). Therefore, in some implementations, the one or more machine learning models described herein can be deployed as inference microservices to accelerate the deployment of one or more models on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations to provide low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The one or more machine learning models described herein, along with acceleration infrastructure, can be included as part of a microservice, capable of deployment with a single command and / or orchestrated and automatically scaled using a container orchestration system on the acceleration infrastructure (e.g., reaching data center scale on a single device). Therefore, inference microservices may include one or more machine learning models (e.g., machine learning models optimized for high-performance inference), inference runtime software for executing one or more machine learning models and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identification, and / or other monitoring functions.In some implementations, the inference microservice may include software for in-situ replacement and / or updating of one or more machine learning models. During replacement or updating, the software performing the replacement / update may maintain user configurations for the inference runtime software and enterprise management software.

[0029] In some implementations, the systems and methods described herein can be deployed in conversational or intelligent self-service terminal applications. That is, the system can retrieve domain-specific information to generate responses or instructions, thereby supporting interactive, context-aware user interactions in environments such as retail or customer service. For example, self-service terminals, tablets, smart displays, or other devices may include one or more onboard processors (e.g., CPUs, GPUs, deep learning accelerators, SoCs) and memory and / or storage devices (e.g., for storing models, image databases, etc.). In some implementations, the self-service terminal / tablet / display can communicate with one or more locally hosted servers / computing devices and / or one or more remotely located servers / computing devices (e.g., located in one or more data centers) using one or more network interface cards (NICs) and / or data processing units (DPUs). In these examples, the self-service terminal can communicate with one or more machine learning models (e.g., language models, LLMs, VLMs, MMLMs, diffusion models, transformer models, NeRF, DNNs, etc.) hosted on local and / or remote servers using one or more APIs (e.g., but not limited to REST APIs).

[0030] In one or more embodiments, the systems and methods described herein can be deployed in gaming applications. That is, the system can manage the retrieval of domain-specific content (e.g., in-game terminology or custom player actions) by integrating private vocabularies into an embedded model. For example, a game console, PC, tablet, or other gaming device may include one or more onboard and / or remote processors (e.g., CPU, GPU, deep learning accelerator, SoC) and memory and / or storage devices (e.g., for storing game models, game assets, player data, etc.). These devices can use one or more machine learning models (e.g., diffusion models, transformer models, neural rendering field (NeRF) models, language models (e.g., LLM, VLM, MMLM, etc.), DNNs, etc.) to enhance the gaming experience, generate real-time dynamic content, and provide personalized user experiences based on in-game behavior or pre-stored player profiles. In some embodiments, the system can be deployed in a cloud gaming environment (e.g., NVIDIA's GeForce Now). In this scenario, client devices (e.g., smart displays, tablets, or game controllers) can be used to interact with the game, while machine learning models and / or visual rendering can occur on one or more remotely located server / computing devices (e.g., one or more data centers). The language model, AI processing, and rendering described herein can run in the cloud, processing player input received from end-user devices (e.g., based on controller, keyboard, mouse, joystick, AR / VR / MR, etc.), generating appropriate in-game responses, rendering content, and sending or transmitting the content to the end-user device. One or more data processing units (DPUs) and / or network interface cards (NICs) can be used when receiving and / or sending data from end-user or edge devices.

[0031] In some implementations, the systems and methods described herein can be deployed within video conferencing applications. That is, the system can retrieve context-specific vocabulary, such as terms unique to the platform or meeting context. For example, video conferencing devices (e.g., dedicated conferencing units, computers, tablets, and / or smartphones) may include one or more onboard processors (e.g., CPUs, GPUs, deep learning accelerators, SoCs) and memory and / or storage devices (e.g., for storing video, audio, or other communication-related data). The system can use machine learning models (e.g., diffusion models, transformer models, neural rendering field (NeRF) models, language models (e.g., LLM, VLM, MMLM, etc.)) to enhance video conferencing capabilities, including real-time or near real-time transcription, speaker separation, language translation, automatic speech recognition (ASR), and / or background noise reduction. In one or more implementations, the system allows users to interact with the video conferencing platform using natural language input. For example, users can issue voice commands to schedule, join, or leave meetings, or manage participants and screen sharing. When receiving data from an end user or one or more edge devices and / or sending data to an end user or one or more edge devices, one or more data processing units (DPUs) and / or network interface cards (NICs) may be used.

[0032] In some implementations, the systems and methods described herein can be deployed in robotic applications. That is, the system can retrieve and process task-specific technical terms. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE)), hardware-based optical flow accelerators (OFA), SoCs, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models). The robotic system can use these processors to execute one or more machine learning models (e.g., language models), enabling it to perform complex tasks autonomously or semi-autonomously, such as interacting with and / or manipulating static and / or dynamic objects, or navigating its environment using sensors such as cameras, LiDAR, RADAR, and ultrasonic sensors. This system can use sensor fusion technology to combine data from multiple sensors (e.g., cameras, infrared, lidar, radar, accelerometers) to create a comprehensive model of the robot's surrounding environment. This data can be processed locally on the robot or sent to a remote server to perform more computationally intensive tasks, such as 3D mapping or SLAM (Simultaneous Localization and Mapping). In one or more embodiments, data from individual robots (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze the data and distribute optimized commands to the entire robot fleet. In some embodiments, the machine learning models described herein (e.g., language models, VLM, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) can be used to enable the robot to perceive and reason about its environment and / or communicate with one or more other robots and / or people in the environment. In some embodiments, the robot can communicate with one or more locally hosted servers / computing devices and / or one or more remotely located servers / computing devices (e.g., in one or more data centers) using one or more network interface cards (NICs) and / or data processing units (DPUs).

[0033] In some implementations, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or in-vehicle experience (IX) applications. That is, the system can retrieve and present customized content or navigation commands. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, construction equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), hardware-based programmable vision accelerator (PVA) – which may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE), hardware-based optical flow accelerators (OFA), SoCs, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models) and memory and / or storage devices (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to execute one or more machine learning models (e.g., language models) to enable functions such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0034] refer to Figure 1 , Figure 1 This is an example block diagram of a system 100 according to some embodiments of the present disclosure. It should be understood that the arrangements and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to the arrangements and elements shown, or may be substituted for the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any combination and in any location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with… Figure 3A Example Generative Language Model System 300 Figure 3B-3C Example Generative Language Model (LM) 330 Figure 4 Example computing device 400 and / or Figure 5 The example data center 500 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0035] refer to Figure 1 , Figure 1 This is an example block diagram of a system 100 according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any combination and at any location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with… Figure 3A Example Generative Language Model System 300 Figure 3B-3C Example Generative Language Model (LM) 330 Figure 4 Example computing device 400 and / or Figure 5 The example data center 500 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0036] System 100 can implement at least a portion of an embedding-based text retrieval pipeline (hereinafter referred to as the "retrieval pipeline"), such as a document retrieval pipeline, a query matching pipeline, or a semantic search pipeline. System 100 can be used by any of the various systems described herein to process proprietary terms and / or generate domain-specific embeddings, including but not limited to internal document retrieval systems, customer support systems, knowledge management systems, research data indexing systems, enterprise search systems, compliance monitoring systems, and / or content recommendation systems.

[0037] Typically, a retrieval pipeline may include operations performed by system 100. For example, a retrieval pipeline may include one or more of a tokenization stage, an extraction stage, a query generation stage, and / or an embedding stage. At least one (e.g., each) stage of the retrieval pipeline includes one or more components of system 100 for performing the functions described herein. In some embodiments, one or more stages may be performed during the training of the AI ​​model. Furthermore, one or more stages may be performed during the inference stage of the AI ​​model.

[0038] System 100 (e.g., implementing a retrieval pipeline) can input one or more terms that meet uniqueness criteria (e.g., extracted from multiple documents) into tokenizer 108, causing tokenizer 108 to map (e.g., tokenize) one or more terms into a lexical dataset 118. Tokenizer 108 can perform the mapping by assigning a unique identifier to each term and constructing the terms based on their positional indexes in the lexical dataset 118. That is, to map terms, tokenizer 108 can analyze at least one (e.g., each) position and frequency of the terms, determine the relevance of the terms to a private context, and / or assign a corresponding token ID to at least one (e.g., each) term. For example, tokenizer 108 can specify identifiers based on the frequency and positional relevance of the terms (e.g., organizing the mapped terms into a format conducive to retrieval).

[0039] In some implementations, tokenizer 108 can create structured data in lexical dataset 118 by mapping terms to unique token IDs through analysis of term context, frequency, and / or domain relevance. At least one token ID can serve as a reference point, allowing embedding models 116 to directly retrieve terms. For example, a single token ID can be assigned to a proprietary term such as “TH500,” which one or more embedding models 116 can then recognize as a single unit rather than breaking it down into characters (e.g., “T”, “H”, “5”, “0”, “0”). This mapping allows one or more embedding models 116 to accurately reference terms using the structured lexical dataset 118, preserving their original integrity and avoiding unintended tokenization errors. Furthermore, tokenizer 108 can also store term metadata, such as co-occurrence counts and position indexes. The mapping can include organizing terms based on observed sequence patterns in documents, creating clusters that embedding models 116 can reference for deeper term recognition and vector encoding.

[0040] Furthermore, tokenizer 108 can be applied as input to one or more terms that meet uniqueness criteria, so that tokenizer 108 tokenizes one or more terms into lexical dataset 118. In some embodiments, implementing a retrieval pipeline may include system 100 extracting (e.g., chunking) multiple parts of multiple documents from multiple documents (e.g., input data 104). Furthermore, implementing a retrieval pipeline may include system 100 generating multiple queries corresponding to the multiple parts based at least on the multiple parts. Furthermore, implementing a retrieval pipeline may include system 100 updating (e.g., fine-tuning, training) embedding model 116 based at least on the multiple queries, the multiple parts, and the lexical dataset. Therefore, the retrieval pipeline can improve the relevance and efficiency of document retrieval across domain-specific sets and improve the accuracy of the embedding model in representing proprietary specialized terms, capturing semantic relationships in specialized vocabulary, and / or updating according to changes in domain-specific language and / or specialized terms.

[0041] In some implementations, the tokenization stage may be a stage in the retrieval pipeline where system 100 can identify and segment text into tokens for further processing by downstream models (e.g., via embedding model 116). System 100 may include at least one tokenizer 108. Tokenizer 108 may input one or more terms that meet uniqueness criteria (e.g., extracted from input data 104 containing multiple documents) to the tokenizer, causing tokenizer 108 to tokenize one or more terms into a lexical dataset 118 (e.g., storing proprietary proper nouns, domain-specific identifiers, unique technical terms, and / or product-specific abbreviations).

[0042] In some implementations, tokenizer 108 can tokenize terms into one or more tokens (e.g., words, subwords, or characters). For example, input data 104 may be a document containing technical descriptions such as “system architecture of the TH500 module,” “thermal resistance test of the T239,” or “configuration guide for the T281 GPU,” and tokenizer 108 may apply these inputs and / or one or more cues to guide one or more language models 112 (e.g., containing standards) to identify proprietary terms based on context. In this example, the cue may be “identify domain-specific terms related to system architecture and configuration that are not common in external datasets,” and language model 112 may process input data 104 to extract terms related to the proprietary vocabulary of an organization, company, entity, customer, and / or user. Furthermore, tokenizer 108 can tokenize the output of one or more language models 112 by assigning a unique token ID (e.g., 10123, 10234, 10345) to each identified term. For example, tokenizer 108 can assign 10123 as a unique ID to "TH500". In this example, new tokens could be new_tokens = ["TH500", "T239", "T281"], where at least one (e.g., each) token can correspond to a private term identified in the vocabulary. In some implementations, these tokens can be stored in a vocabulary dataset 118.

[0043] In some implementations, tokenizer 108 can perform tokenization by assigning unique token IDs to the terms provided by language model 112 after merging and removing duplicates. That is, tokenizer 108 can store at least one (e.g., each) term and its token ID in the vocabulary dataset 118. For example, if the vocabulary contains terms such as “TH500” and “EC077”, tokenizer 108 can assign a token ID to at least one (e.g., each) token and store it in the vocabulary dataset 118. In some implementations, when a compound term or unknown term does not match an existing vocabulary entry in the vocabulary dataset 118, tokenizer 108 can perform tokenization by splitting the compound term or unknown term into subwords. That is, tokenizer 108 can apply subword tokenization (e.g., byte-pair encoding). For example, “GPUMod” can be split into “GPU” and “Mod” for a finer-grained representation. In some implementations, tokenizer 108 can perform tokenization by processing terms as individual units. It should be understood that although various tokenization methods are described herein, tokenizer 108 may also perform various other tokenization methods or processes, such as, but not limited to, character-level tokenization (e.g., splitting text into individual characters) and / or sentence-level tokenization (e.g., dividing text based on sentence boundaries).

[0044] In some implementations, standards and / or multiple standards can define which terms are domain-specific (e.g., not present in general vocabulary, appearing at a frequency below a threshold frequency, such as less than 0.05% in standard documents, fewer than 50 instances in a corpus, and / or infrequent in industry-standard vocabulary, and / or primarily appearing in proprietary datasets). That is, tokenizer 108 can filter and / or extract terms from input data 104 (e.g., using language model 112), tag proprietary proper nouns (or terms), and / or store the identified proprietary terms in vocabulary dataset 118. Tokenizer 108 can perform filtering and / or extraction by prompting at least one neural network with context (e.g., “Identify domain-specific terms related to ‘chip’ and ‘GPU’ in the text above and provide the output in JSON format.”) and input data 104 including text and / or other content (e.g., files (e.g., videos, PDFs, images), research documents, product manuals, engineering specifications, project reports, and / or any proprietary technical materials).

[0045] In some implementations, tokenizer 108 may extract and / or recognize one or more terms, at least based on context and criteria, using at least one neural network (e.g., a large language model (LLM), a transformer model, a recurrent neural network (RNN), a support vector machine (SVM), and / or any domain-specific neural network model). That is, tokenizer 108 may generate terms (e.g., represented in JSON format) using language model 112. For example, language model 112 may recognize terms such as “TH500” as relevant domain-specific terms for tokenization. The output of language model 112 (e.g., in a specified or default format, such as JSON) may include these terms (e.g., private vocabularies) that may be tokenized (e.g., after merging and / or removing duplicates) for storage in vocabulary dataset 118. For example, private vocabularies may include internal product names and identifiers. In this example, proprietary terms for chips and / or graphics processing units (GPUs) may include “MODS”, “T264”, “T281”, “TH500”, “T239”, “T234”, “L4T”, “MP”, “NPI”, “351194” and “Blackwell”.

[0046] The tokenizer 108 may include any one or more artificial intelligence models (e.g., machine learning models, supervised models, neural network models, deep neural network models), rules, heuristics, algorithms, functions, or various combinations thereof to perform operations including token recognition and term extraction, such as filtering and classifying domain-specific vocabulary. That is, the language model 112 may be a neural network trained to generate document terms (e.g., domain-specific terms) and / or vocabulary. For example, the tokenizer 108 may prompt one or more language models 112 to extract one or more terms using multiple instructions to identify (e.g., search, classify, retrieve) multiple terms in the input data 104 that correspond to at least one parameter (e.g., a criterion).

[0047] In some implementations, tokenizer 108 may output a set of extracted terms (e.g., domain-specific vocabulary, unique identifiers, filtered proprietary terms, and / or any context-dependent tags). For example, language model 112 may output terms related to internal systems and processes that conform to specified uniqueness criteria (e.g., internal project codes, proprietary software names, technical component identifiers, company-specific abbreviations, and / or any custom hardware terms). For example, language model 112 may output proprietary technical terms specific to a company's product line. In some implementations, tokenizer 108 may merge extracted terms and remove duplicates from the terminology list before tokenization. Tokenizer 108 may process lists generated from multiple documents, identifying identical or equivalent entries. For example, if language model 112 outputs variations such as "EC077," "EC-077," and "EC077," tokenizer 108 may merge these entries into a single representation.

[0048] In some implementations, tokenizer 108 may generate a cue specifying uniqueness criteria for identifying domain-specific terms in a document set. For example, tokenizer 108 may generate a cue such as, “Identify internal system-specific terms that are unique to our organization and occur infrequently (e.g., less than 0.05% frequency or less than 10 times per million words in an external corpus),” thereby guiding language model 112 to facilitate the extraction of specific terms. In this example, the uniqueness criterion could be terms that occur in a general language dataset (e.g., a dictionary) with a frequency of less than 0.05% or less than 10 times per million words. In some implementations, one or more terms may be provided to tokenizer 108 to perform tokenization. Furthermore, language model 112 may be a neural network trained to also generate queries corresponding to document portions (described in more detail below with reference to query generator 124).

[0049] In some implementations, tokenizer 108 may maintain, execute, train, and / or update one or more machine learning models during the tokenization phase. In some implementations, language model 112 may include any type of neural network-based machine learning model capable of processing large text corpora (e.g., extracting unique specialized terms) to refine vocabulary for domain-specific context. For example, language model 112 may be trained and / or updated to identify and recommend word additions, as well as other vocabulary refinement tasks. One or more language models 112 may be or include transformer-based models (e.g., generative pre-trained transformer (GPT) models). In some implementations, one or more language models 112 may be or include bidirectional encoder representation (BERT) models derived from transformers. Tokenizer 108 may execute machine learning models to generate outputs. Tokenizer 108 may receive data to provide as input to language model 112; these inputs may include document metadata, extracted candidate terms, document fragments, and / or any data from other text sources.

[0050] The tokenizer 108 may include at least one neural network (e.g., one or more language models 112). One or more language models 112 may include an input layer, an output layer, and / or one or more intermediate layers (e.g., hidden layers), each layer may have corresponding nodes. That is, one or more language models 112 process input text features through structured layers to perform term extraction. For example, the input layer may accept document fragments and context for processing. For example, the output layer returns the extracted terms based on specified criteria. For example, the intermediate layers facilitate context-aware term recognition by encoding textual relationships.

[0051] In some implementations, system 100 can configure (e.g., train, update, fine-tune, apply transfer learning) language model 112 by modifying or updating one or more parameters (e.g., weights and / or biases) of individual nodes of language model 112 in response to an estimated output of the language model 112 (e.g., generated in response to a domain-specific term extraction task). The tokenizer 108 can be or includes various neural network models, including models for manipulating or generating data, including but not limited to proprietary lexical terms, domain-specific text features, lexical refinement suggestions, query samples, and / or various combinations thereof.

[0052] In some implementations, uniqueness criteria may include the following: multiple frequencies of one or more terms are below a threshold frequency in the tokenizer's vocabulary (e.g., less than 0.1% in a standard corpus, less than 50 occurrences in an open-domain language model, used only in internal documents, and / or absent or missing in external vocabularies). For example, tokenizer 108 may set the threshold frequency based on the multiple occurrences of multiple domain-specific terms in input data 104 (e.g., training manuals, technical specifications, engineering documents, company-specific data sources) (e.g., occurrences in specific documents, internal knowledge base frequencies, occurrences in product specifications, technical manual frequencies).

[0053] Furthermore, the tokenizer 108 can determine a threshold based on the frequency of domain-specific terms in internal documents (e.g., the number of times they appear in internal document segments, the number of times they are cited in specific project files, and / or the number of times they are mentioned in private datasets). That is, the tokenizer 108 can prioritize terms that are relevant to specialized content and / or context but are less common in general language (e.g., dictionaries, external corpora, general datasets) (e.g., document-specific, not frequently used externally (e.g., less than 0.05% frequency in external corpora), primarily used in private contexts, and / or uncommon in general language (e.g., less than 5 times per million words)).

[0054] In some implementations, the extraction phase may be a stage in a retrieval pipeline where system 100 can identify and segment relevant text portions for downstream processing. System 100 may include at least one extractor 120. Extractor 120 may extract multiple portions of multiple documents (e.g., input data 104) from multiple documents. That is, extractor 120 may segment multiple documents based on at least one marker (e.g., punctuation, paragraph separators, section headings, and / or any predefined keywords) to divide the content of multiple documents (e.g., sentences, paragraphs, sections, and / or any logical document divisions) into multiple portions. For example, in the extraction phase, extractor 120 may use punctuation to segment text into sentences for analysis. For example, extractor 120 may use section headings to create sections aligned with specific topics.

[0055] In some implementations, input data 104 may include any document, text content, or structured information. Extractor 120 may segment and / or chunk the input data 104 based on sentence boundaries, paragraphs, punctuation, keywords, and / or any document-specific markers. For example, extractor 120 may segment text based on punctuation to create sentence-level chunks for analysis. Extractor 120 may use predefined keywords or headings (e.g., “Overview” or “Details”) to segment the parts to be processed. Extractor 120 may use various techniques and markers to facilitate segmentation across documents. That is, extractor 120 may use various techniques and / or features to segment the input data 104 for processing by query generator 124.

[0056] In some implementations, extractor 120 can determine the block size to improve the quality of questions generated by query generator 124. That is, extractor 120 can determine and / or select the block size by adjusting hyperparameters based on content type and task requirements. For example, a larger block size can preserve more context within a fragment. For example, a smaller block size can improve specificity and relevance. To determine the optimal and / or desired block size, extractor 120 can sample document portions of different lengths, generate corresponding questions, and evaluate the quality of these questions. Extractor 120 can then iteratively tune the block size parameter to balance context preservation and specificity.

[0057] In some implementations, extractor 120 can determine the block size to enhance downstream processing, such as query generation performed by query generator 124. That is, extractor 120 can tune the block size by optimizing hyperparameters based on content type and segmentation requirements. For example, a larger block size can retain more contextual information within each segment, which can support comprehensive analysis but may introduce irrelevant content. Conversely, a smaller block size can improve focus and specificity but may reduce context, potentially impacting the relevance of subsequent processing tasks. To determine the block size, extractor 120 can sample segments of varying lengths, analyze the accuracy of content segmentation, and / or update hyperparameters based on data consistency across different segment sizes. That is, hyperparameters can be configuration settings used to control the segmentation process, such as block length thresholds or minimum context requirements (e.g., word count per segment, maximum tag length, or punctuation-based boundaries).

[0058] In some implementations, the extractor 120 can determine and / or select the block size by adjusting hyperparameters based on content type and segmentation requirements. To determine the block size, the extractor 120 can sample document portions of varying lengths, pair each portion with a corresponding question, and store these pairings in a training dataset 128. The extractor 120 can further iteratively tune the block size parameters to balance context preservation and specificity. In some implementations, the extractor 120 can sample document fragments to determine block sizes that support consistent pairings between questions and document portions, adjusting hyperparameters such as block length thresholds or minimum context requirements (e.g., word count per fragment, maximum tag length, or punctuation-based boundaries) to maintain the training dataset 128.

[0059] In some implementations, the query generation phase may be a phase in a retrieval pipeline, where system 100 may generate queries based on segmented document portions (e.g., representing possible search inputs). System 100 may include at least one query generator 124. Query generator 124 may generate multiple queries corresponding to at least multiple portions (e.g., provided by extractor 120). In some implementations, query generator 124 may apply extracted data (e.g., segmented) and / or one or more prompts to guide one or more language models 112 in generating queries. In this example, the prompt may be "Give multiple questions that correspond to the actual user scenario based on the provided context and output them in JSON format containing only the 'question' field," and language model 112 may process the extracted data (e.g., ":\nBootloader and BCTs\nLinux kernel binaries and source code (available via nv-tegra)\nReference filesystem (basic version, currently derived from Ubuntu)"). Linux)\nDriver Selection\nDemo Application\nMain Utility for Flashing\nMore information can be found here -L4T-Embedded\n1.5. Motivation\nThe TH500 chip boot uses BaseOS. Various benefits have been observed due to the use of an operating system widely used across various SW teams.\n1.5.1. Potential Advantages\nSolution Speed\t\nSpeed ​​of resolving various SLT / SSG issues. Since various SW teams use L4T as their default development tool, it is easier to reproduce and debug problems if they occur on L4T.\nThis is also evident on the TH500, where BaseOS issues are reproduced and resolved more quickly without much involvement from the MODS team. ") to output queries aligned with specific content. That is, portions of the extracted data can represent fragments so that at least one fragment can be used as input for query generation. Furthermore, in this example, the output of language model 112 could be "[{"question": "How does migrating to L4T benefit the resolution of SLT / SSG issues?" {"question": "What are the potential advantages of using L4T instead of TinyLinux for flash infrastructure?"}, {"question": "Why is L4T considered to be widely used in SW, and what benefits does this bring to the MODS team?"}] Furthermore, the output can be in JSON format or other suitable formats, such as, but not limited to, XML, CSV, plain text, and / or any structured format.

[0060] In some implementations, language model 112 can be a neural network, trained to generate queries corresponding to parts of a document. That is, language model 112 can be trained to extract domain-specific terms (as described above with reference to tokenizer 108) and can be trained to generate questions from various extracted and / or segmented input data 104. For example, language model 112 can be executed in parallel, thereby generating tokens and queries simultaneously, at least based on input data 104.

[0061] In some implementations, query generator 124 may prompt language model 112 with multiple instructions (e.g., “generate questions related to configuration details in the content,” “identify terms related to diagnostic procedures in this context,” “generate questions based on performance metrics outlined in this section”) based on the content of multiple sections (e.g., the content of the text) and context (e.g., how the text relates to the purpose of other sections or content of the document) and corresponding to at least one parameter (e.g., a standard or guideline). That is, query generator 124 can determine or identify the content of the text by analyzing keywords or terms in each section. For example, query generator 124 can identify a topic such as “system architecture” as content by matching words in a fragment. For example, query generator 124 can determine that “performance metrics” is content by detecting phrases related to technical specifications. Furthermore, query generator 124 can determine or identify the context of the text by associating text sections with surrounding and / or adjacent document sections. For example, query generator 124 can identify a section as background information by linking it to an introductory document fragment. For example, query generator 124 can determine that a troubleshooting context is a context by associating the troubleshooting context with segments marked as problem-solved.

[0062] In some implementations, the query generator 124 may determine or identify parameters by applying predefined settings related to document purpose or user needs. For example, parameters may be (but are not limited to) a specific question format, keyword density, sentence length, specificity level, and / or any semantic similarity threshold. Therefore, prompts may include questions and / or relevant contextual markers to guide the language model 112 by identifying and / or determining the content and context of each document section and by identifying and / or determining the parameters used to generate relevant questions.

[0063] In some implementations, query generator 124 may use language model 112 to output questions aligned with document fragment content (e.g., constructed as JSON objects, grouped by document segments, categorized by topic, and / or any custom query format). For example, query generator 124 may output queries that simulate a user search that might occur in a private document collection. In some implementations, query generator 124 may generate and / or create training dataset 128 to contain query pairs (e.g., generated by language model 112) and document chunks (e.g., segmented portions extracted from the original document by extractor 120). That is, language model 112 may generate queries and / or questions based on document chunks. Queries may simulate user input, representing how someone might search for information, for example, in a private document collection. For example, for a document chunk about the advantages of a particular technology, the generated query could be “What are the advantages of using technology X?”. In this example, the document chunk could represent the content that language model 112 should be trained to retrieve in response to a relevant query.

[0064] In some implementations, query generator 124 may maintain, execute, train, and / or update one or more machine learning models during the query phase. For example, language model 112 may be trained and / or updated to generate context-relevant questions based on extracted document fragments. Language model 112 may be or include a transformer-based model (e.g., a generative pre-trained transformer (GPT) model). Query generator 124 may execute language model 112 to generate output (e.g., for pairing with document parts and / or training data stored in training dataset 128). Query generator 124 may receive data to provide as input to language model 112, which may include extracted document parts, user-context-specific segmented content, domain-specific feedback, and / or any data from a structured dataset. In some implementations, system 100 can configure (e.g., train, update, fine-tune, apply transfer learning) one or more language models 112 by modifying or updating one or more parameters (e.g., weights and / or biases) of each node of one or more language models 112 in response to an estimated output of language model 112 (e.g., output generated in response to training metrics related to question relevance and accuracy). Query generator 124 can be or includes various neural network models, including models capable of manipulating or generating data, including but not limited to private vocabularies, structured question formats, domain-specific terminology identifiers, content segmentation, and / or various combinations thereof.

[0065] In some implementations, query generator 124 can generate and / or create training dataset 128 by pairing generated queries with corresponding document portions. That is, the output of language model 112 and the output of extractor 120 can be paired as query-document pairs for training purposes. For example, query generator 124 can use extracted segments from the system architecture and generate relevant queries to create paired data for training (e.g., stored in training dataset 128). For example, query generator 124 can generate queries for troubleshooting segments and pair these queries with document portions describing diagnostic steps. Furthermore, query generator 124 can perform pairing by matching queries with document portions based on content similarity metrics. For example, queries for performance parameters can (e.g., in the training dataset) be paired with document portions describing performance metrics or related technical data. In some implementations, training dataset 128 can be provided to tokenizer 108 to train, update, and / or implement one or more embedding models 116.

[0066] In some implementations, the embedding stage may be a stage in the retrieval pipeline where system 100 can generate embeddings to represent document portions and queries in a semantic space. One or more embedding models 116 may include any one or more artificial intelligence models (e.g., machine learning models, supervised models, neural network models, deep neural network models), rules, heuristics, algorithms, functions, or various combinations thereof to perform operations including embedding generation, such as transforming textual data into a vector representation for a retrieval task. That is, one or more embedding models 116 may be neural networks trained to map domain-specific terms to a dense vector space that captures semantic relationships.

[0067] In some implementations, one or more embedding models 116 may output embeddings of document blocks and queries (e.g., vectorized representations, context embeddings, domain-relevant feature vectors, and / or any query-document alignment indicator). For example, one or more embedding models 116 may process document blocks to output vector outputs that capture relational information. For example, embedding model 116 may generate embeddings for queries that are semantically aligned with document content. In some implementations, one or more embedding models 116 may be provided and / or implemented to perform embedding retrieval in private domains and / or secure data environments. That is, once the retrieval model has converged (i.e., achieved a threshold level of retrieval accuracy when matching queries with relevant documents), one or more embedding models 116 can be used to retrieve semantically similar documents across various private datasets.

[0068] In some implementations, one or more embedding models 116 can be configured (e.g., trained, updated, fine-tuned, or have transfer learning applied) by modifying or updating one or more parameters (e.g., weights and / or biases) of individual nodes within one or more embedding models 116 in response to the evaluation of estimated outputs of one or more embedding models 116 (e.g., generated in response to receiving training examples from training dataset 128 and tokenized vocabulary from vocabulary dataset 118). That is, embedding models 116 can be updated by learning from training pairs of document parts and queries. For example, vocabulary dataset 118 may include domain-specific terms for semantic mapping, while training dataset 128 may include pairs of query and document parts for supervised learning. In this example, embedding model 116 can use vocabulary dataset 118 and training dataset 128 to learn a semantically content-based mapping between queries and documents. Furthermore, embedding model 116 may include various neural network architectures, including models capable of semantic encoding, such as transformers, recurrent networks, feedforward networks, and / or convolutional layers.

[0069] In some implementations, one or more embedding models 116 may be configured (e.g., trained, updated, fine-tuned, or perform transfer learning, etc.) based at least on domain-specific (tagged) vocabulary in the vocabulary dataset 118 and / or training data in the training dataset 128. For example, a query for an entity (e.g., company, organization, department, sole proprietorship, research group) and / or a query for a domain-specific vocabulary paired with one or more examples from the corresponding document portion may be applied (e.g., via embedding model 116) as input to an encoder to generate an estimated output. The estimated output may be evaluated and / or compared with target embeddings (or expected outputs) from the training data corresponding to query pairings for entities in the training data and / or domain-specific vocabulary with one or more examples from the document portion, and one or more embedding models 116 may be updated at least based on the comparison results and / or evaluation metrics. For example, one or more parameters (e.g., weights and / or biases) of one or more embedding models 116 may be updated at least based on the output of a similarity score.

[0070] In some implementations, one or more embedding models 116 may be maintained, executed, trained, and / or updated during the embedding phase. In some implementations, the one or more machine learning models may include any type of deep learning machine learning model capable of performing semantic vector encoding (e.g., embedding query-document pairs into a dense space) to support efficient document retrieval. That is, the vector encoding may be a vector representation (or embedding) that captures the semantics or relationships between items such as words, entities, or domain-specific concepts in numerical form, thereby facilitating similarity-based retrieval. For example, vector embeddings generated from technical terms in proprietary documents can be used to identify semantically related documents or segments.

[0071] Furthermore, one or more machine learning models can be trained and / or updated to embed private vocabulary and context-sensitive document information, as well as other domain-specific tasks. One or more machine learning models can be or include transformer-based models (e.g., generative pre-trained transformer (GPT) models). One or more embedding models 116 can execute these models to generate embeddings. One or more embedding models 116 can receive data as input, which may include questions and / or domain-specific document fragments from user and / or internal data sources. In some implementations, one or more embedding models 116 may include an input layer, an output layer, and / or one or more intermediate layers (e.g., hidden layers), each intermediate layer may have corresponding nodes. That is, one or more embedding models 116 can process input data through multiple layers to generate embeddings. For example, the input layer receives vectorized text data. For example, the output layer generates the final embedding vector for a retrieval task. For example, the intermediate layers encode relationships between domain-specific terms.

[0072] refer to Figure 2 This flowchart illustrates an example flowchart of an embedding-based document retrieval method in a text retrieval pipeline according to some embodiments of this disclosure. It should be understood that such and other schemes described herein are presented only as examples. Other schemes and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to the schemes shown, and certain elements may be omitted entirely. Furthermore, many elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any combination and in any location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, one or more processors can be used to execute instructions stored in one or more memories to perform various functions. For example, in some embodiments, the systems and methods described herein may use one or more generative language models (e.g., such as...). Figures 3A-3C(as shown), one or more computing devices or components thereof (e.g., such as...) Figure 4 (as shown) and / or one or more data centers or components thereof (e.g., such as) Figure 5 This can be achieved as shown in the diagram.

[0073] Now for reference Figure 2 Each block of the method 200 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed using one or more processors executing instructions stored in one or more memories. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (standalone or in combination with other managed services), or as a microservice via an application programming interface (API), or as a plug-in to other products, etc. Furthermore, this document uses... Figure 1 Method 200 is described using the system shown as an example. However, this method may be additionally or alternatively performed by any system or any combination of systems, including but not limited to the system described herein.

[0074] Figure 2 This is a flowchart illustrating a method 200 for input, extraction, generation, and / or updating operations according to some embodiments of the present disclosure. Various operations of method 200 may involve improving the efficiency and relevance of document retrieval in an embedded system. Existing systems typically rely on and / or use generic terms and static query-document matching, which can lead to inaccurate retrieval results for domain-specific content. The problems of the prior art may arise when these systems lack adaptability to proprietary or specialized terminology, resulting in degraded retrieval performance and increased human intervention. Figure 2 The method 200 shown can address these technical issues by implementing dynamic embedding updates and domain-specific vocabulary integration, thereby improving retrieval accuracy and the alignment of the system with domain-specific content.

[0075] Method 200 includes, at block 210, inputting one or more terms that meet uniqueness criteria into a tokenizer. That is, the input operation allows processing circuitry to tokenize one or more terms (e.g., extracted from multiple documents) into a lexical dataset. Furthermore, processing circuitry can generate terms that meet predefined uniqueness criteria from multiple documents (e.g., corresponding to a specific domain) and input them into the processing circuitry to prompt its tokenization into the lexical dataset. For example, processing circuitry can use a language model (LLM) (or a neural network model, unsupervised model, decision tree) to identify domain-specific terms from various internal documents or content based on the low frequency of domain-specific terms in a general language dataset. Additionally, processing circuitry can perform the input operation by prompting LLM to detect infrequent technical terms and inputting these terms into a tokenizer for structured tokenization.

[0076] In some implementations, the processing circuitry can use an LLM trained to identify data segments based on uniqueness criteria to extract one or more terms from multiple documents. That is, the processing circuitry can use an LLM to identify and / or extract text containing terms that meet and / or satisfy a uniqueness threshold. For example, an LLM can scan document segments to identify terms that do not appear frequently in a standard dataset. Furthermore, the processing circuitry can refine the extraction to analyze terms relevant to the uniqueness criteria. That is, the processing circuitry can prioritize segments that highlight proprietary or technical terms (e.g., internal procedures, proprietary workflows, organization-specific standards). For example, an LLM can identify terms specific to a particular procedure or internally used component.

[0077] In some implementations, the processing circuitry can prompt the LLM with instructions to extract terms from multiple documents that meet uniqueness criteria. That is, the processing circuitry can guide the LLM to locate terms that are unique to a specific context and / or context-specific (e.g., industry-specific terms, rare terms, jargon). For example, the LLM can be instructed to extract terms that occur less frequently in a general dataset but are common in internal documents. Furthermore, the processing circuitry can include additional parameters in the instructions to narrow the LLM's modeling scope for domain-specific terms. That is, the processing circuitry can set parameters for term frequency or context usage (e.g., minimum frequency of occurrence, usage relevance, domain-specific context) to meet extraction requirements. For example, the instructions can facilitate the extraction of terms that appear only in proprietary technology sections.

[0078] In some implementations, the processing circuitry can define a uniqueness criterion based on the frequency of a term's occurrence in the tokenizer's vocabulary (e.g., stored in a vocabulary dataset) being below a threshold frequency. That is, the processing circuitry can set a frequency threshold that a term must meet to be considered unique. For example, the processing circuitry can determine that terms with an occurrence rate below 0.05% in an external dataset are eligible for inclusion in a private vocabulary. Furthermore, the processing circuitry can filter terms that meet the criteria for integration into the vocabulary dataset. For example, terms that meet the uniqueness threshold can be stored for future retrieval (e.g., by an embedding model).

[0079] In some implementations, the processing circuitry can set a threshold frequency based on the frequency of occurrence or co-occurrence of terms across multiple documents. That is, the processing circuitry can analyze the frequency of a term occurring alone or together with other terms to establish relevance. For example, terms that frequently co-occur in a particular document segment can be prioritized in the vocabulary dataset. Furthermore, the processing circuitry can determine the threshold by analyzing and determining the overall occurrence of domain-specific terms. That is, the processing circuitry can update the threshold frequency to capture the characteristics of domain-specific terms. For example, terms that frequently co-occur with technical terms can be included (e.g., even if their individual frequencies are low).

[0080] Method 200, at block 220, includes extracting multiple sections from multiple documents. That is, processing circuitry can identify and segment sections within documents to create sections for query generation. Documents may include terms corresponding to a domain (e.g., company, organization, department, research group, and / or any proprietary entity). For example, processing circuitry can segment text from a series of documents by dividing content at natural language markers (e.g., periods, question marks, or other punctuation marks) to create blocks of information. Furthermore, processing circuitry can also perform extraction using markers such as section headings or keywords to divide documents into defined sections. For example, processing circuitry can extract sections containing program information or step-by-step instructions.

[0081] In some implementations, the processing circuitry can extract multiple parts by segmenting multiple documents based on at least one marker. That is, the processing circuitry can apply segmentation markers (such as punctuation marks or specific boundary indicators) to divide the document content into multiple parts. For example, the processing circuitry can detect periods, question marks, or section headings to divide the content into multiple discrete parts for processing. Furthermore, the processing circuitry can also identify markers that indicate topic transitions or logical breaks in the document. In other words, the processing circuitry can use these markers as boundaries to separate related content.

[0082] Method 200 includes at block 230: generating queries based on at least a plurality of parts. That is, the processing circuitry can generate queries from each extracted document part, simulating content-related questions a user might ask. For example, the processing circuitry can form queries based on the action steps described in the document part, generating questions such as “How do I complete task X?” or “What are the steps in process Y?”. For example, the processing circuitry can generate queries reflecting common query structures. In some implementations, the processing circuitry can sample the document to determine block sizes related to query relevance and content specificity. For example, the processing circuitry can vary the block size, such as fragments of 50, 100, or 200 words, and determine which size yields context-relevant queries. Furthermore, the processing circuitry can generate blocks of the document and pair these blocks with corresponding generated questions, such that at least one pair (e.g., each) forms a dataset entry for training the embedding model. For example, the processing circuitry can align extracted blocks with questions reflecting the content of that block.

[0083] In some implementations, the processing circuitry can use a Large Language Model (LLM) trained on the content and context of multiple extracted segments to generate multiple queries. That is, the processing circuitry can use the LLM to interpret the segmented content and generate questions related to the information in at least one (e.g., each) segment. For example, the LLM can output queries corresponding to technical descriptions or procedure steps present in each content segment. Furthermore, the processing circuitry can apply the LLM to analyze the contextual relevance of at least one (e.g., each) document segment to refine query generation. That is, the processing circuitry can prompt the LLM to include terms or phrases consistent with the document content. For example, the LLM can generate queries such as “How to troubleshoot component X?”

[0084] In some implementations, the processing circuitry can generate queries based on content comprising textual information from multiple sections. The context can include the association between each query and its corresponding section (e.g., the relevance of the query to a document segment). That is, the processing circuitry can identify text in at least one (e.g., each) document section and determine the connection between the query and specific content from which the query originates. For example, the processing circuitry can verify that at least one (e.g., each) query reflects information in its corresponding document segment. Furthermore, the processing circuitry can maintain the association between the generated queries and the source document sections. That is, the processing circuitry can track links between the queries and their source segments. For example, a query about setup steps can link to a section describing setup instructions.

[0085] In some implementations, the processing circuitry can prompt the LLM to generate multiple queries based on the content and context of multiple sections. That is, the processing circuitry can provide prompts to the LLM to guide query creation based on content characteristics and contextual relevance within each section. For example, the instructions can direct the LLM to generate questions about technical specifications or operating procedures described in a section. Furthermore, the processing circuitry can include at least one parameter to further refine query generation. That is, the processing circuitry can apply parameters (such as specific keywords or query structures) to facilitate alignment with the intended search criteria. For example, parameters can specify the use of industry-specific terms relevant to each section.

[0086] Method 200, at block 240, includes updating the embedding model using a tokenizer based on multiple queries, multiple parts, and a vocabulary dataset. That is, the processing circuitry can train and / or update the embedding model to maintain alignment with specialized vocabulary and the relationships between queries, document fragments, and terms. For example, the processing circuitry can fine-tune the representation of proprietary language and technical concepts in the embedding model using tokenized terms (e.g., proprietary terms, unique technical identifiers, organization-specific acronyms) and their associated queries (e.g., generated user queries, procedural questions, technical tips) and blocks (e.g., segmented document parts, extracted content fragments, specific document segments). Furthermore, the processing circuitry can perform updates by refining the vector embeddings by applying the vocabulary dataset. For example, the processing circuitry can update the embeddings to prioritize context-relevant results for technical queries.

[0087] In some implementations, the processing circuitry may employ an embedding model that includes a transformer model trained to convert text input into a continuous vector representation. That is, the processing circuitry can use the transformer model to process text by generating embeddings that capture semantic relationships. For example, the transformer model (e.g., BERT-based models, GPT architectures, domain adaptive encoders) can encode terms used in a specific domain context and link them to technical concepts. Furthermore, the processing circuitry can perform embedding generation by passing tokens through multiple attention layers. That is, the processing circuitry can use attention mechanisms (e.g., contextual attention, hierarchical attention techniques) to place each term within the context of the text. For example, multi-layered attention can represent relationships between terms, thereby improving retrieval accuracy in collections of specialized content. In some implementations, the processing circuitry can refine the embeddings using data pairings from a training dataset and a vocabulary dataset. Specifically, the training dataset may contain paired query and document parts, and the vocabulary dataset may contain domain-specific tokens with unique token IDs. By pairing these datasets, the embedding model can establish associations between terms and context-related content. For example, embeddings generated from paired query-document examples can be updated to prioritize context-specific retrieval of technical queries.

[0088] The disclosed implementations can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots or robotic platforms, aerial systems, medical systems, rowing systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in driving or vehicle simulations, in robot simulations, in smart city or supervised simulations, etc.), systems for performing digital twin operations (e.g., combined with a collaborative content creation platform or system, such as, but not limited to, NVIDIA's OMNIVERSE and / or another platform, system, or service using USD or OpenUSD data types), systems implemented using edge devices, systems combining one or more virtual machines (VMs), and systems for performing synthetic data generation operations (e.g., Systems that are at least partially implemented in a data center, including systems that use one or more neural rendering fields (NERF), Gaussian splashing techniques, diffusion models, converter models, etc.; systems that perform conversational AI operations; systems that implement one or more language models (e.g., one or more large language models (LLM), one or more small language models (SLM), one or more visual language models (VLM), one or more multimodal language models, etc.); systems that perform optical transmission simulations; systems that perform collaborative content creation of 3D assets (e.g., using generic scene descriptor (USD) data, such as OpenUSD, computer-aided design (CAD) data, 2D and / or 3D graphics or design data and / or other data types); systems that are at least partially implemented using cloud computing resources; and / or other types of systems.

[0089] Example language model

[0090] In at least some implementations, language models, such as large language models (LLMs), visual language models (VLMs), multimodal language models (MMLMs), and / or other types of generative artificial intelligence (AI), can be implemented. Typically, language models can generate queries, identify terms, and determine associations within input data to facilitate operations in retrieval, tokenization, and embedding processes. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like, based on context provided in input prompts or queries. In implementations, these language models may be considered “large” because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. This disclosure allows for the implementation of LLM / VLM / MMLM, etc., for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / , etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM, etc., of this disclosure may be specifically designed for text processing, while in other embodiments, a multimodal LLM may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., USD format) and / or video. For example, a Visual Language Model (VLM) or more specifically a Multimodal Language Model (MMLM) may be implemented to accept images, videos, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0091] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. disclosed herein may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0092] In various implementations, unsupervised learning can be used to train LLM / VLM / MMLM / etc., where LLM / VLM / MMLM / etc. learn patterns from large amounts of unlabeled text / audio / video / image / design / USD / etc. data. Due to extensive training, in these implementations, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on large amounts of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques within a specific domain.

[0093] In some implementations, the LLM / VLM / MMLM / etc. of this disclosure can be implemented using various model alignment techniques. For example, in some implementations, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some implementations, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0094] In some implementations, LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / operation / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0095] In some implementations, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one implementation, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more implementations, the language models can be different versions of the same base model. In one or more implementations, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting implementations, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases (as defined by the provided hints).

[0096] In any such implementation, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more implementations, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more implementations, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. This association can include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more implementations, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in the curatorial dataset, for example, but not limited to this.

[0097] Figure 3A This is a block diagram of an example generative language model system 300 suitable for implementing at least some embodiments of the present disclosure. Typically, the example generative language model system 300 can process input data to generate queries, extract relevant terms, and support the generation of terminology through embeddings for retrieval applications. Figure 3A In the example shown, the generative language model system 300 includes a retrieval augmentation generation (RAG) component 392, an input processor 305, a tokenizer 310, an embedding component 320, a plugin / API 395, and a generative language model (LM) 330 (which may include LLM, SLM, VLM, multimodal LM, etc.).

[0098] At a high level, the input processor 305 can receive input 301, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 330 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 301 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 301 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In generative LM In some implementations of 330 capable of handling multimodal input, input 301 can combine text (or text that may be omitted) with image data, audio data, video data, design data, USD data, and / or other types of input data, such as, but not limited to, those described herein. Taking raw input text as an example, input processor 305 can prepare the raw input text in various ways. For example, input processor 305 can perform various types of text filtering to remove noise from the relevant text content (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.). In examples involving stop words (common words that often have little semantic meaning), input processor 305 can remove stop words to reduce noise and allow the generative LM 330 to focus on more meaningful content. Input processor 305 can apply text normalization, for example, by converting all characters to lowercase, removing accent marks, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency. These are just a few examples; other types of input processing can be applied.

[0099] In some implementations, the RAG component 392 (which may include one or more RAG models, and / or may be performed using the generative LM 330 itself) can be used to retrieve additional information to be used as part of the input 301 or prompt. The RAG can be used to enhance the input to the LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, such as where specific knowledge is required. The RAG component 392 can obtain this additional information from one or more external sources (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.), which can then be fed along with the prompt to the LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0100] For example, in some implementations, in addition to the data retrieved using RAG component 392, input 301 may be generated using query or model inputs (e.g., questions, requests, etc.). In some implementations, input processor 305 may analyze input 301 and communicate with RAG component 392 (or in some implementations, RAG component 392 may be part of input processor 305) to identify relevant text and / or other data to provide to generative LM 330 as additional context or information sources, typically from which responses, answers, or outputs 390 are identified. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 392 may use the RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits the chatbot related to a specific product sale or service, the RAG component 392 can retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history, along with the current inquiry / request, as part of the generative LM 330 as input 301.

[0101] RAG component 392 can use various RAG techniques. For example, it can use naive RAG ( The document is indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of RAG component 392, and the embeddings of chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to generative LM 330 to generate output.

[0102] In some implementations, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0103] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0104] As another example, Graph RAG can use knowledge graphs as a source of context or factual information. Graph RAG can be implemented using graph databases as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with data chunks extracted from larger documents (which may lead to a lack of context, factual correctness, linguistic accuracy, etc.) (or anything other than providing the model with data chunks extracted from larger documents), Graph RAG can also provide structured entity information to LLM / VLM / MMLM / etc. by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described in this paper use graphs as content stores and extract relevant document chunks, requiring LLM / VLM / MMLM / etc. to use them to answer questions. In such implementations, the knowledge graph may contain relevant textual content and metadata about the knowledge graph, or it may be integrated with a vector database. In some implementations, Graph RAG can use graphs as subject matter experts, where descriptions of concepts and entities relevant to the query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can aggregate the results. In such examples, the graph can store relevant factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some implementations, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0105] In any implementation, the RAG component 392 can implement plugins, APIs, user interfaces, and / or other functionalities to perform RAG. For example, LLM / VLM / MMLM / etc. can use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries on vector databases. For example, the graph database can interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0106] The tokenizer 310 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 330 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 330 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 310 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular implementation.

[0107] Embedding component 320 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 320 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0108] In some implementations where input 301 includes image data / video data, etc., input processor 301 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 320 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 301 includes audio data, input processor 301 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 320 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 301 includes video data, input processor 301 may extract frames or apply resizing to extracted frames, and embedding component 320 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 301 includes multimodal data, the embedded component 320 can use techniques such as early fusion (concatenation), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, data, video, design, etc.).

[0109] Other components of the generative LM 330 and / or generative LM system 300 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (e.g., the architecture used in models such as GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 320 can apply the encoded representation of the input 301 to the generative LM 330, and the generative LM 330 can process the encoded representation of the input 301 to generate an output 390, which may include response text and / or other types of data.

[0110] As described herein, in some implementations, the generative LM 330 can be configured to access or use (or be able to access or use) plugins / APIs 395 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 330 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 392) to access one or more plugins / APIs 395 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 395 to the plugin / API 395, which can process the information and return an answer to the generative LM 330, which can then use the response to generate output 390. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated using any number of plugins / APIs 395 until an output 390 that resolves each query / question / request / process / action / etc. from input 301 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 392, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 395).

[0111] Figure 3B This is a block diagram of an example implementation scheme where the generative LM 330 includes a converter encoder-decoder. Typically, the generative LM 330 can analyze input data to generate queries, identify domain-specific terms, and create embeddings for retrieval and contextual representation tasks. For example, suppose the input text (e.g., “Who discovered gravity”) is tokenized (e.g., by...) Figure 3A The tokenizer 310) is used for tokens such as words, and each token is encoded (e.g., by...). Figure 3A The embedding component 320 is a corresponding embedding (e.g., of size 512). Since these token embeddings do not typically represent the position of the tokens in the input sequence, positional encoding can be added to each token embedding using any known technique to encode the order relation and context of the tokens in the input sequence. Thus, (e.g., the resulting) embeddings can be applied to one or more encoders 335 of the generative LM 330.

[0112] In the example implementation, encoder 335 forms an encoder stack, where each encoder includes a self-attention layer and a feedforward network. In the example converter architecture, each token (e.g., a word) flows through a separate path. Therefore, each encoder can accept a sequence of vectors, pass each vector through the self-attention layer, then through the feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token. The self-attention score for a token pair can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention, where the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be cascaded to generate a context vector encoding the input. The attention projection layer 340 can transform the context vector into an attention vector (key and value) for the decoder 345.

[0113] In the example implementation, decoder 345 forms a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer that uses attention vectors (keys and values) from the encoder to focus on relevant parts of the input sequence, and a feedforward network. Similar to encoder 335, in the example converter architecture, each token (e.g., a word) flows through a separate path in decoder 345. During the first pass, decoder 345, classifier 350, and generation mechanism 355 can generate a first token, and generation mechanism 355 can apply the generated token as input during a second pass. This process can be repeated cyclically, generating tokens (e.g., words) and adding them to the output of the previous pass, and in subsequent passes applying token embeddings of positionally encoded composite sequences as input to decoder 345, generating one token at a time (called autoregression) until a symbol or token representing the end of the response is predicted. In each decoder, the self-attention layer is typically restricted to focusing only on preceding positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before the softmax operation. In the example implementation, the encoder-decoder attention layer operates similarly to the (e.g., multi-head) self-attention operation in encoder 335, except that it creates its query from the layer below it and obtains keys and values ​​(e.g., matrices) from the output of encoder 335.

[0114] Therefore, decoder 345 can output some decoded (e.g., vector) representation of the input applied during a particular pass. Classifier 350 can include a multi-class classifier comprising one or more neural network layers and a softmax operation that transforms logit probabilities into probabilities, the neural network layers projecting the decoded (e.g., vector) representation onto corresponding dimensions (e.g., one dimension for each supported word or token in the output vocabulary). Thus, generation mechanism 355 can select or sample words or tokens based on corresponding predicted probabilities (e.g., selecting the word with the highest predicted probability) and append it to the output of the previous pass, thereby generating each word or token sequentially. Generation mechanism 355 can repeat this process, triggering successive decoder inputs and corresponding predictions until a symbol or token representing the end of the response is selected or sampled, at which point generation mechanism 355 can output the generated response.

[0115] Figure 3C This is a block diagram of an example implementation where the generative LM 330 includes a decoder-only converter architecture. For example, Figure 3C The 360 ​​decoder can be used with Figure 3B The decoder 345 operates similarly, except... Figure 3C Each decoder 360 omits the encoder-decoder self-attention layer (because there is no encoder in this implementation). Therefore, decoders 360 can form a decoder stack, where each decoder includes a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, a symbol or tag indicating the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, and the resulting sequence (e.g., a corresponding embedding with positional encoding) can be applied to decoder 360. Figure 3B Similar to decoder 345, each tag (e.g., a word) can flow through a separate path in decoder 360, and decoder 360, classifier 365, and generation mechanism 370 can use autoregression to generate one tag at a time sequentially until a symbol or tag indicating the end of the response is predicted. Classifier 365 and generation mechanism 370 can be combined with... Figure 3B The classifier 350 and generation mechanism 355 operate similarly, wherein generation mechanism 370 selects or samples each consecutive output label based on the corresponding predicted probability and appends it to the output of the previous iteration, generating each label sequentially until a symbol or label representing the end of the response is selected or sampled. The architectures described herein, and others, are merely examples, and other suitable architectures may be implemented within the scope of this disclosure.

[0116] Example computing device

[0117] Figure 4This is a block diagram of an example computing device 400 suitable for implementing some embodiments of the present disclosure. Typically, one or more example computing devices 400 may perform multiple processing flows, including inputting data, performing tokenization, generating embeddings, and managing retrieval operations for data processing tasks. The computing device 400 may include an interconnect system 402 directly or indirectly coupled to the following devices: memory 404, one or more central processing units (CPUs) 406, one or more graphics processing units (GPUs) 408, a communication interface 410, input / output (I / O) ports 412, input / output components 414, a power supply 416, one or more presentation components 418 (e.g., one or more displays), and one or more logic units 420. In at least one embodiment, one or more computing devices 400 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 408 may include one or more vGPUs, one or more CPUs 406 may include one or more vCPUs, and / or one or more logic units 420 may include one or more virtual logic units. Accordingly, one or more computing devices 400 may include discrete components (e.g., a full GPU dedicated to computing device 400), virtual components (e.g., a portion of a GPU dedicated to computing device 400), or a combination thereof.

[0118] although Figure 4 The various boxes are shown as being connected to lines via interconnect system 402, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 418 (such as a display device) may be considered I / O component 414 (e.g., if the display is a touchscreen). As another example, CPU 406 and / or GPU 408 may include memory (e.g., memory 404 may represent a storage device in addition to the memory of GPU 408, CPU 406, and / or other components). Therefore, Figure 4 The computing devices described are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop computer," "desktop computer," "tablet computer," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are conceived in… Figure 4 Within the scope of computing devices.

[0119] Interconnect system 402 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 402 may include one or more bus or link types, such as Industry Standard Architecture (ISA) buses, Extended Industry Standard Architecture (EISA) buses, Video Electronics Standards Association (VESA) buses, Peripheral Component Interconnect (PCI) buses, Fast Peripheral Component Interconnect (PCIe) buses, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, CPU 406 may be directly connected to memory 404. Further, CPU 406 may be directly connected to GPU 408. In cases where there is a direct connection or point-to-point connection between components, interconnect system 402 may include a PCIe link to perform that connection. In these examples, a PCI bus is not required in computing device 400.

[0120] Memory 404 may include any of a variety of computer-readable media. Computer-readable media can be any available medium accessible by computing device 400. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0121] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 404 may store computer-readable instructions (e.g., representing programs and / or program elements, such as operating systems). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 400. As used herein, computer storage media does not include the signal itself.

[0122] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals (such as carrier waves or other transmission mechanisms) and include any information transmission medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the foregoing should also be included within the scope of computer-readable media.

[0123] CPU 406 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 400 to perform one or more of the methods and / or processes described herein. Each CPU 406 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. CPU 406 may include any type of processor and may include different types of processors depending on the type of computing device 400 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 400, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplemental coprocessors such as math coprocessors, computing device 400 may also include one or more CPUs 406.

[0124] In addition to or replacing CPU 406, one or more GPUs 408 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 400 to perform one or more of the methods and / or processes described herein. One or more GPUs 408 may be integrated GPUs (e.g., having one or more CPUs 406) and / or one or more GPUs 408 may be discrete GPUs. In embodiments, one or more GPUs 408 may be a coprocessor of one or more CPUs 406. GPUs 408 may be used by computing device 400 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 408 may be used for general-purpose computing on a GPU (GPGPU). GPUs 408 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPUs 408 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received from CPU 406 via a host interface). GPU 408 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 404. GPU 408 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 408 may generate pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0125] In addition to or replacing CPU 406 and / or GPU 408, one or more logic units 420 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 400 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 406, one or more GPUs 408, and / or one or more logic units 420 may perform any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more logic units 420 may be part of and / or integrated into one or more of CPUs 406 and / or GPUs 408, and / or one or more logic units 420 may be discrete components or otherwise external to CPUs 406 and / or GPUs 408. In embodiments, one or more logic units 420 may be coprocessors of one or more CPUs 406 and / or one or more GPUs 408.

[0126] Examples of logic unit 420 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems, one or more vision or vector processing units (VPU), one or more pixel processing engines (PPE)). Examples include 2D processing element arrays, each of which communicates with one or more other processing elements in the array in north, south, east, and west directions; one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units); vision processing units (VPUs); optical flow accelerators (OFAs); field-programmable gate arrays (FPGAs); neuromorphic chips; quantum processing units (QPUs); associative processing units (APUs); arithmetic logic units (ALUs); application-specific integrated circuits (ASICs); floating-point units (FPUs); input / output (I / O) elements; peripheral component interconnects (PCIs) or peripheral component interconnect fast (PCIe) elements; and so on.

[0127] Communication interface 410 may include one or more receivers, transmitters, and / or transceivers that allow computing device 400 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). Communication interface 410 may include components and functions that allow communication over any of a plurality of different networks (e.g., wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wireless bandwidth), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet). In one or more embodiments, logic unit 420 and / or communication interface 410 may include one or more data processing units (DPUs) for directly transmitting data received via a network and / or via interconnect system 402 to one or more GPUs 408 (e.g., the memory of one or more GPUs 408).

[0128] I / O port 412 allows computing device 400 to be logically coupled to other devices including I / O component 414, presentation component 418, and / or other components, some of which may be built into (e.g., integrated into) computing device 400. Illustrative I / O component 414 includes a microphone, mouse, keyboard, joystick, gamepad, game controller, disc satellite dish, scanner, printer, wireless device, etc. I / O component 414 provides a natural user interface (NUI) that handles air gestures, voice, or other physiological input generated by the user. In some instances, the input may be passed to appropriate network elements for further processing. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 400 (as described in more detail below). The computing device 400 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touchscreen technology, and combinations thereof, for attitude detection and recognition. Additionally, the computing device 400 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the computing device 400 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0129] Power supply 416 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 416 may provide power to computing device 400 to enable the components of computing device 400 to operate.

[0130] One or more presentation components 418 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 418 may receive data from other components (e.g., GPU 408, CPU 406, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0131] Example Data Center

[0132] Figure 5 An example data center 500 that can be used in at least one embodiment of this disclosure is illustrated. Typically, the example data center 500 can support large-scale data processing, manage distributed storage, and execute models for tokenization, embedding generation, and retrieval tasks. The data center 500 may include a data center infrastructure layer 510, a framework layer 520, a software layer 530, and / or an application layer 540.

[0133] like Figure 5 As shown, the data center infrastructure layer 510 may include a resource coordinator 512, grouped computing resources 514, and node computing resources (“nodes CR”) 516(1)-516(N), where “N” represents any integer, a positive integer. In at least one embodiment, the nodes CR 516(1)-516(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW”) and network network interface cards (“NW”). I / O devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some implementations, one or more nodes CR516(1)-516(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some implementations, nodes CR516(1)-516(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CR516(1)-516(N) may correspond to virtual machines (VMs).

[0134] In at least one embodiment, the grouped computing resources 514 may include separate groups of nodes CR516 housed within one or more racks (not shown) or within a plurality of racks in data centers (also not shown) located in different geographical locations. The separate groups of nodes CR516 within the grouped computing resources 514 may include grouped computing, networking, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of nodes CR516, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0135] Resource coordinator 512 may be configured or otherwise control one or more nodes CR516(1)-516(N) and / or grouped computing resources 514. In at least one embodiment, resource coordinator 512 may include a Software Design Infrastructure (“SDI”) management entity for data center 500. Resource coordinator 512 may include hardware, software, or some combination thereof.

[0136] In at least one implementation, such as Figure 5 As shown, framework layer 520 may include a job scheduler 528, a configuration manager 534, a resource manager 536, and / or a distributed file system 538. Framework layer 520 may include a framework of software 532 supporting software layer 530 and / or one or more applications 542 supporting application layer 540. Software 532 or application 542 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 520 may be, but is not limited to, a type of free and open-source software web application framework that can utilize the distributed file system 538 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM(Hereinafter referred to as "Spark"). In at least one embodiment, job scheduler 528 may include Spark drivers to facilitate the scheduling of workloads supported by various layers of data center 500. Configuration manager 534 may be able to configure different layers, such as software layer 530 and framework layer 520 including Spark and distributed file system 538 for supporting large-scale data processing. Resource manager 536 may be able to manage cluster or grouped computing resources mapped to or allocated to support distributed file system 538 and job scheduler 528. In at least one embodiment, cluster or grouped computing resources may include grouped computing resources 514 at data center infrastructure layer 510. Resource manager 536 may coordinate with resource coordinator 512 to manage these mapped or allocated computing resources.

[0137] In at least one embodiment, the software 532 included in software layer 530 may include software used by at least a plurality of portions of nodes CR516(1)-516(N), grouped computing resources 514, and / or the distributed file system 538 of framework layer 520. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0138] In at least one embodiment, the application 542 included in the application layer 540 may include one or more types of applications used by at least a plurality of portions of nodes CR516(1)-516(N), grouped computing resources 514, and / or the distributed file system 538 of the framework layer 520. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in combination with one or more embodiments).

[0139] In at least one implementation, any of the configuration manager 534, resource manager 536, and resource coordinator 512 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can free the data center operator of data center 500 from making potentially undesirable configuration decisions and potentially avoid underutilized and / or poorly performing portions of the data center.

[0140] According to one or more embodiments described herein, data center 500 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by calculating weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 500. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 500 by using weight parameters calculated through one or more training techniques (such as, but not limited to, those described herein).

[0141] In at least one implementation, the data center 500 may use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured to allow users to train or execute information inference services, such as image recognition, speech recognition, or other artificial intelligence services.

[0142] Example network environment

[0143] A suitable network environment for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 4 The implementation is carried out on one or more instances of computing device 400, for example, each device may include similar components, features and / or functions of computing device 400. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 600, an example of which is described in this document. Figure 5 To describe in more detail.

[0144] Components of a network environment can communicate with each other via one or more networks, which may be wired, wireless, or both. A network can include multiple networks or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0145] A compatible network environment may include one or more peer-to-peer network environments—in which case the network environment may not include a server—and one or more client-server network environments—in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein with respect to one or more servers can be implemented on any number of client devices.

[0146] In at least one implementation, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, job scheduler, resource manager, and distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for software supporting the software layer and / or one or more applications supporting the application layer. The software or application may include web-based service software or applications, respectively. In an implementation, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, free and open-source software web application frameworks, such as those used for large-scale data processing (e.g., "big data") using distributed file systems.

[0147] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions may be distributed across multiple locations from a central or core server (e.g., across one or more data centers distributed across states, regions, countries, globally, etc.). If the connection to the user (e.g., client device) is relatively close to the edge server, the core server may assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0148] One or more client devices may include the information described in this article. Figure 4At least some of the components, features, and functions of one or more example computing devices 400 described. By way of example and not limitation, a client device may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0149] This disclosure can be described in the general context of computer code or machine-usable instructions (including computer-executable instructions, such as program modules) that are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked via a communication network.

[0150] As used herein, the phrase “and / or” relating to two or more elements should be interpreted as meaning only one element, or a combination of elements. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0151] This document provides a detailed description of the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have anticipated that the claimed subject matter may also be embodied in other ways in combination with other current or future techniques to include combinations of different steps or steps similar to those described in this document. Furthermore, although the terms “step” and / or “box” may be used herein to refer to different elements of the method employed, such terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless and only if the order of individual steps is explicitly described.

Claims

1. One or more processors, said one or more processors including processing circuitry, said processing circuitry being used to: One or more terms that meet the uniqueness criteria are input into a tokenizer, which then tokenizes the one or more terms into a vocabulary dataset, the one or more terms being domain-specific and extracted from multiple documents; Extract from the plurality of documents a plurality of portions of the plurality of documents that contain one or more terms corresponding to the domain; Generate at least a plurality of queries corresponding to the plurality of parts; as well as The embedding model is updated based on at least the multiple queries, the multiple parts, and the vocabulary dataset.

2. The processors according to claim 1, wherein, The plurality of documents are segmented into the plurality of parts by segmenting the content of the plurality of documents based on at least one marker, and the plurality of parts of the plurality of documents are extracted.

3. The processors according to claim 1, wherein, The multiple queries corresponding to the multiple parts are generated by a large language model (LLM), which is trained to generate the multiple queries based on the extracted content and context of the multiple parts.

4. The processor according to claim 3, wherein, The content includes text information in the plurality of parts, and the context includes the association between the plurality of queries and the plurality of parts.

5. The processor according to claim 3, wherein, The generation of the multiple queries includes prompting the Large Language Model (LLM) with multiple instructions based on the content and context of the multiple parts, and the multiple instructions correspond to at least one parameter.

6. The processors according to claim 1, wherein, The term one or more from the multiple documents is extracted by a large language model (LLM), which is trained to identify multiple data segments based on the uniqueness criterion.

7. The processors according to claim 6, wherein, The extraction of the one or more terms includes prompting a large language model (LLM) with multiple instructions to identify multiple terms in the multiple documents that correspond to the uniqueness criteria.

8. The processors according to claim 1, wherein, The uniqueness criteria include multiple frequencies of the one or more terms that are lower than a threshold frequency in the vocabulary of the tokenizer.

9. The processors according to claim 8, wherein, The threshold frequency corresponds to the occurrence frequency or co-occurrence frequency, and wherein the threshold frequency is set based on the multiple occurrences of multiple domain-specific terms in the multiple documents.

10. The processors according to claim 1, wherein, The embedding model includes a transformer model that is trained to convert multiple text inputs into multiple continuous vector representations based on processing multiple tokens through multiple multi-layer attention mechanisms to encode multiple semantic relationships between the one or more terms.

11. The processors according to claim 1, wherein, The one or more processors are included in at least one of the following: A system that implements one or more large language model LLMs; A system that implements one or more small language models (SLMs); A system that implements one or more Visual Language Models (VLMs); A system for generating synthetic data; Systems for generating synthetic data using AI; Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system used to perform deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming processing; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; A system for performing conversational AI operations; A system that implements one or more multi-model language models; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

12. A system comprising: One or more processors for performing operations, said operations including: Extract from multiple documents multiple parts of the documents that contain one or more terms corresponding to a domain; Generate at least a plurality of queries corresponding to the plurality of parts; The one or more terms that meet the uniqueness criteria are input into a tokenizer, which then tokenizes the one or more terms into a lexical dataset, the one or more terms being extracted from the plurality of documents; and The embedding model is updated based on at least the multiple queries, the multiple parts, and the vocabulary dataset.

13. The system according to claim 12, wherein, The plurality of documents are segmented into the plurality of parts by segmenting the content of the plurality of documents based on at least one marker, and the plurality of parts of the plurality of documents are extracted.

14. The system according to claim 12, wherein, The multiple queries corresponding to the multiple parts are generated by a large language model (LLM), which is trained to generate the multiple queries based on the content and context of the extracted multiple parts.

15. The system according to claim 14, wherein, The content includes text information in the plurality of parts, and the context includes the association between the plurality of queries and the plurality of parts.

16. The system according to claim 14, wherein, Generating the multiple queries includes prompting the Large Language Model (LLM) with multiple instructions based on the content and context of the multiple parts, and the multiple instructions correspond to at least one parameter.

17. The system according to claim 12, wherein, The term one or more from the multiple documents is extracted by a large language model (LLM), which is trained to identify multiple data segments based on the uniqueness criterion.

18. The system according to claim 17, wherein, Extracting the one or more terms includes prompting the Large Language Model (LLM) with multiple instructions to identify multiple terms in the multiple documents that correspond to the uniqueness criteria.

19. The system according to claim 12, wherein, The uniqueness criteria include multiple frequencies of the one or more terms that are lower than a threshold frequency in the vocabulary of the tokenizer.

20. A method, the method comprising: One or more terms that meet a uniqueness criterion are input using one or more processors, such that the one or more processors tokenize the one or more terms into a lexical dataset, the one or more terms being domain-specific and extracted from multiple documents; The one or more processors are used to extract from the plurality of documents a plurality of parts of the plurality of documents that contain one or more terms corresponding to a domain; The one or more processors are used to generate multiple queries corresponding to the multiple parts at least based on the multiple parts; as well as The embedding model is updated using one or more processors based at least on the plurality of queries, the plurality of parts, and the vocabulary dataset.

Images