Multi-language and multi-dialect text processing is implemented by a graph neural network-based large language model

Abstract

The technical solution utilizes an LLM based on a GNN model to provide multilingual and multi-dialect text processing. The system can receive a query in a first language, the query corresponding to a wordpiece. The DPS can generate a first vector for a first wordpiece in response to inputting the wordpiece into a GNN trained based on a corpus, the wordpiece corresponding to a word graph that indicates associations between wordpieces in multiple languages. The DPS can generate a second vector for a second wordpiece in response to inputting the wordpiece into an LLM trained based on a plurality of interactions. The DPS can establish a priority for one of the first wordpiece or the second wordpiece based on at least a first weight of the first vector and a second weight of the second vector, and provide a response to the query using the first wordpiece or the second wordpiece according to the priority.

Description

[0001] Cross-reference to related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 589,663, filed October 12, 2023, pursuant to 35 USC § 119, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0002] This disclosure generally relates to a multilingual response framework based on machine learning (ML), including but not limited to providing responses covering a variety of languages, domain-specific languages ​​and regional dialects. Background Technology

[0003] The language used in a particular context or for a particular field or application will vary not only based on the language spoken, but also on regional dialects, local terms, or field-specific terminology. Summary of the Invention

[0004] The technical solution disclosed herein combines a graph neural network (GNN) model trained on multilingual corpus data with a large language model (LLM) trained on contextual user interactions to process word tokens across multiple layers and provide AI-based customized outputs for user input in corresponding languages, regions, and contexts. Generating context-appropriate text outputs across different languages ​​and regions can be challenging for AI models, especially in fields such as e-commerce sales, where regional nuances such as local terminology may be preferred communication methods. Implementing multilingual text classification based on AI models can also be difficult due to the lack of labeled data for multiple languages, regional dialects, and even domain-specific terminology across various languages. Even if data is available, training models for multiple languages ​​and arbitrary language variants can still consume significant computational and processing resources and is time-consuming.

[0005] This disclosed technical solution overcomes these challenges by combining corpus-level and example-level data to represent lexicalized word representations in a multi-layered lexical space and utilizing a weighting mechanism to evaluate and establish their relative priorities, while providing responses to user queries. This technical solution enables AI systems to effectively process text from multiple languages, dialects, domain-specific terms, and regions, expanding the global capabilities of the solution framework. It also provides domain-specific assistance (e.g., communicating e-commerce sales of high-tech services based on specific regions within a country), thereby generating customized responses for specific languages, regions, domains, and / or applications. Furthermore, this technical solution streamlines AI model training by leveraging transformer-based language models and GNN-based text classification frameworks, adapting to monolingual, cross-lingual, and multilingual scenarios, facilitating generative AI for domain-specific applications (e.g., online sales and customer support).

[0006] One aspect of this technical solution relates to a system. The system may include a data processing system comprising at least one processor coupled to a memory. The data processing system may be configured to receive a query in a first language. The query corresponds to a first lexical unit. The data processing system may be configured to: in response to the first lexical unit input to a graph neural network (GNN) model trained on a text-based corpus, generate a first vector of a first word, the first lexical unit corresponding to a word graph indicating associations between words in multiple languages, including the first language. The data processing system may be configured to: in response to the first lexical unit input to a transformer-based, multi-interaction-trained large language model (LLM), generate a second vector of a second word. The data processing system may be configured to: establish a priority for either the first word or the second word based at least on a first weight of the first vector and a second weight of the second vector. The data processing system may be configured to: provide a response to the query using at least one of the first word or the second word according to the established priority.

[0007] One aspect of this technical solution relates to a method. The method may include: receiving a query via a data processing system, the data processing system including one or more processors coupled to a memory, the query including a first word in a first language. The first word may correspond to a first lexical unit in a word graph, the word graph indicating associations between words in multiple languages, including the first language. The method may include: generating a first vector of the first word via the data processing system in response to the first lexical unit input to a graph neural network (GNN) model, the model being trained using a text-based corpus of the word graph. The method may include: generating a second vector of a second word via the data processing system in response to the first lexical unit input to a transformer-based, multi-interaction-trained large language model (LLM). The method may include: establishing a priority for either the first word or the second word via the data processing system, at least based on a first weight of the first vector and a second weight of the second vector. The method may include: providing a response to the query via the data processing system using at least one of the first word or the second word according to the established priority.

[0008] One aspect of this technical solution relates to a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to receive a query. The query may include a first word in a first language. The first word may correspond to a first lexical unit in a word graph indicating associations between words in multiple languages, including the first language. When executed, the instructions may cause the one or more processors to generate a first vector of the first word in response to inputs to a first lexical unit in a graph neural network (GNN) model trained using a text-based corpus of the word graph. When executed, the instructions may cause the one or more processors to generate a second vector of a second word in response to inputs to a first lexical unit in a transformer-based, multi-interaction-trained large language model (LLM). When executed, the instructions may cause the one or more processors to establish a priority for either the first word or the second word based at least on a first weight of the first vector and a second weight of the second vector. When executed, the instructions may cause the one or more processors to provide a response to the query using at least one of the first word or the second word according to the established priority. Attached Figure Description

[0009] Various aspects of this disclosure will be described below in detail, with reference to the numerous accompanying drawings, through non-limiting examples of exemplary embodiments of this disclosure.

[0010] Figure 1This is an example block diagram of a system according to an embodiment of the present solution, which improves the context of the query request using an ML model before inputting the query request into a domain-specific trained ML model for providing a response.

[0011] Figure 2 An example architecture of a computing system implemented in an embodiment of this disclosure is shown.

[0012] Figure 3 An example block diagram illustrating the structure of a query context model according to an embodiment of this scheme is shown.

[0013] Figure 4 An example block diagram of a system for multilingual and multidialect text processing using an LLM based on a GNN model is shown.

[0014] Figure 5 An example of a process for providing multilingual and multidialect text processing using an LLM based on a GNN model is shown. Detailed Implementation

[0015] Machine learning (ML) or artificial intelligence (AI) models can be trained to communicate across various languages ​​and domains. Training AI models for different communication types and domains may depend on the availability of data for such training. Technical solutions combine corpus-level datasets with context-based examples to represent vocabulary in a multi-layered lexical space and provide AI-based model outputs for user input across various languages, regions, contexts, and domains. AI models can be used to generate text outputs by processing text data covering different languages. However, generating or implementing communication or text in the context of a specific domain (e.g., e-commerce and sales) can be challenging for AI models, where domain-specific or language-specific contexts or regionally specific dialects may be a significant part of communication. In such systems, AI-based multilingual text classification capabilities may struggle to identify and classify actionable information in domain-specific communication (e.g., in professional sales). Building AI-based solutions that operate across a wide range of languages ​​and regions, domain-specific or regional dialects, may be hampered by a lack of labeled data for each language and region / domain-specific dialect, as such data may be insufficient or unavailable. Furthermore, training AI models for so many different languages, language variants, or domain-specific terms consumes significant computational and processing resources, making it time-consuming and energy-inefficient.

[0016] This technical solution overcomes these challenges by combining corpus-level datasets with example-level contextual information (e.g., examples of sales interactions from a specific domain or region) to represent each word in two different tokenization spaces and leveraging attention mechanisms to determine the association or relevance between two lexical units in the context of the task generating the response. For example, this solution addresses the complexity of processing multilingual text and providing efficient sales assistance, enabling AI-based multilingual systems to handle multiple languages, dialects, and regional terms, expanding their capabilities to target a global user base and operations. By combining training data from various languages ​​and dialects to simultaneously identify example-based interactions, this solution can be implemented even with limited training datasets that include both labeled and unlabeled data. Furthermore, this solution simplifies and accelerates the training of multilingual and multi-dialect models while conserving computational resources, as transformer-based language models and GNN-based multilingual text classification frameworks can be trained to provide responses covering monolingual, cross-lingual, or multilingual settings using word graphs derived from available text corpora. This technology can also help generate AI solutions for online sales campaigns and customer service that cover a wide range of languages, regions, and sectors.

[0017] This technical solution improves energy efficiency in multilingual AI systems and reduces the risk of model illusion and model drift. By integrating Graph Neural Networks (GNNs) with Transformer-based Large Language Models (LLMs), the system improves the training process through a hybrid approach that uses both labeled and unlabeled data. This enables efficient learning even with limited datasets, saving computational resources and reducing energy consumption associated with extensive model training. The implementation of a multi-layered lexicalization framework allows the model to generate context-relevant outputs while minimizing the likelihood of generating erroneous or irrelevant responses (e.g., illusions). The attention mechanism of the Query Context Model (QCM) further optimizes this process by dynamically evaluating the relevance of lexicals based on contextual relationships. This improves the accuracy of generated responses and maintains stability across various language variants. This innovative approach not only improves operational efficiency but also allows the AI ​​system to remain robust and reliable in a variety of applications, including domains using nuanced and specialized communication styles, such as e-commerce, domain-specific customer support, or specialized medical practices.

[0018] Figure 1This is an example block diagram of system 100, which uses one or more ML models within a GNN and LLM framework to provide responses to user queries. System 100 may utilize validation parameters and context control parameters to facilitate the generation of ML-based query responses. Example system 100 may include a data processing system (DPS) 102 that can transmit one or more queries 108 containing words 110 and responses 116 to these queries 108 via network 104 to user device 106. DPS 102 may include one or more queries 108 and their words 110, one or more model trainers, an NLP model 114, responses 116, a similarity function, and a lexicalization function 120 for generating lexical units 122. DPS 102 may include one or more encoders 124, decoders 126, a training dataset 130 stored in a database 132, and one or more query context models (QCM) 140. Each QCM 140 may include one or more graph neural networks (GNNs), transformers, nodes 146, node features 148, and contextual data structures (CDS) 150.

[0019] System 100 may include any combination of hardware and software utilizing ML models, encoders, and decoders to provide responses to user queries. System 100 may include data processing system 102, which may include any computing device that provides or performs the functions described herein. System 100 may include computing system 200 (e.g., Figure 2 (as described in the document), and may include one or more processors 210, which may be coupled to one or more memories 215, which may store any number of commands, instructions and / or data for performing or implementing the functions of system 100 or its components.

[0020] Query 108 may include any text-based input from user device 106 for finding specific information or content within a given dataset or domain. Query 108 may include natural language elements such as words, sub-words, phrases, sentences, or paragraphs, which may be referred to as words 110. Words 110 can then be lexicalized and processed by lexicalization function 120 of DPS 102. Lexicalization function 120 can decompose query 108 into words 110, generating a lexical unit 122 for each word 110 (e.g., each word, sub-word, or sentence), thereby facilitating efficient retrieval and understanding of information by DPS. Once lexicalized, DPS 102 can utilize QCM 140 (e.g., multi-layer array GNN 142 and transformer 144) to analyze contextual relationships and dependencies between lexical units 122 (e.g., nodes 146 in QCM 140 corresponding to lexical units 122) to gain a comprehensive understanding of the context of query 108.

[0021] Natural Language Processing (NLP) model 114 may include any ML model trained using multiple descriptions, texts, or corpora related to a specific domain. NLP model 114 may include any ML model based on deep learning techniques (e.g., computational frameworks). NLP model 114 may be trained to process and understand human language, thereby enabling or performing tasks such as text classification, sentiment analysis, language translation, and question answering (e.g., providing a response 116 to query 108) by leveraging patterns, semantics, and context within text data.

[0022] NLP model 114 can be an LLM trained on a large training dataset 130 of descriptions or text corresponding to any specific domain. For example, NLP model 114 can be trained on e-commerce based using a variety of textual data related to e-commerce, such as product descriptions, interactions between salespeople and customers, FAQs, or any other data related to responding to user questions in an e-commerce environment (e.g., for queries 108 regarding purchasing products or services in a specific domain or region). For example, NLP model 114 can be trained on any specific domain, such as medicine, law, engineering, technical support, history, geography, translation of different languages ​​or locales, or any other domain that users might explore with their queries 108.

[0023] NLP model 114 can be trained by model trainer 112 to provide responses 116 for a specific domain. The model trainer can include any combination of hardware and software used to train NLP model 114 using a specific training dataset 130. Model trainer 112 can guide the learning process of NLP model 114 by optimizing the parameters and weights of NLP model 114 based on a domain-specific dataset. Training dataset 130 can include input samples, such as user queries 108 or sentences from a text corpus, seller-consumer interactions, or other data, along with corresponding target responses or annotations, forming a supervised learning setting. During training, model trainer 112 can iteratively feed batches of input data into NLP model 114, which then predicts responses 116. Model trainer 112 can compare these predictions with the true responses from training dataset 130 and provide the model's prediction error. Through backpropagation, model trainer 112 can update the parameters of the NLP model, thereby minimizing the error and improving the NLP model's ability to generate accurate responses 116. This process continues for multiple training rounds until the NLP model 114 achieves a satisfactory performance level on the domain-specific dataset 130, thus forming a well-trained NLPML model.

[0024] The lexicalization function 120 may include any combination of hardware and software for dividing the query 108 into words 110 and assigning lexical units 122 to the elements. Lexical units 122 may include one or more elements of the query 108 or a portion of the words 110 of the query 108. Each lexical unit 122 may be represented as one or more values, or a vector, such as an embedding vector.

[0025] Encoder 124 may include any combination of hardware and software for transforming query 108 and its elements into a meaningful representation (e.g., contextual data structure 150), which may be input into QCM 140 to generate response 116. Encoder 124 may include a component of an ML model responsible for transforming input data (e.g., words 110 of query 108) into a meaningful and compact representation (e.g., contextual data structure 150). Encoder 124 may process text input, using lexicalization function 120 to lexicalize the input into smaller units (e.g., words or subwords), and mapping each lexical 122 to one or more corresponding values, embedding vectors, or feature vectors. Encoder 124 may utilize contextual data structure 150 to transform user word selections from query 108 into input that includes improved contextual and semantic information within the input data, enabling NLP model 114 to understand the relationships between different lexical 122.

[0026] Similar to encoder 124, decoder 126 may include any combination of hardware and software for converting response 116 into a desired or anticipated representation to be provided to user device 106. Decoder 126 may include functionality to convert response 116 into an output sequence to be provided to user device 106. For example, decoder 126 may receive contextual data structure 150 generated by QCM 140 and / or encoder 124, and use the contextual data structure to generate a coherent and context-appropriate response 116 to be provided to user device 106. Decoder 126 may be used to provide language translation, text generation, and summarization for the response to be sent to user device 106. Decoder 126 may leverage knowledge learned from encoder 124 to generate an accurate and coherent sequence of words for response 116.

[0027] The Query Context Model (QCM) 140 may include any combination of hardware and software for improving or enhancing the context of query 108 before it is input into the NLP model 114 that generates the response 116. The QCM 140 may include functionality for providing ML-based validation, improvement, or enhancement of the query context before it is input into the NLP model that generates the response to the user query 108. The QCM 140 may include a nested tree structure of nodes 146 that can be used to represent hierarchical relationships (e.g., identified in node features 148) between words 110 and / or lemmas 122 of query 108. The nested tree structure of the QCM 140 may include an array of GNNs 142 and transformers 144, thereby facilitating the integration of sequence dependencies and graph-based dependencies of nodes 146. QCM 140 can combine transformer 144 and GNN 142 to create context-rich representations (e.g., node features 148) between different nodes 146, which correspond to various words 110 in query 108 and similar words 110 (e.g., words, phrases) that are not included in query 108 but have related or similar meanings (e.g., based on similarity function 118 or based on co-occurrence in the dataset).

[0028] Graph Neural Network (GNN) 142 can include any deep learning architecture designed to operate on and model data with complex graph structures. GNN 142 can operate on graph-structured data in which the word 110 of query 108 can be represented as nodes 146. GNN 142 can include the ability to learn or develop associations in node features 148, thereby providing representations of contextual information and dependencies between nodes 146 in the graph. At each layer of the GNN, nodes 146 can aggregate information from their neighboring nodes 146 in the GNN 142 and update their own representations based on the information aggregated from their neighboring nodes 146. This iterative process of updating nodes 146 and node features 148 allows the GNN to propagate information throughout the graph, thereby capturing global dependencies and contextual relationships identified in node features 148 (e.g., for each layer of nodes 146).

[0029] Nodes 146 in the Query Context Model (QCM) 140 can include units, parts, or numerical constructs representing elements 110 of a query 108 within a nested tree structure. Nodes 146 can form building blocks of a graph, serving as distinguishing points representing specific tokens 122 of elements 110 input to the query 108. For example, the query 108 “Which mobile phones are best for playing games?” can include nodes 146 representing single words or phrases (e.g., element 110), such as “mobile phones” and “games”. Each node 146 can represent or indicate the identity of a given element 110 and can be used to establish associations with other nodes 146 in the graph, which can be used by the ML model to implement hierarchical and sequential dependencies in the query 108.

[0030] Node features 148 may include any features, attributes, or characteristics associated with each node 146. Node features 148 can provide contextual information for processing nodes 146 via the GNN and transformer 144 within the QCM 140. Node features 148 may include or be represented as vectors or tensors for encoding or numerically representing aspects of node 146. Node features 148 may include or represent word embedding vectors, positional encodings, semantic meaning, and contextual relevance. For example, in the query 108 “Which mobile phones are best for playing games?”, the node 146 representing “mobile phone” may include node features 148, such as its semantic embedding vector obtained from a pre-trained language model, its position in the query, and any relevant contextual clues that associate it with other nodes such as “games”. By associating nodes 146 with node features 148, the QCM 140 or response function 160 can create context-rich representations that enhance the understanding and processing of complex queries 108, thereby generating more accurate and relevant responses.

[0031] Transformer 144 may include an architecture for natural language processing tasks, characterized by its attention mechanism and parallel processing capabilities. For example, transformer 144 can process input data in parallel (e.g., in contrast to serial processing), making it more efficient in processing and better suited for long-distance dependencies. Transformer 144 includes a self-attention mechanism that allows it to capture contextual relationships between words in query 108 by assigning different weights to each word 110 based on the importance of other words in the same sequence (e.g., the same layer) for each word 110. Through self-attention, transformer 144 can effectively learn complex language patterns and contexts, making it suitable for tasks such as machine translation, sentiment analysis, and text generation.

[0032] After processing or lexicalizing the query word 110, the lexical units 122 and / or words 110 (e.g., their embedding vectors) can be processed by layers of transformer 144, which can generate contextual relationships between lexical units of the query word 110, as well as contextual relationships between lexical units of other words 110 (e.g., words or phrases) not included in query 108 but relevant to query 108 in meaning and context. A graph structure can be constructed in which nodes 146 represent lexical units 122 of element 110. Relationships between lexical units 122 can be represented as node features 148, allowing the incorporation of graph-based information. A GNN-like information aggregation process can aggregate data from adjacent nodes 146 in the structure, enabling QCM 140 to capture contextual information and global relationships between nodes 146. The outputs of the layers of transformer 144 and GNN 142 can be combined to create a hierarchical contextual representation, forming a multi-layered nested tree structure.

[0033] QCM 140 can utilize a similarity function 118 (e.g., Euclidean distance or cosine similarity function) to generate weights and apply them to lexical units 122, and determine relationships (e.g., similarity) between nodes 146. For example, based on the similarity function 118, QCM 140 can determine that some nodes 146 (e.g., corresponding to some lexical units 122) are more similar to each other (e.g., more relevant to each other) than other nodes 146. Such relationships between nodes 146 can be stored as node features 148 (e.g., stored in a repository 132 such as a hard drive). Associations between nodes 146 can include indications of relationships (e.g., similarity or contextual association) between nodes 146 to describe relevant search features or topics, these nodes 146 corresponding to each lexical unit 122 of query 108 and other lexical units 122 (e.g., words or phrases) that do not appear in query 108 but are available in NLP model 114 (e.g., phrases or descriptions using similar features or concepts from training dataset 130).

[0034] Figure 2 A block diagram of an example computing system 200 (also referred to as computer system 200) is shown, which can be used to implement elements of the systems and methods described and illustrated herein. Computing system 200 may include or be used to implement any computing or functionality described herein, such as using commands, instructions, or data described herein. Computing system 200 may be included and run in any device (e.g., a server, client device 106, or any other device communicating via network 104, such as a company performing a data processing system). Computing system 200 can be used to operate, execute, or provide model trainer 112, NLP model 114, QCM 140, lexicalization function 120, repository 132 (e.g., storage device 225), and implement any functionality of these components described herein.

[0035] The computing system 200 may include at least one data bus 205 or other communication components for transmitting information. The computing system 200 may include at least one processor 210 or processing circuitry coupled to the data bus 205 for executing instructions or processing data or information. The computing system 200 may include one or more processors 210 or processing circuitry coupled to the data bus 205 for exchanging or processing data or information with other computing systems 200. The data processing system 120 may include one or more main memories 215, such as random access memory (RAM), dynamic RAM (DRAM), or other dynamic storage devices, coupled to the data bus 205 for storing information and instructions to be executed by the processor 210. The main memory 215 may be used to store information (e.g., data, computer code, commands, or instructions) during the execution of instructions by the processor 210.

[0036] The computing system 200 may include one or more read-only memory (ROM) 220 or other static storage devices 225 coupled to the data bus 205 for storing static information and instructions for the processor 210. The storage device 225 may include any storage device, such as a solid-state device, disk, or optical disk, which may be coupled to the data bus 205 for persistent storage of information and instructions.

[0037] The computing system 200 may be coupled to one or more output devices 235, such as speakers or displays (e.g., liquid crystal displays or active matrix displays), via a data bus 205 for displaying or providing information to a user. Input devices 230 (e.g., keyboards, touchscreens, or voice interfaces) may be coupled to the data bus 205 for transmitting information and commands to the processor 210. Input devices 230 may include, for example, touchscreen displays (e.g., output devices 235). Input devices 230 may include cursor controllers (e.g., mice, trackballs, or cursor arrow keys) for transmitting directional information and command selection to the processor 210 for controlling cursor movement on the display.

[0038] Computer system 200 may also include one or more interfaces 240 coupled via data bus 205. Interface 240 may include any physical or virtual component that enables communication between computer system 200 and any external network (e.g., the Internet of Things, network 104). Interface 245 may include a network interface that provides data transfer between processor 210, memory 215, and any external network (e.g., network 104) to communicate with other devices on network 104 (e.g., client device 106, third-party entities, servers, or the cloud providing data processing system 102).

[0039] The processes, systems, and methods described herein can be implemented by computing system 200 in response to processor 210 executing an instruction set contained in main memory 215. Such instructions may be read into main memory 215 from another computer-readable medium (e.g., storage device 225). Execution of the instruction set contained in main memory 215 causes computing system 200 to perform the illustrative processes described herein. Alternatively, one or more processors 210 in a multiprocessor configuration may be used to execute the instructions contained in main memory 215. Hardwired circuitry may be used in place of or in combination with software instructions with the systems and methods described herein. The systems and methods described herein are not limited to any particular combination of hardware circuitry and software.

[0040] Although already Figure 2 An example computing system is described herein, but the subject matter including the operations described herein can be implemented in other types of digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed herein and their equivalents), or in one or more combinations thereof.

[0041] Figure 3An example block diagram of the structure 300 of a query context model 140 utilizing a graph transformer 144 and a graph neural network 142 is shown. Structure 300 may include an architecture of multi-level or multi-level (e.g., layer-by-layer) GNN 142 components nested side-by-side and across transformer 144 (e.g., transformer blocks of a language model). By utilizing the example structure 300 of the QCM 140, more accurate contextual cues can be provided for pre-trained domain-specific NLP 114. Encoder 124 can be used to provide output (e.g., contextual data structure 150) that can drive decoder 126. The determination results of each layer (e.g., from transformer 144) can be validated using contextual data structure 150 and its validation parameters or contextual parameters.

[0042] Unlike extracting a random text segment from query 108 as a hint for NLP model 114 (e.g., without further analysis of the text's meaning), the example structure 300 of QCM 140 allows DPS 102 to analyze the text of query 108 to understand the semantic roles and associations between the query's words 110 (e.g., words and phrases). For example, QCM 140 can generate or determine validation parameters or contextual parameters for each node 146 or the relationships between nodes (e.g., node features 148). Using example structure 300, QCM 140 can identify enhanced or improved hints for NLP model 114 and more accurately control the response 116 from NLP model 114. For example, QCM 140 can utilize contextual data structure 150 to keep the nodes 146 of the hints for NLP model 114 consistent with validation parameters and contextual parameters.

[0043] Example structure 300 uses a graph transformer with GNN components nested in layers alongside and across the language model's transformer 144. Example structure 300 allows text encoding and graph aggregation to be fused or combined into an iterative workflow. This can improve the accuracy of understanding each node 146 from a global perspective (e.g., keeping it within the context of the query). This scheme provides a progressive learning strategy in which the model can be continuously trained based on manipulated and raw data to enhance its ability to integrate information on the graph.

[0044] The example structure 300 of QCM 140 uses a GNN, such as a pre-trained language model, nested alongside and across each layer of the model's transformer 144. Therefore, the latent semantics of each text node 146 can be accurately captured and efficiently integrated for high-quality text graph representations. This architecture can introduce a two-stage progressive training strategy to improve the representation quality of the graph transformers (e.g., structure 300). This scheme simplifies the model through unidirectional graph aggregation, as this reduces some unnecessary computational costs, thereby improving efficiency and scalability.

[0045] In example structure 300, the GNN 142 component can be nested side-by-side and across transformers 144 arranged in layers or rows. Thus, text encoding and graph aggregation can be fused or combined into an iterative workflow. Each iteration of the workflow can include or utilize nodes 146 that can be linked to each other, allowing nodes 146 to exchange information with other nodes 146 in the layer-by-layer GNN 142 component (e.g., within a single layer of GNN 142 and transformer 144). Therefore, each node 146 can be augmented through other nodes 146 surrounding it in structure 300 (e.g., its neighbors).

[0046] Each transformer 144 component can process the amplified node features 148, which can generate increasingly richer representations of nodes 146 and the relationships between nodes 146 (e.g., contextual similarity determined from the similarity function 118) for the next iteration. Therefore, the QCM 140 of example structure 300 can continuously improve the contextual representation of nodes 146 corresponding to the elements of the input query, thereby generating an output in the form of a contextual data structure 150, which can include or reference node features 140 and can be used as input to the NLP model 114. Thus, the graph transformer in example structure 300 enables more efficient utilization of cross-node information on the graph, thereby improving the quality of data representation while reducing computational or operational costs.

[0047] QCM 140 can be trained using various techniques. For example, training QCM 140 (e.g., a graph transformer) can involve a central node 146 that is "sufficiently informative" to provide the desired response. In this case, the training task can be completed without relying on information from neighboring nodes 146. To ensure that GNN 142 is adequately trained, QCM 140 can be trained incrementally. In the first round of training, manipulated data can be used, in which nodes 146 may be randomly contaminated with erroneous data. This makes it more difficult to make predictions relying solely on the central node 146 (e.g., without utilizing neighboring nodes 146). Then, training the QCM 140 model can force the QCM model 140 to utilize all input nodes 146 (e.g., the neighborhood of node 146, including, for example, nodes 146 in one or more layers).

[0048] The second round of training can use uncontaminated data, where QCM 140 can be fitted to the target distribution. QCM 140 can be arranged to include a one-way graph to avoid all linked nodes 146 depending on each other during representation, thus preventing the representation of a new node 146 from triggering zero-based encoding of all neighboring nodes 146. In the one-way graph method, only the central node 146 can reference its neighboring nodes 146, while neighboring nodes 146 remain independently encoded. This allows for caching and reuse of existing neighboring node encodings, significantly saving computational costs.

[0049] Figure 4 An example of system 400 is shown, which is used for multilingual and multi-dialect text processing via a transformer-based Large Language Model (LLM) using a graph neural network (GNN) model. System 400 may include one or more corpus texts 405, a lexical graph 430, a graph feature extractor 450, a text representation 460, a weight assigner 470, and a classifier 490. The corpus text 405 may include any training dataset 130, which includes one or more labeled data 420 or unlabeled data 425 for one or more languages ​​410 or regions 415. The lexical graph 430 may include levels 435, associations 440, and an unlabeled data association unit 445. The graph feature extractor 450 may include one or more GNN models 142 and vectors 455. The text representation 460 may include one or more NLP models 425, vectors 455, and example data 465. The weight assigner 470 may include one or more weights 475 and a reweighter 480.

[0050] Corpus text 405 may include any collection of written or spoken language samples or texts that can be collected and used as a comprehensive text data source for studying language patterns, regions, and communications in the context of an industry or field (e.g., e-commerce, technology sales, medical applications, space physics, software engineering, architectural design, or any other field). Corpus text may include archives of books, historical letters, diaries, medical records, legislative texts, regulations, newspaper articles, social media posts or comments, transcripts of recorded interviews, dialogues, or any other communication.

[0051] The corpus text 405 can be expressed or provided in any language 410. Language 410 can include any systematic or structured communication system characterized by words or symbols, with agreed-upon meanings and rules. Language 410 can be any natural language (e.g., spoken language), such as English, Spanish, Mandarin, Japanese, Arabic, Turkish, Hindi, Russian, or Urdu. Language 410 can be any artificial language, such as a programming language (e.g., Python, Ruby, Java). Language 410 can include any simplified language that arises among groups of people who do not use a common language, such as Pichin or Creole.

[0052] The corpus text 405 may include any labeled data 420 and unlabeled data 425. Labeled data 420 may include any annotated or tagged textual information (e.g., using categories, attributes, or metadata) to provide context or classification for different parts of the text. The annotations on the labeled data 420 can be used to convey the expected meaning, sentiment, or category of a portion of the text (e.g., a word or sentence), making it suitable for training machine learning models for various tasks, such as text classification.

[0053] Unlabeled data 425 can include any data that contains text content but is not labeled or identified, or has no indication of any category, attribute, or metadata. Unlabeled data 425 can include raw text content without any annotations or categories. Additional analysis and processing can be performed on unlabeled data 425 (e.g., via an unlabeled data correlator 445) to extract meaningful insights, patterns, or associations 430 with other words or text that may have been labeled.

[0054] Lexical graph 430 may include any structured representation of the text data in which individual words 110 or lexical units 122 may be organized into an organized structure or format, such as a network, table, grid, or class graph structure. Each lexical unit 122 within the corpus text 405 may be considered a node (e.g., 146) in lexical graph 430. Relationships between different lexical units 122 in lexical graph 430 may be represented as edges. Lexical graph 430 may provide or indicate associations 430 between lexical units 122 (e.g., words 110) based on their respective positions within the lexical space of lexical graph 430. For example, lexical units 122 that are closer to each other or more closely related may be more similar than those that are more distant or far apart.

[0055] Lexical graph 430 may include multiple levels 435 (e.g., a multi-level lexical graph 430), where each level 435 may correspond to a different language 410 or region 415. Lexical graph 430 may have the ability to capture linguistic features and relationships across different dimensions or granularities (including across different levels 435). For example, lexical graph 430 may include lexical units 122 that may exist in multiple levels 435 (or layers) and be related to each other, such that a lexical unit 122 in a first level 435 (e.g., a first language 410) may be associated with another lexical unit 122 (e.g., in another language 410). Therefore, the lexical graph 430 can discover close connections 430 between lexical units 430 across different languages, thereby the meaning of a particular word 110 indicated by a lexical unit 122 in the first language 410 or the first region 415 (e.g., at the first level 435) can be supplemented, clarified, or explained by another word 110 indicated by another lexical unit 122 in the second language 410 or the second region 415 (e.g., at the second level 435).

[0056] Lexical graph 430, sometimes also called word graph 430, may include different levels 435 corresponding to different levels of linguistic abstraction. For example, one level 435 may represent a single word 110, another level 435 may correspond to a phrase or sentence, a third level 435 may correspond to sub-word units, and a fourth level 435 may correspond to syntactic or semantic relationships between lexical units 122. Lexical graph 430 may allow lexical units 122 across levels or 435 to provide supplementary, clarifying, or more detailed meanings or analyses of the text, thereby contributing to context-aware language modeling, semantic role labeling, and multi-level text understanding.

[0057] The unlabeled data correlator 445 may include any combination of hardware and software for creating associations 430 about parts of word 110 or unlabeled data 420. The unlabeled data correlator 445 may include functionality for determining associations 430 between unlabeled data 420 and words in the labeled data 420, or between two or more unlabeled data words, using the labeled data 420. For example, a multi-level word graph 430 may be used to map unlabeled data 425 to lexical units 120. For example, a graph feature extractor 450 may represent raw, unannotated text as a structured network of lexical units 120 (e.g., using different types or levels of granularity to capture linguistic features and relationships). Lexical units 120 may be generated from unlabeled data and connected within the word graph 430 based on various associations, including co-occurrence patterns and other linguistic associations. These relationships 430 may be quantized and used to establish connections between lexical units 120 (including lexical units of other labeled data 420 or unlabeled data 425). Multilevel word graph 430 can be used to establish or represent the relationships between various labeled and unlabeled data across different levels 435, thereby allowing texts in different languages ​​410 to supplement, clarify, indicate, or define the meaning of words 110 (including contextual nuances) within the text of unlabeled data 420.

[0058] Graph feature extractor 450 may include any combination of hardware and software that transforms graph structure data (e.g., lexical units 122, nodes 146, and edges) into meaningful numerical features (e.g., vector 455 or embedding vectors). Graph feature extractor 450 can identify and quantify the complex relationships and topological information inherent in the graph and transform them into vector representations (e.g., vector 455) that can be used or processed by ML algorithms. Graph feature extractor 450 may leverage graph convolutional networks (GCNs) to help the feature extractor capture and encode information from neighboring nodes 146 or edges (e.g., node features 148) in the graph (e.g., lexical graph 430). Graph-based features can be used in downstream ML tasks, enabling models to understand and make predictions based on the underlying graph structure.

[0059] Vector 455 can include any mathematical representation of a quantity. Vector 455 can include magnitude and direction in a multidimensional space (e.g., an embedding space or a lexical space). Vector 455 can be represented as an ordered list of numbers or parts, such as x, y, and z, and can be represented in a coordinate system (e.g., a Cartesian coordinate system). Vector 455 can be used to describe associations 430 between two or more values ​​or representations (e.g., two or more lexical units 122, nodes 146, embedding vectors). For example, vector 455 can be used in a lexical graph 430 or an embedding space to represent a single lexical unit 122, node 146, or embedding vector. Vector 455 can facilitate the encoding of semantic, syntactic, or contextual information associated with each value or representation (e.g., lexical unit 122 or node 146) within the space (e.g., lexical graph 430).

[0060] The text representation 460 (or TR 460) may include any combination of hardware and software for converting textual information into numerical representations (e.g., values, vectors, matrix elements, etc.). The text representation 460 may encode each word 110 in the text into a vectorized form (e.g., lexical unit 122) and capture any contextual information and meaning associated with that word 110. This contextual word embedding vector can help to effectively understand and process multilingual text while preserving context-specific nuances and details. The TR 460 may include functionality that takes multilingual examples as input and generates example-level and word-level representations as output. The TR 460 may capture example-level context to facilitate efficient category separation via classifier 490.

[0061] Example data 465 may include any text or data, including specific dialogues, communications, transcripts, or interactions between individuals, such as texts of conversations within different fields (e.g., science, trade, or technology), languages ​​410, or regions 415. Example data 465 may include transcripts or text that can be used to provide context for individual words 110 (e.g., lexical units 120 corresponding to word 110) or phrases within the text. For example, example data 465 may include examples of conversations in specific industries (e.g., e-commerce, healthcare, finance, academia, engineering, or banking), as well as examples of conversations in various languages ​​410 or specific geographic regions 415. Example data 465 may include examples of how specific words 110 are used, emphasized, or weighted (e.g., regarding weight 475) in a specific communication or conversation (e.g., corresponding to specific lexical units 120 or vector 455).

[0062] The weight allocator 470 (sometimes also called an importance estimator) may include any combination of hardware and software for determining the relevance of word priors (e.g., prior definitions or understandings of words in text) based on contextual representations in a specific user example (e.g., an e-commerce sales dialogue sample). The weight allocator 470 may assign weights 475 to each of the vectors 455 provided by the graph feature extractor 450 (e.g., a corpus-trained GNN) or the text representation 460 (e.g., an example-trained LLM). The weight allocator 470 may use a scaled dot product attention scheme to determine attention scores based on the similarity between the vectors 455 (e.g., embedding vectors) of the example representations from the text representation 460 and the node embedding vectors (e.g., lexical units 120) from the graph feature extractor 450.

[0063] The weight allocator 470 may include a reweighter 480 that combines the attention-weighted node vector 455 from the graph feature extractor 450 with the word vector 455 from the text representation 460 to create a composite representation (reweighted representation) that reflects the importance of a given word or phrase in the context of a given example. The weight allocator 470 may use the reweighter 480 to determine and assign higher weights to the words 110 or lemmas 122 most relevant to the user's query or context, thereby contributing to improved understanding and response generation.

[0064] Classifier 490 may include any ML model trained to classify or categorize input data into predefined categories or classifications. Classifier 490 may include sentiment labels (e.g., positive, neutral, or negative labels), topic labels, or any other labels or classifications. Classifier 490 may process weighted vectors 455 (e.g., lexical units 120 or words 110) from weight assigner 470 or reweighter 480, and text representations (e.g., vector 455) from text representation unit 460. The received vector 455 may include or convey information about the importance of each word 110 or lexical unit 122, while the text representation may capture the word context. Classifier 490 may combine these two sets of information (e.g., according to any assigned weights 475) to form a representation that reflects the contextual importance of the words in the user's query or context. Classifier 490 may help provide a response 116 to user query 108.

[0065] In some respects, system 400 can use a transformer-based language model and a GNN-based multilingual text classification framework, which utilizes a combination of labeled data 420 and unlabeled data 425 to process monolingual, cross-lingual, and multilingual text. System 400 can leverage a lexical graph 430 (e.g., a word graph) constructed from an available text corpus 405 and contribute to generative AI solutions that can be used to deliver online sales campaigns (e.g., real-time communication with customers) or customer support.

[0066] System 400 may include an LLM using a GNN NLP machine, which can leverage a learned model to process e-commerce data of various types and sources to encode monolingual, cross-lingual, and multilingual content. System 400 may include an end-to-end trainable framework that combines the generality of a GNN applied to corpus text 405 with the functionality of a transformer-based LLM applied to examples (e.g., e-commerce sales conversations with customers), where contextual associations 430 can be provided within the applied context, such as a sales application in a particular language 410 or region 415, and then provides an answer (e.g., query response 116) by means of cross-attention between the two. System 400 may include an LLM using text classification and NLP prior to any embeddings used for product discovery and search, which helps support multilingual sales and e-commerce data and generate content that understands the context of the input, regardless of the language used in the source data, to generate response 116 (e.g., via associations across multiple levels 435 of a lexical graph 430).

[0067] System 400 can use a corpus text dataset 405 and example data 465 providing contextual information (e.g., sales chat messages or interactions) to represent each word 110 in two distinct embedding spaces (e.g., an example embedding space and a corpus text-based embedding space). System 400 can use an attention mechanism (e.g., a weight allocator 470) to use, assign, or apply weights 475 and / or scores to determine the importance of two corresponding embedding vectors or vectors 455 relative to the task for which a response is being provided.

[0068] Corpus-level information (e.g., corpus text 405) can serve as word representations that can be aggregated in the contexts in which the corresponding words appear (e.g., sentences, phrases, or descriptions in various languages, regions, or areas). Example-level information (e.g., example data 465) can capture context-specific information (e.g., the usage of words in a specific domain such as e-commerce sales), allowing this information to be used to improve category separation (e.g., context-based classification). By combining analysis based on corpus text 405 and its vectors 455 (e.g., dictionary definitions of words) with example data 465 indicating the usage of words in examples of certain types of communication in a specific domain (e.g., e-commerce sales in British English) using a weight assigner 470, system 400 can produce improved generalization performance in low-resource text classification settings using limited labeled data 420 across various languages, dialects, or regions. This allows the technical approach to help save computational and processing resources and complete model training across various languages ​​and regions in a shorter timeframe. By doing so, the technology solution can contribute to performance improvements across a variety of context-specific applications, such as region-specific and domain-specific communications, including UK-based e-commerce sales communications or Argentina-based business customer service.

[0069] Given the use of a multilingual transformer model as a text representation (TR) 460, system 400 is capable of facilitating multilingual, multiregional, or multidialect schemes. Vectors 455, serving as word embedding vectors at the corpus text level 405, can be obtained using a graph feature extractor 450. The graph feature extractor 450 can be based on or utilize a GCN on a lexical graph 430, which can be constructed from available labeled and unlabeled data (e.g., 420 and 425). The lexical graph 430 can be multilingual and includes nodes (e.g., lexical units 120) from multiple languages ​​410 and multiple regions 415 (e.g., regional phrases or specific professional, academic, domain-specific, or industry-specific languages ​​or expressions). Edges in the lexical graph 430 can be defined using cross-lingual word embedding similarities from a pre-trained multilingual transformer model.

[0070] System 400 can use a weight allocator 470 (e.g., an importance estimator) to combine word embedding vectors (e.g., vector 455) based on corpus text 405 and example data 465. This weight allocator can utilize a scaled dot product attention scheme to combine corpus-level and example-level word embedding vectors. The weight allocator 470 can use a similarity function between example embedding vectors (e.g., vector 455 of TR 460) and individual GCN node embedding vectors (e.g., vector 455 of GF 450) to determine, establish, or compute attention scores or weights 475.

[0071] The text representation 460 may include a transformer-based model that leverages its self-attention structure to learn example-level contextual word embedding vectors for each word based on the similarity between each word and all other words in the example. The text representation 460 may contribute to representing multilingual text and learning an embedding space that can increase the separation between classifications to be used by the classifier 490.

[0072] For example, in monolingual applications, System 400 can use, for instance, a bidirectional encoder representation (BERT) scheme from a transformer, while in a multilingual setting, the scheme can use multilingual BERT (mBERT) to represent examples. Pooled token embedding vectors (i.e., [CLS] for BERT) can be considered as example embedding vectors. Text representations based on [CLS] token embedding vectors can be used for various classification tasks. System 400 can utilize schemes and architectures other than BERT and can be replaced with any transformer-based text representation model architecture, such as XLM-RoBERTa.

[0073] System 400 may include or provide a GNN-based graph feature extractor 450 to learn word embedding vectors (e.g., vector 455) based on corpus-level context. System 400 may define a lexical graph 430, the vertices of which may include expressions from examples, such as... In this example, the word (x) is included in both the labeled and unlabeled data. In some respects, there may be tasks without an unlabeled dataset, or there may be tasks with an unlabeled dataset.

[0074] Edges in the lexical graph 430 can be defined or identified based on the co-occurrence (e.g., within a window) of words 110 from examples from an underlying corpus (e.g., corpus text 405 or example data 465). Sometimes, capturing multilingualism can be challenging because words from different languages ​​rarely co-occur in a single example, which in turn can result in the lexical graph 430 having disconnected parts, leading to erroneous results (e.g., no association 430 between related words). System 400 can overcome this challenge by obtaining embedding similarity from an embedding layer of a transformer-based LLM, in which co-occurrence (within a window of examples) can be based on a similarity matrix. Furthermore, the similarity based on the embedding vector can be represented as a matrix. These two matrices, i.e., matrices (e.g., co-occurrence matrix) and matrix (e.g., embedding matrices) can be row-normalized and summed (e.g., This allows us to obtain a combined similarity metric (e.g., based on co-occurrence and embedding vectors).

[0075] In some embodiments, above the threshold ( The similarity value can be used to define edges in a graph. Threshold ( () can be a hyperparameter in the model. As a preprocessing step, very rare words (e.g., the lowest frequency of 3 in the corpus) and high-frequency stop words can be excluded and not considered as nodes in the lexical graph 430. For example, co-occurrence similarity can help expand the context of words within a language (e.g., expressions that are more common in the monolingual example data 465), while embedding similarity can capture relationships between words from multiple languages ​​(e.g., dictionary translations across various languages).

[0076] System 400 can initialize, use, or combine the initial embedding vectors of lexical graph 430 with word embedding layers from word representations of text representation 460 (e.g., vectors 455 based on example data 465). Combining vectors from lexical graph 430 with vectors from example data 465 offers various advantages. For example, since multilingual TR 460 can be pre-trained using a large corpus, this approach can incorporate prior information into the construction of lexical graph 430, thereby improving contextual understanding. Combining vectors from lexical graph 430 with vectors from example data 465 can help text representation 460 and graph feature extractor 450 have the same vocabulary. For example, this approach can be applied to lexical graph 430. Jump( -hop) GCN is used to obtain graph-based word (node) embedding vectors. GF ​​450 can extend the contextual information representation of nodes in the graph (i.e., frequently occurring words / high embedding similarity) in their direct neighborhoods by aggregating information across multiple hops. Jump to the neighboring region.

[0077] Figure 4 This is an example processing flow for Method 500, which leverages a GNN-based LLM model to provide multilingual and multi-dialect text processing. Method 500 can be used (e.g., in conjunction with...) Figures 1 to 4 The system and components described herein are used to implement this method, and example systems 100-400 are included. Method 500 may include steps 505-525. In step 505, the data processing system may receive a query. In step 510, the data processing system may generate a first vector using corpus text. In step 515, the data processing system may generate a second vector using example data. In step 520, the data processing system may establish a priority between the first and second vectors. In step 525, the data processing system may provide a response based on the priority.

[0078] In step 505, the data processing system may receive a query, such as a query from a client device. The method may include a data processing system comprising one or more processors coupled to memory to receive the query. The query may include a first word in a first language, first locale, first dialect, or regional terminology, or a phrase from a first region or specific to a first field (e.g., technical, industrial, medical, or legal). The first word may correspond to a first lexical element in a word graph indicating associations between words in multiple languages. Multiple languages ​​may include the first language. The first language may be any natural language (e.g., English, Arabic, Spanish, Mandarin, or Japanese) or a programming or coded language (e.g., Python, Java, C).

[0079] A query may include a description of a product or product features. A query may include a description of a product or service that a user of a client device is seeking to purchase. A query may specify a product's price range, delivery date, or specifications, or information about the product's use or purpose (e.g., a gift for a child), or the product's technical specifications. A query may include a selection of phrases or words specific to any field or subject (e.g., e-commerce, the medical field (e.g., related to specific medical or health-related diagnosis, drug, or treatment), the legal field (e.g., legal standards or regulatory terminology), or engineering or scientific terminology specific to a particular technical or scientific field), or the use of such language, dialect, regional terminology, phrases, or vocabulary. For example, a query may be directed to questions about one or more events, history, political developments, geography, markets, transportation, instructions, or any other field or subject.

[0080] A data processing system can receive queries from users or client devices (such as a user's computer). A query may include one or more words, such as sentences or paragraphs. The data processing system can generate multiple lexical units (lexical units) for the multiple words in the query. Each of the multiple lexical units in the query can correspond to a lexical unit in a word graph.

[0081] In step 510, the data processing system can generate a first vector using corpus text. This method may include: the data processing system generating a first vector for a first word in response to a first lexical unit input into a graph neural network (GNN) model trained using a text-based corpus of word graphs. For example, the data processing system may generate a first vector or embedding vector corresponding to a first lexical unit representing a word. The data processing system may generate multiple vectors or embedding vectors corresponding to multiple lexical units representing multiple words, which may be represented or provided in a lexical graph.

[0082] This method may include a lexicalization function to generate or identify lexical units. The lexicalization function may utilize one or more ML models (e.g., a query context model including any graph neural network or transformer thereof, or any natural language processing model) to generate one or more embedding vectors (e.g., vector representations) for one or more elements of a query or one or more lexical units of an element. The embedding vectors may include a vector representation of the meaning or context of each lexical unit in the lexical unit. The lexicalization function may utilize one or more ML models to generate an instance representing the context of the query, or to generate a vector representation of a sequence of lexical units corresponding to a sequence of elements of the query (e.g., context parameters of a contextual data structure). The lexicalization function may utilize one or more ML models to generate vector representations (e.g., embedding vectors) of individual lexical units or corresponding elements, thereby indicating the contextual representation or meaning of each element in the element (e.g., and any weights assigned to individual elements).

[0083] This method can utilize a lexical graph, which may include multiple levels or layers. Each level or layer can correspond to one of multiple languages. For any number of languages, each level or layer can correspond to one of multiple regions. Each level or layer can correspond to a language, terminology, dialect, regional terminology, or field of study (e.g., medicine, law, academia, sports, biotechnology, computer science, finance, or banking).

[0084] This method may include using lexical units at the first level of a multi-level lexical graph to indicate a first association between a first word and a third word in a first language. The method may also include using lexical units at the second level of the multi-level lexical graph to indicate a second association between a second word and a fourth word in a second language, different from the first language. For example, a lexical graph can provide or indicate associations between words or phrases from different languages, and can use words or phrases from different languages ​​to describe, define, designate, or provide meaning or context for specific words or phrases (e.g., lexical units).

[0085] This method may include and utilize a data structure (e.g., a contextual data structure) to identify a relationship between a first node and a second node among a plurality of nodes in the data structure, the first node corresponding to a first lexical unit at a first level and the second node corresponding to a second lexical unit at a second level. The relationship may be indicated, for example, by features of the first node having or including numerical representations indicating the relationship between the first and second nodes. For example, the relationship between the first and second nodes may be a contextual relationship. This contextual relationship may be defined, quantified, indicated, or determined based on the distance between the first numerical representation (e.g., a first embedding vector) of the first node and the second numerical representation (e.g., a second embedding vector) of the second node in a vector space (e.g., the embedding space of a knowledge graph). The relationship may be determined or defined based on one or more contextual parameters for a query or one or more validation parameters for a node. The contextual data structure may include a plurality of lexical units organized hierarchically, such that a graph transformer of an ML model updates each node based on the values ​​of neighboring nodes (e.g., nodes adjacent to the node being updated).

[0086] For example, a data processing system can identify the relationship between a first word element at the first level of a word graph and a second word element at the second level of the word graph, where the first word element corresponds to a first language and the second word element corresponds to a second language different from the first language. The data processing system can then use a GNN model and, in response to this relationship, generate a first vector for the first word.

[0087] For example, a data processing system can identify the relationship between a first word element at the first level of a word graph and a second word element at the second level of the word graph, where the first word element corresponds to a first language and the second word element corresponds to a second language different from the first language. The data processing system can then use a GNN model and, in response to this relationship, generate a first vector for the first word.

[0088] For example, a data processing system can identify a relationship between a first lexical unit and a second lexical unit in a word graph, where the second lexical unit corresponds to a word from unlabeled data and is defined based on the co-occurrence of a subset of lexical units in the word graph. The data processing system can then utilize a GNN model and, in response to this relationship, generate a first vector for the first word.

[0089] The data processing system can identify a first word by responding to the relationship between a first word element in the first level of the word graph and a second word element in the second level of the word graph, where the first word element corresponds to a first region of the first language and the second word element corresponds to a second region of the first language. The data processing system can utilize a GNN model and, in response to this relationship, generate a first vector for the first word.

[0090] In step 515, the data processing system can generate a second vector using the example data. This method may include: the data processing system generating a second vector for a second word in response to input into a first lexical unit in a large language model (LLM) trained on multiple interactions based on a transformer. For example, the data processing system can use example data, including transcripts or records of domain-specific, language-specific, region-specific, or industry-specific interactions between groups of people (e.g., users, customers and service providers, students and teachers, doctors and patients, or any other group). The text representation can leverage the transformer-based LLM model to generate contextual understanding, associations, or meanings from the example data interactions and specify them in the generated vector.

[0091] In step 520, the data processing system may establish a priority between the first vector and the second vector. This method may include: the data processing system establishing a priority for either a first word or a second word based at least on a first weight of the first vector and a second weight of the second vector. The data processing system may receive a first lexical unit corresponding to a first word in an e-commerce interaction within a first region of a first language.

[0092] The method may include: assigning a first weight to a first vector in response to a first word in a first region, first dialect, or phrase corresponding to a first domain, via a data processing system. The method may also include: assigning a second weight to a second vector in response to a second word in a second region, second dialect, or phrase corresponding to a second domain, wherein the second region, second dialect, or phrase differs from the first region, first dialect, or phrase. The data processing system may use the first word to provide a response to a query, at least based on the first weight exceeding the second weight.

[0093] In step 525, the data processing system may provide a response based on priority. This method may include providing a response to the query using at least one of a first word or a second word, based on established priorities. For example, the data processing system may select or choose one or more words to use based on the priority of vectors (e.g., lexical units or embedding vectors) corresponding to weighted words. The response may include a text response generated based on domain-specific language, dialect, regional terminology, dialect, or specific terms or phrases detected or identified based on the query. The response may include information about the requested product, information about the requested service, or information in response to the question or query provided in step 505. The response may include text that may be tailored to conform to, or expressed using, the language, dialect, regional terminology, dialect, or specific terms or phrases identified from or corresponding to the query.

[0094] The method may include: a data processing system constructing a first matrix of a GNN model, the first matrix including values ​​indicating co-occurrence patterns among one or more words in a corpus. The data processing system may include constructing a second matrix of a transformer-based LLM, the second matrix including values ​​indicating vector representations of words in multiple interactions. The method may include: the data processing system using the sum of the first and second matrices to establish prioritization.

[0095] The examples provided above are for illustrative purposes only and should not be construed as limiting this disclosure. While aspects of this disclosure have been described with reference to exemplary embodiments, it should be understood that the language used herein is descriptive and illustrative, not restrictive. Changes may be made within the scope of the appended (currently stated and amended) claims without departing from the scope and spirit of the various aspects of this disclosure. Although aspects of this disclosure have been described herein with reference to specific means, materials, and embodiments, this disclosure is not intended to be limited to the specific content disclosed herein; rather, it extends to all functionally equivalent structures, methods, and uses, such as those within the scope of the appended claims.

[0096] The aforementioned system can provide multiple components or each of these components, and these components can be provided on a standalone system or on multiple instances of a distributed system. Furthermore, the aforementioned system and method can be provided as one or more computer-readable programs or executable instructions embodied on or therein one or more artifacts. The artifact can be cloud storage, a hard disk, a CD-ROM, a flash memory card, a PROM, RAM, ROM, or magnetic tape. Typically, the computer-readable program can be implemented in any programming language (e.g., LISP, PERL, C, C++, C#, PROLOG) or any bytecode language (e.g., JAVA). The software program or executable instructions can be stored as object code on or therein of one or more artifacts.

[0097] The subject matter and operations described in this specification can be implemented in digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their equivalents), or in one or more combinations thereof. The subject matter described in this specification can be implemented as one or more computer programs, such as one or more computer program instruction circuits, encoded on one or more computer storage media for execution by or control of the operation of a data processing device. Alternatively or additionally, program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be or is included in a computer-readable storage device, a computer-readable storage matrix, a random or serial access memory array or device, or one or more combinations thereof. While the computer storage medium is not a propagating signal, it can be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium can also be or be included in one or more separate components or media (e.g., multiple CDs, disks, or other storage devices including cloud storage). The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or on data received from other sources.

[0098] The terms "computing device," "component," or "data processing apparatus," or similar terms, cover a wide range of means, devices, and methods for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination thereof. The apparatus may include dedicated logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or one or more combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0099] Computer programs (also known as programs, software, software applications, applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may correspond to a file in a file system. A computer program may be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple coordinated files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers located in one location or distributed across multiple locations and interconnected via a communication network.

[0100] The processes and logic described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform actions by manipulating input data and generating output. These processes and logic can also be executed by a dedicated logic circuit system, and the device can be implemented as a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). Suitable devices for storing computer program instructions and data include non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0101] The subject matter described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), front-end components (e.g., a client computer with a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described herein), or a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), interconnected networks (e.g., the Internet) and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0102] Although the operations are depicted in a specific order in the accompanying drawings, these operations do not need to be performed in the specific order shown or in a sequential order, and not all of the illustrated operations need to be performed. The actions described herein can be performed in different orders.

[0103] Having described some illustrative embodiments, it is clear that the foregoing is illustrative rather than limiting, and has been presented by way of example. In particular, although many of the examples presented herein relate to specific combinations of method steps or system elements, these steps and elements can be combined in other ways to achieve the same objective. The steps, elements, and features discussed in relation to one embodiment are not intended to exclude similar roles in other embodiments or implementations.

[0104] The wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising,” “including,” “having,” “containing,” “involving,” “characterized by,” “featured in,” and variations thereof throughout this document is intended to cover the items listed thereafter, their equivalents and additional items, as well as alternative embodiments consisting only of the items listed thereafter. In one embodiment, the systems and methods described herein include one, each combination of, or all of the described elements, steps, or components.

[0105] Any implementation, element, or step of a system or method mentioned in the singular herein may also encompass implementations that include a plurality of such elements, and any implementation, element, or step mentioned in the plural herein may also encompass implementations that include only a single element. The use of singular or plural references is not intended to limit the systems or methods, their components, steps, or elements of this disclosure to a singular or plural configuration. A reference to any step or element based on any information, step, or element may include an implementation of that step or element that is at least partially based on any information, step, or element.

[0106] Any implementation disclosed herein may be combined with any other implementation or embodiment. References to “implementation,” “some implementations,” “one implementation,” or similar terms are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with that implementation may be included in at least one implementation or embodiment. Such terms as used herein do not necessarily refer to the same implementation. Any implementation may be combined inclusively or exclusively with any other implementation in any manner consistent with the aspects and implementations disclosed herein.

[0107] A reference to “or” can be interpreted as inclusive, so any term described using “or” can refer to any one, multiple, or all of the terms described. A reference to at least one in a list of connected terms can be interpreted as an inclusive “or” to refer to any one, multiple, or all of the terms described. For example, a reference to “at least one of 'A' and 'B'” can include only 'A', only 'B', and both 'A' and 'B'. Such references used in conjunction with “include” or other open-ended terms can include additional items.

[0108] When a technical feature in the drawings, detailed embodiments, or any claim is followed by a reference numeral, these reference numerals are included to enhance the comprehensibility of the drawings, detailed embodiments, and claims. Therefore, the presence or absence of reference numerals does not limit the scope of any claim element.

[0109] Modifications can be made to the described elements and steps, such as substitutions, changes, and omissions in the design, operating conditions, and arrangement of the disclosed elements and operations, without departing from the scope of this disclosure.

[0110] Unless otherwise expressly stated, references to "approximately," "substantially," or other terms of degree include variations of + / - 10% from a given measurement, unit, or range. Coupled elements may be electrically, mechanically, or physically coupled to each other, directly or through intermediate elements. Therefore, the scope of the systems and methods described herein is indicated by the appended claims, not by the foregoing description, and modifications falling within the equivalent meaning and scope of the claims are included.

Claims

1. A system comprising: A data processing system, comprising at least one processor coupled to a memory, the data processing system being used for: Receive a query in a first language, the query corresponding to a first word element; In response to the first word element input to a graph neural network (GNN) model trained on a text-based corpus, a first vector of the first word is generated, the first word element corresponding to a word graph indicating the associations between words in multiple languages, including the first language; In response to the first lexical unit input into a transformer-based, multi-interaction-trained large language model (LLM), a second vector of the second word is generated; Priority is established for either the first word or the second word, based at least on the first weight of the first vector and the second weight of the second vector. as well as Based on the established priority, at least one of the first word or the second word is used to provide a response to the query.

2. The system according to claim 1, comprising the data processor, the data processor being used for: Receive the query from the user equipment, the query including multiple words of the query; and Generate multiple word elements for the multiple words in the query, where each word element in the multiple word elements corresponds to a word element in the word graph.

3. The system according to claim 1, wherein, The word map includes multiple levels, including a first level and a second level. The first level includes word units of the first level, which indicate a first association between the first word and the third word in the first language. The second level includes word units of the second level, which indicate a second association between the second word and the fourth word in the second language, which is different from the first language.

4. The system according to claim 1, comprising the data processor, the data processor being used for: Identify the relationship between a first lexical unit at the first level of the word map corresponding to the first language and a second lexical unit at the second level of the word map corresponding to the second language, wherein the second language is different from the first language; and The first vector of the first word is generated by the GNN model in response to the relation.

5. The system according to claim 1, comprising the data processor, the data processor being used for: Identify the relationship between a first lexical unit and a second lexical unit in the word graph, wherein the second lexical unit corresponds to a word from unlabeled data, and the second lexical unit is defined based on the co-occurrence of a subset of lexical units in the word graph; and The first vector of the first word is generated by the GNN model in response to the relation.

6. The system according to claim 1, comprising the data processor, the data processor being configured to: The first word is identified in response to the relationship between a first lexical unit in a first level of the word map and a second lexical unit in a second level of the word map, wherein the first lexical unit corresponds to a first region of the first language and the second lexical unit corresponds to a second region of the first language; and The first vector of the first word is generated by the GNN model in response to the relation.

7. The system according to claim 1, comprising the data processor, the data processor being configured to: Receive the first word element, which corresponds to the first word in e-commerce interaction in the first region of the first language; In response to the first word in the first region, the first weight is assigned to the first vector, and in response to the second word in the second region, the second weight is assigned to the second vector, the second region being different from the first region; as well as The first term is used to provide the response to the query, at least based on the first weight exceeding the second weight.

8. The system according to claim 1, comprising the data processor, the data processor being configured to: Construct a first matrix for the GNN model, the first matrix including values ​​indicating co-occurrence patterns among one or more words in the corpus; Construct a second matrix for the transformer-based LLM, the second matrix including values ​​indicating vector representations of words in the plurality of interactions; and The priority is established using the sum of the first matrix and the second matrix.

9. A method comprising: A query is received through a data processing system, the data processing system including one or more processors coupled to a memory, the query including a first word in a first language, the first word corresponding to a first word element in a word graph, the word graph indicating the association between words in multiple languages ​​including the first language; The data processing system generates a first vector of the first word in response to the first word element input into the graph neural network (GNN) model, the graph neural network (GNN) model being trained using the word graph based on a text corpus; The data processing system generates a second vector of a second word in response to a first word input into a transformer-based large language model (LLM), the transformer-based large language model (LLM) being trained based on multiple interactions; The data processing system establishes a priority for either the first word or the second word based at least on the first weight of the first vector and the second weight of the second vector. as well as The data processing system provides a response to the query using at least one of the first word or the second word, based on an established priority.

10. The method of claim 9, comprising: The data processing system receives queries from user equipment, the queries including multiple words. as well as The data processing system generates multiple word elements for the multiple words in the query, and each word element in the multiple word elements corresponds to a word element in the word graph.

11. The method of claim 9, comprising: The word graph uses first-level lexical units across multiple levels to indicate a first association between the first word and the third word in the first language. as well as The second-level lexical units of the multiple levels indicate a second association between the second word and the fourth word in the second language, which is different from the first language.

12. The method of claim 9, comprising: The data processing system identifies the relationship between the first lexical unit at the first level of the word graph corresponding to the first language and the second lexical unit at the second level of the word graph corresponding to the second language, wherein the second language is different from the first language; as well as The first vector of the first word is generated using the GNN model and in response to the relation.

13. The method of claim 9, comprising: The data processing system identifies the relationship between the first lexical unit and the second lexical unit of the word graph, where the second lexical unit corresponds to a word from unlabeled data and is defined based on the co-occurrence of a subset of lexical units in the word graph. as well as The first vector of the first word is generated using the GNN model and in response to the relation.

14. The method of claim 9, comprising: The data processing system identifies a first word by responding to the relationship between a first word element in the first level of the word graph and a second word element in the second level of the word graph, wherein the first word element corresponds to a first region of the first language and the second word element corresponds to a second region of the first language. as well as The first vector of the first word is generated using the GNN model and in response to the relation.

15. The method of claim 9, comprising: The data processing system receives the first word element, which corresponds to the first word in e-commerce interaction in the first region of the first language. The data processing system, in response to the first word in the first region, assigns the first weight to the first vector, and in response to the second word in the second region, assigns the second weight to the second vector, the second region being different from the first region; as well as The data processing system uses the first term to provide the response to the query, based at least on the first weight exceeding the second weight.

16. The method of claim 9, comprising: The data processing system constructs a first matrix of the GNN model, the first matrix including values ​​indicating co-occurrence patterns among one or more words in the corpus; The data processing system constructs a second matrix of the transformer-based LLM, the second matrix including values ​​indicating vector representations of words in the plurality of interactions; as well as The priority is established by the data processing system using the sum of the first matrix and the second matrix.

17. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: Receive a query, the query including a first word in a first language, the first word corresponding to a first word element in a word graph, the word graph indicating the association between words in multiple languages ​​including the first language; In response to the first word element input into the graph neural network (GNN) model, a first vector of the first word is generated, wherein the graph neural network (GNN) model is trained using the word graph based on a text corpus; In response to the first word element input into a transformer-based large language model (LLM), a second vector of the second word is generated, the transformer-based large language model (LLM) being trained based on multiple interactions; Priority is established for either the first word or the second word, based at least on the first weight of the first vector and the second weight of the second vector. as well as Based on the established priority, at least one of the first word or the second word is used to provide a response to the query.

18. The non-transitory computer-readable medium according to claim 17, wherein, When the instructions are executed by one or more processors, the one or more processors cause the processors to: The query is received from the user equipment, the query comprising multiple words. as well as Generate multiple word elements for the multiple words in the query, where each word element in the multiple word elements corresponds to a word element in the word graph.

19. The non-transitory computer-readable medium according to claim 17, wherein, The word map includes multiple levels, including a first level and a second level. The first level includes word units of the first level, which indicate a first association between the first word and the third word in the first language. The second level includes word units of the second level, which indicate a second association between the second word and the fourth word in the second language, which is different from the first language.

20. The non-transitory computer-readable medium according to claim 17, wherein, When the instructions are executed by one or more processors, the one or more processors cause the processors to: Identify the relationship between the first lexical unit at the first level of the word graph corresponding to the first language and the second lexical unit at the second level of the word graph corresponding to the second language, which is different from the first language; as well as The first vector of the first word is generated by the GNN model in response to the relation.

Images