Computer implementation methods, computer programs, and computer systems (mixed expert networks for determining text data sources)

Abstract

As the use of LLM becomes more widespread, certain deviant behaviors are becoming apparent. [Solution] A computer implementation method, computer program product, and computer system for training a transformer machine learning model using each cluster n of N clusters as training data to generate a trained transformer n, which is a machine learning model with expert n (n=1, ..., N). Each of the N experts is associated with one of the N clusters, where N is at least 2. The N trained transformers together form a MoE network. The N clusters are derived from the input text.

Description

Technical Field

[0001] The present invention relates to machine learning for generative models, training of such generative models, and machine learning for tracing the source of data used to train a generative model.

Summary of the Invention

Problems to be Solved by the Invention

[0002] As the use of LLMs becomes more widespread, certain deviant behaviors are emerging.

Means for Solving the Problems

[0003] Embodiments of the present invention provide a computer-implemented method, a computer program product, and a computer system. A Transformer machine learning model is trained using each of N clusters as training data to generate a trained Transformer n that is a machine learning model of an expert n (n = 1,..., N), where the N experts are each associated with one of the N clusters, where N is at least 2, and where the N trained Transformers together form an N-expert Mixture of Experts (MoE) network, and where the N clusters are derived from input text.

Brief Description of the Drawings

[0004] [Figure 1] A flowchart illustrating a method for generating and using a Mixture of Experts (MoE) network according to an embodiment of the present invention.

[0005] [Figure 2A] A flowchart illustrating the generation of N clusters from input text according to an embodiment of the present invention.

[0006] [Figure 2B]Figure 2A shows a step according to an embodiment of the present invention, in which a consanguineous dense vector, a prosodic dense vector, and a semantic dense vector are collectively combined into a composite dense vector, and then the composite dense vector is clustered into N clusters denoted as C1, C2, ..., CN.

[0007] [Figure 3A] This invention provides an example demonstrating how to segment input text into paragraph text units according to an embodiment of the present invention.

[0008] [Figure 3B] This invention provides an example demonstrating how to segment input text into sentence text units according to an embodiment of the present invention.

[0009] [Figure 4A] This is a flowchart illustrating the process for generating a mixed expert (MoE) network from N clusters, each containing N experts associated with the N clusters, according to an embodiment of the present invention.

[0010] [Figure 4B] This describes a step in training a Large Language Model (LLM) network and a MoE network together, according to an embodiment of the present invention, using input text as the source of training data for both the LLM network and the MoE network.

[0011] [Figure 5] The present invention describes a process for training a transformer model using clusters as training data to generate experts associated with the clusters, according to an embodiment of the present invention.

[0012] [Figure 6]A flowchart explaining how to determine with high probability the data source of a given text unit using a trained MoE network according to an embodiment of the present invention.

[0013] [Figure 7] A computer system according to an embodiment of the present invention is shown.

[0014] [Figure 8] A computing environment including an example of an environment for at least some executions of computer code involved in implementing the method of the present invention according to an embodiment of the present invention is shown.

Mode for Carrying Out the Invention

[0015] According to a first aspect of the present invention, each cluster n among N clusters is used as training data to train a transformer machine learning model to generate respective trained transformers n that are machine learning models of experts n (n = 1,..., N), where the N experts are respectively associated with the N clusters, where N is at least 2, and where the N trained transformers together form a mixture-of-experts (MoE) network, and where the N clusters are a computer-implemented method derived from input text.

[0016] The foregoing first aspect of the present invention advantageously trains a transformer model to generate a mixture-of-experts (MoE) network including N experts specific to each of the N clusters, which enables each expert to specialize in the specific characteristics of each cluster.

[0017] According to a second aspect of the present invention, the N clusters are generated through: (i) segmenting the input text into text units; (ii) extracting semantic features from each text unit and encoding the semantic features into semantic dense vectors for each text unit by using an encoder part of a large language model to generate semantic dense vectors; (iii) extracting consanguineous features from each text unit and encoding the consanguineous features into consanguineous dense vectors for each text unit by using the encoder part of the large language model to generate consanguineous dense vectors; (iv) extracting prosodic features from each text unit, generating one-hot vectors for each prosodic feature of each text unit, and generating prosodic dense vectors for each text unit by using the one-hot vectors through a fully connected network to generate prosodic dense vectors; (v) calculating a composite dense vector for each text unit as a weighted sum of the semantic dense vector, the consanguineous dense vector, and the prosodic dense vector; and (vi) clustering the composite dense vectors of all text units into the N clusters and associating the N clusters with the number of respective data sources, where the number of each unique data source is equal to or less than N.

[0018] The foregoing second aspect of the present invention advantageously generates clusters that: (i) include dense vectors associated with semantic, consanguineous, and prosodic features; and (ii) are linked to respective data sources.

[0019] According to a third aspect of the present invention, the MoE network is an external adapter for a large-scale language model (LLM) network, and the method comprises the steps of: (i) training the generative language machine learning model using the input text as training data, and (ii) performing the step of training the transformer model using each cluster n as training data (n=1, ..., N).

[0020] A prior third aspect of the present invention advantageously includes a step of jointly training an LLM network and N transformer models using input text as initial input, thereby improving the ability of the LLM network and MoE network to process the input text collectively. According to a prior fourth aspect of the present invention, the joint training step includes a step of determining a first loss of the LLM network and the MoE loss of the MoE network; a step of calculating a total loss equal to the first loss + α × MoE loss, where α is a defined constant in the range of 0.005 to 0.10; and a step of incorporating the total loss into both training the generative language machine learning model and training the transformer models.

[0021] A fourth prior aspect of the present invention advantageously achieves the following: (i) in the case of an LLM network, the total loss influences the optimization of the LLM network by including the MoE loss, which helps the LLM network remain sensitive to cluster-specific features by slightly fitting the parameters of the LLM network to the discriminative characteristics of the clusters; and (ii) in the case of an MoE network, the MoE loss directly influences each expert in the MoE network based on cluster assignment, and by including the LLM loss in the calculation of the total loss, training the MoE network encourages the MoE network to maintain consistency aligned with broader language modeling objectives while refining the cluster-specific focus of the MoE network.

[0022] According to a fifth aspect of the present invention, the method provides the N experts to one or more processors by: (i) receiving a given text unit; (ii) generating a composite dense vector of features present in the given text unit from the given text unit; (iii) calculating the distance between the composite dense vector and the centroid of each of the N clusters, thereby generating N distances associated with each of the N experts; (iv) performing a logistic operation on each of the N distances, thereby generating N logistic distances associated with each of the N experts, indicated as Expert 1, ..., Expert N; and (v) calculating the smallest logistic distance D among the N logistic distances from the N experts. logistic_min The process further comprises a step of selecting an expert n1 having, where n1 is 1, ..., or N.

[0023] A fifth prior aspect of the present invention advantageously involves expert selection having the highest probability associated with a composite dense vector derived from a given text unit.

[0024] According to a fifth aspect of the present invention, the method is to use the one or more processors to perform D logistic_min The defined logistic distance threshold distance D logistic_th The further step is to determine that it is smaller than and, accordingly, to identify the data source n1 associated with the cluster n1 as the most likely data source for the given text unit by the one or more processors.

[0025] A fifth prior aspect of the present invention advantageously identifies the most likely data sources for a given test unit.

[0026] According to a seventh aspect of the present invention, the text data collectively comprises a plurality of text units, and the training data within each cluster n (n=1, ..., N) comprises a plurality of dense vectors associated with each of the plurality of text units, each dense vector being based on the presence or absence of a feature within the plurality of text units to which the plurality of dense vectors are associated, and the feature comprises a plurality of prosodic features.

[0027] A seventh prior aspect of the present invention is advantageous in that the training data for each cluster includes the most invariant elements of language, namely prosodic features.

[0028] According to an eighth aspect of the present invention, the feature further includes one or more closely related features, one or more semantic features, or a combination thereof.

[0029] An earlier eighth aspect of the present invention advantageously includes related and / or semantic features in addition to prosodic features, which improves the quality of features in the training data for each cluster.

[0030] Generative language machine learning models encompass a range including large-scale language models (LLMs). Therefore, the discussion herein regarding the use of LLMs in embodiments of the present invention is generally applicable to generative language machine learning models.

[0031] Currently, large-scale language models (LLMs) are widely applied across various industries. However, as the use of LLMs becomes more widespread, certain deviations have emerged, accompanied by controversial practices employed by many organizations and individual developers. These practices involve using third-party LLM models to "distill" or extract useful directive data, and then applying that directive data to train one's own LLM model. This controversial practice makes it easy to obtain high-quality directive data and raises a number of serious intellectual property (IP) infringement concerns.

[0032] In a sense, this controversial method is akin to illegally obtaining unauthorized information and could result in potential economic losses and unfair competition for the creators of the original large-scale language models. This situation raises numerous legal and ethical issues and necessitates measures to protect the legality of intellectual property rights and the principles of fair competition. Another concern is that, due to a lack of sufficient data traceability in the dissemination process, this method could lead to the spread of artificial intelligence (AI)-generated "fake news" and "disinformation." This information could be misused or abused, which could have negative consequences for society and undermine the reliability and credibility of public information.

[0033] Accordingly, embodiments of the present invention provide an innovative solution analogous to the concept of isotopes in the physical world to track and trace the flow of data, thereby ensuring that the source of the data can be reliably traced and reducing the spread of misinformation. This “digital isotope” method functions as a tracer analysis analogous to tracking and tracing matter in the real world, ensuring the legitimacy and reliability of data, protecting intellectual property rights and the credibility of information, and maintaining order and sustainability in the digital world.

[0034] Accordingly, embodiments of the present invention use the concept of text prosodic statistics to encode digital isotopes of data in large-scale language models, which leverage the most immutable element of language, namely prosody, which can be present in all kinds of text-centric data, and this prosodic element is not significantly affected as the data changes and flows.

[0035] Therefore, embodiments of the present invention add a prosodic mapping discrimination task to the original training task according to the following steps (a) to (d).

[0036] (a) Prosodic processing is performed on all text training data, and a set of prosodic features is formed using prosodic features from aspects such as rhyme, meter, mora, tone, and homophones.

[0037] (b) Cluster the original dataset based on the prosodic feature set, added semantic features, and added related features, automatically dividing the original dataset into multiple subsets with significantly different prosodic features.

[0038] (c) For different subsets of the original dataset, sequence regression is used to model each dimension of prosody, and a final fusion is performed to obtain a mixed expert (MoE) network that can fit the prosodic features of the different subsets. The MoE network can determine which set of prosodic features the text belongs to in terms of prosody.

[0039] (d) For different texts under test, the MoE network can be used to trace the source of the text under test in a large language model training set in terms of relevant prosodic features.

[0040] Figure 1 is a flowchart illustrating a method for generating and using a mixed expert (MoE) network according to an embodiment of the present invention. The flowchart in Figure 1 includes steps 10 to 50.

[0041] Step 10 generates a dense vector from the input text and T feature types, where T is at least 1.

[0042] The input text could be, for example, a text document, which could include text paragraphs, text sentences, text phrases, text words, or combinations thereof.

[0043] The input text contains T features, for example, three features in particular: prosodic features, related features, and semantic features (T=3).

[0044] Dense vectors are encoded using features found within the input text.

[0045] Prosodic features are characteristics that arise from combining sounds in connected elements of text in a way that contributes to meaning and communication.

[0046] The prosodic features may include, in particular, some or all of the prosodic features listed in Table 1.

[0047] [Table 1. Prosodic Characteristics] [Table 1]

[0048] Related features are metadata that indicates the data source (e.g., database, file, sensor, etc.). Examples of related features are listed in Table 2.

[0049] [Table 2. Related Characteristics] [Table 2]

[0050] Semantic features relate to the meaning and relationships conveyed by words and phrases within a sentence or document. They help us understand what a text actually means, going beyond its literal form.

[0051] Step 20 clusters the dense vectors into N clusters of composite dense vectors and associates the N clusters with N data sources, where N is at least 2.

[0052] Steps 10 and 20 are described in detail in Figures 2A and 2B.

[0053] Step 30 generates a mixed expert (MoE) network from N clusters, containing N experts associated with each of the N clusters, where the N clusters are derived from the input text. Step 30 is described in detail in Figures 4A, 4B, and 5.

[0054] Step 40 uses the MoE network to identify which of the N experts n1 is most relevant to a given text unit, where n1 is selected from the group consisting of 1, ..., N.

[0055] Stage 50 identifies the source associated with expert n1 as the source most likely to be the source of a given text unit.

[0056] Stages 40 and 50 are described in detail in Figure 6.

[0057] Figure 2A is a flowchart illustrating the process of generating N clusters from input text according to an embodiment of the present invention.

[0058] Figure 2A includes steps 210-270 and provides a detailed explanation of steps 10 and 20 in Figure 1.

[0059] Step 210 segments the input text into text units. Text units may include, in particular, paragraphs, sentences, phrases, etc. Text units include prosodic features, semantic features, and / or related features.

[0060] Figure 3A provides an example illustrating the segmentation of input text 310 into paragraph 321 and 322 text units according to an embodiment of the present invention.

[0061] Figure 3B provides an example illustrating an embodiment of the present invention that segments input text 310 into text units of sentences 331, 332, 333, and 334.

[0062] Step 220 determines the logical path for generating dense vectors. The logical path consists of processing steps 231-232, 241-243, and 251-252, respectively, for processing semantic features to generate semantic dense vectors, processing prosodic features to generate prosodic dense vectors, and processing related features to generate related dense vectors.

[0063] Regarding related features, step 251 extracts related features from each text unit. To carry out this extraction, in some embodiments, metadata accompanying the text is obtained and analyzed to obtain related features from the text.

[0064] Step 252 uses the encoder portion of a Large-Scale Language Model (LLM) to encode related features into dense vectors, each dense vector being specific to a text unit.

[0065] As an example, consider a sentence as a text unit, and from it, the following related features were extracted in the format (feature:feature value): Data source: External API, Data collection time: September 28, 2023, 10:35:00; Data producer: OpenAI API; Data transfer path: API to database; Data modification history: No modifications, Data storage location: Cloud storage; Data usage record: 5 queries executed.

[0066] These aforementioned related features are passed to an encoder, which converts them into dense vectors containing real values; for example, [0.34, 0.58, 0.12, 0.76, 0.19, 0.09, 0.45, ...].

[0067] With respect to semantic features, step 231 extracts semantic features from each text unit. In at least some embodiments of the present invention, an embedding model is used to directly convert the natural language semantic and related features of the text into embedding vectors.

[0068] Step 232 uses the encoder portion of a large-scale language model (LLM) to encode semantic features into semantic dense vectors, each dense vector being specific to a text unit.

[0069] As an example, a semantic-dense vector generated by an encoder may have the following form: [0.45, 0.23, 0.81, ...].

[0070] Regarding prosodic features, step 241 extracts prosodic features from each text unit.

[0071] Step 242 generates a one-hot vector for each prosodic feature for each text unit.

[0072] As an example, consider a sentence as a text unit, from which the following prosodic features were extracted (YES means extracted; No means not extracted): (i) End rhyme: No; Visual rhyme: No; Middle rhyme: Yes; One-hot vector (0,0,1); (ii) Prosody: No for each of the four meter features; One-hot vector (0,0,0,0); (iii) Morph, tone, homophone, alliteration, synonym: No for each; Five one-hot values ​​of 0; (iv) Final one-hot vector text units of the sentence: (0,0,1), (0,0,0,0,), 0,0,0,0,0.

[0073] Step 243 generates a prosodic dense vector for each text unit using a fully connected network and the one-hot vector as input to a fully connected network trained to generate dense vectors using the one-hot vector as input, with each dense vector being specific to the text unit.

[0074] Semantic dense vectors, prosodic dense vectors, and related dense vectors are real vectors, and each dense vector has the same embedding size.

[0075] In stage 260, for each text unit, a composite dense vector is calculated as the weighted sum of dense vectors over T feature types as shown in Equation (1), where the weights w1, w2, and w3 satisfy 0≦w1≦1, 0<w2<1, 0<w3<1, and w1 + w2 + w3 = 1.

[0076] Composite DV = w1 × Rhythmic DV + w2 × Related DV + w3 × Semantic DV Equation (1)

[0077] In Equation (1), DV represents a dense vector.

[0078] For example, w1, w2, and w3 may have values of w1 = 0.5, w2 = 0.3, and w3 = 0.2.

[0079] In different embodiments, w2 can be zero, w3 can be zero, or both w2 and w3 can be zero.

[0080] In all embodiments, w1 is a positive real number.

[0081] In stage 270, by implementing a clustering algorithm, such as k - means, DBSCAN, etc., the composite dense vectors of all text units are clustered into N clusters. Each cluster contains a plurality of composite dense vectors, and each composite dense vector is unique to a text unit. Thus, different composite dense vectors within a cluster can be unique to different text units because all the composite dense vectors within a cluster share significantly similar rhythmic features, related features, and / or semantic features.

[0082] After each cluster n among the N clusters is formed, a data source n is identified, where the data source n has the same or significantly similar rhythmic features, related features, and / or semantic features as the cluster n (n = 1, …, N), and as a result, the N clusters are respectively associated with N data sources.

[0083] The N data sources, each associated with N clusters, are determined by applying the process in Figure 2A to each of the M distinct data sources as follows, where M is at least 2, and where each of the M distinct data sources contains source text.

[0084] The process in Figure 2A is performed for each data source m by modifying step 210 to segment the source text within data source m into text units instead of segmenting the input text into text units, resulting in the generation of multiple source clusters for each data source m (m=1, ..., M) in step 270. The number of source clusters generated may vary across different data sources as a function of the source text within each data source m and the parameter values ​​of the clustering algorithm (k-means, DBSCAN, etc.) used to form multiple source clusters for data source m. Let S collectively represent the total number of source clusters generated for all M data sources.

[0085] After the process in Figure 2A is performed for all M data sources, each cluster n of the N clusters is matched with each source cluster in a subset of S source clusters (in one embodiment, the subset may encompass all S source clusters) to find a match based on a similarity metric such as Euclidean distance or cosine similarity. When a match is found between cluster n and one source cluster of one of the S data sources, cluster n is associated with that data source. In this way, a data source matching each cluster n (n=1, ..., N) is found, and the N clusters are each associated with the N data sources found. The N data sources are used to determine the most likely data source for the input text in step 670 of Figure 6, which will be described later. Data sources of the M data sources that are not found to match any of the N clusters are not subsequently used in the implementation of the embodiments of the present invention. For example, if M=30 and N=10, at stage 670 in Figure 6, 10 of the 30 data sources found are used, and the remaining 20 data sources are not used thereafter. In an alternative scenario, if the number of actual data sources M is less than the number of identified clusters N (e.g., M=20 and N=25), the MoE network still works, but 5 of the N clusters (i.e., 25 clusters) are not associated with any of the data sources M. In this alternative scenario, if a new given text sample matches one of the unassociated clusters during inference (e.g., one of the 5 unassociated clusters in the M=20 and N=25 example), the system produces a blank output (e.g., no identified associated data sources).Even when N is greater than M, the system can still produce associated data source matches for a new given input data text that matches one of the clusters that have associated data sources (for example, it can still produce data source matches for 20 clusters).

[0086] Figure 2B shows steps 260 and 270 in Figure 2A, which, according to an embodiment of the present invention, collectively combine the related dense vector 281, the prosodic dense vector 282, and the semantic dense vector 283 to form a composite dense vector 290 (step 260), and then cluster the composite dense vector into N clusters denoted as C1, C2, ..., CN (step 270).

[0087] Figure 4A is a flowchart illustrating the process for generating a mixed expert (MoE) network from N clusters, each containing N experts associated with the N clusters, according to an embodiment of the present invention. Figure 4A includes steps 410-440 and provides a more detailed explanation of step 30 in Figure 1.

[0088] Step 410 initializes the cluster index n to zero.

[0089] Steps 420-440 define a loop for iterating through the clusters among the N clusters.

[0090] Step 420 increments n by 1.

[0091] Step 430 trains a transformer model n using cluster n as training data, and generates an expert n, which is a machine learning model n consisting of the trained transformer n, where n is one of 1, ..., N. In step 430, only the encoder portion of the transformer model is used.

[0092] A transformer model is a neural network that learns the context of the sequential data from which it generates new data.

[0093] The transformer model possesses a self-attention mechanism, which allows it to process input data more efficiently and capture long-range dependencies within a sequence without relying on recursion, in contrast to conventional recurrent neural networks (RNNs). The self-attention mechanism calculates the relevance of each token in a sequence, where a token can be a word, by comparing it to other tokens in the sequence, which allows the transformer model to understand the contextual relationships between tokens regardless of the distance between them in the sequence. The transformer model processes all tokens in parallel using several self-attention layers in parallel, which allows the transformer model to focus on different parts of the sequence simultaneously and capture a richer set of relationships. The encoder portion trained in stage 430 includes, for example, the self-attention mechanism described above.

[0094] The encoder portion of the transformer model transforms input tokens into contextualized representations. Unlike traditional models that processed tokens individually, the transformer encoder captures the context of each token within the entire sequence of tokens.

[0095] Given a sequence of features in a dense vector of cluster n, each expert n's machine language model predicts the next feature f+1 from the current feature f and previous features f-1, f-2, ... in the sequence of features.

[0096] Stage 430 is explained in detail in Figure 5.

[0097] Step 440 determines whether n=N. If so (Yes branch from step 440), the process in Figure 4A ends; otherwise (No branch from step 440), the process loops back to step 420 and performs the next iteration n+1.

[0098] Figure 4B shows a step in training a large-scale language model (LLM) network 460 and a MoE network 470 together, according to an embodiment of the present invention, using the input text 450 as the source of training data for both the LLM network 460 and the MoE network 470.

[0099] The LLM network 460 calculates the LLM loss 481, which is minimized via backpropagation configured to adjust the weights and / or parameters of the LLM 460, and the MoE network 470 calculates the MoE loss 482, which is minimized via backpropagation configured to adjust the weights and / or parameters of the MoE network 470, which focuses on optimizing each expert within the MoE network 470 based on cluster-specific features, which are combined in equation (2) to calculate the total loss 490.

[0100] Total loss = LLM loss + α × MoE loss Equation (2)

[0101] The parameter α is a real number in the range of 0.005 < α < 0.100, weighted by the MoE loss 482 to the LLM loss 481. In one embodiment, α = 0.10.

[0102] The MoE loss 482 for MoE network 470 is a composite cluster loss calculated as the sum of the cluster losses of clusters 1 through N.

[0103] The cluster loss of the cluster is calculated in step 590 in Figure 5.

[0104] The total loss 490 is incorporated into both the training of the LLM network 460 and the training of the MoE network 470 via feedback paths 491 and 492, respectively.

[0105] In the case of the LLM network 460, the total loss of 490, including the MoE loss of 482, influences the optimization of the LLM network 460, which helps the LLM network 460 remain sensitive to cluster-specific features by slightly fitting the parameters of the LLM network 460 to the discriminative characteristics of the cluster.

[0106] In the case of MoE network 470, the MoE loss 482 directly affects each expert within MoE network 470 based on cluster assignment. By including the LLM loss 481 in the calculation of the total loss 490, training MoE network 470 encourages the refinement of cluster-specific focus within MoE network 470 while maintaining consistency aligned with broader language modeling objectives.

[0107] In an alternative embodiment, the MoE network 470 is trained independently of the LLM network 460 in the same manner as described above, except that in this alternative embodiment, there is no feedback path 492.

[0108] Figure 5 illustrates a step according to an embodiment of the present invention for training a transformer model n using cluster n as training data to generate an expert n associated with cluster n.

[0109] Figure 5 includes stages 510-590 and provides a more detailed explanation of stage 430 in Figure 4A.

[0110] The transformer model n is one of the transformer models n such that (n=1, ..., N). Cluster n contains D dense vectors, each dense vector containing F features, where F can vary among the dense vectors.

[0111] Step 510 initializes the dense vector index d to zero.

[0112] Steps 520-580 define the outer loop over the dense vectors 1-D.

[0113] Step 520 increments d by 1.

[0114] Step 530 initializes the feature index f to zero.

[0115] Steps 540-560 define the inner loop spanning features 1-F.

[0116] Step 540 increments f by 1.

[0117] In step 550, the transformer model encoder predicts feature f+1, calculates the feature loss of feature f+1 which is equal to the difference between the predicted feature f+1 and the actual feature f+1 in the dense vector d, and adjusts the expert model n (e.g., via backpropagation; e.g., by adjusting the model weights and / or parameters) to minimize the feature loss of feature f+1.

[0118] The prediction of feature f+1 is based on the current feature f and previous features f-1, f-2, ...

[0119] Step 560 determines whether f = F-1. If so (Yes branch from step 560), then step 570 is executed; otherwise (No branch from step 560), the process loops back to step 540 and processes the next feature f+1.

[0120] Step 570 calculates the dense vector loss of the dense vector d.

[0121] In one embodiment, the dense vector loss of the dense vector d is equal to the sum of the feature losses of the F features, where the feature loss of each feature f is calculated in step 550.

[0122] In one embodiment, the dense vector loss of the dense vector d is equal to the magnitude of the difference between the dense vector d and the centroid dense vector of cluster n.

[0123] Step 580 determines whether d=D. If so (Yes branch from step 580), then step 590 is executed; otherwise (No branch from step 580), the process loops back to step 520 and processes the next dense vector d+1.

[0124] Step 590 calculates the cluster loss for cluster n, which is equal to the sum of the dense vector losses of the D dense vectors, and the dense vector loss for each dense vector d is the one calculated in step 570. As discussed above in conjunction with Figure 4B, the cluster loss for cluster n is used to calculate the MoE loss 482 for MoE network 470, where the MoE loss 482 is a composite cluster loss calculated as the sum of the cluster losses n of clusters 1 to N.

[0125] Thus, the MoE structure allows transformer models to specialize for different clusters. For example, expert n is tuned based on the data of cluster Cn, and for each cluster n (n=1, ..., N), it learns the subtle relationships between features specific to cluster n.

[0126] Figure 6 is a flowchart illustrating how a trained MoE network is used to determine the data source of a given text unit with high probability, according to an embodiment of the present invention. The input to the MoE network is a given text unit. The output from using the trained MoE network is a discriminant output, represented as an N-dimensional vector, whose elements represent a composite dense vector derived from a given text input and the similarity between the cluster centroids associated with each expert in the MoE network. The LLM460, which participated in the co-training, may but is not required to be used in the inference portion, in which, in step 630 of Figure 6, the trained MoE network 470 is used to generate a discriminant output for later use in performing a data trace from a given text to a specific data source.

[0127] The flowchart in Figure 6 includes steps 610 to 670.

[0128] Stage 610 receives a given text unit as input.

[0129] Step 620 generates a composite dense vector of features present in a given text unit from a given text unit, using the methodology for calculating composite dense vectors described above for Figures 2A and 2B, including step 260 in Figure 2A.

[0130] Stage 630 is an inference stage in which the trained MoE network is used to compute a discriminant output, which is the distance between the cluster centroids used to generate each expert, and a composite dense vector. Thus, stage 630 generates N distances associated with each of the N experts, which are represented as an N-dimensional output vector. The smallest distance has the highest similarity between the composite dense vector derived from the given text input and the cluster centroids associated with each expert in the MoE network.

[0131] In one embodiment, the distance is Euclidean distance. In another embodiment, the distance is based on cosine similarity.

[0132] Step 640 performs a logistic operation on each of the N distances, which generates N logistic distances associated with each of the N experts. The logistic function converts a real value x into a probability between 0 and 1, so logistic(x) = 1 / (1-e -x ) is defined as follows.

[0133] Stage 650 is the minimum logistic distance D logistic_min Select an expert n1 (from expert 1, ..., expert N) that possesses the following characteristics.

[0134] Stage 660 is the minimum logistic distance D logistic_min The defined logistic distance threshold D logistic_th Determine if it is less than or equal to . If so (Yes branch from step 660), then step 670 is executed; otherwise (No branch from step 660), the process ends.

[0135] Step 670 identifies the data source n1 associated with the cluster n1 used to generate expert n1 as the most likely data source for a given text unit. This identification is performed in various embodiments by generating, sending, and presenting a message indicating the identified data source n1. The presentation is performed in some embodiments via a visible display on a computer display screen, such as the display screen of the UI device set 123 of computer 101, which is shown in Figure 8 and will be described later.

[0136] As an example, consider a trained MoE network with 5 experts (N=5). The 5-dimensional output vectors are 0.58, 0.342, 0.12, 0.94, and 0.74, which are the logistic distances calculated at stage 630. Expert 3 has the smallest logistic distance (0.12) and is therefore the closest expert to cluster 3, and its centroid density vector has the highest similarity to the composite density vector derived from a given text input.

[0137] Therefore, if the logistic distance of 0.12 is less than the threshold logistic distance, source 3 associated with cluster 3 is identified as the most likely data source for a given text unit.

[0138] As another example, suppose we have training data from two different sources S1 and S2, where S1 contains text with weak-strong prosody (e.g., "To be or not to be") and S2 contains text with strong-strong prosody (e.g., "Peter, Peter, pumpkin eater").

[0139] Based on prosodic features, sources S1 and S2 are clustered into different clusters C1 and C2, respectively, where cluster C1 corresponds to weak-strong prosodices and cluster C2 corresponds to strong-strong prosodices. Clusters C1 and C2 are linked to experts E1 and E2, respectively. In this example, we assume N=2 (i.e., two experts and associated clusters).

[0140] Assume that a third party generates a text unit, which is then treated as a given text unit processed in Figure 6 using an N=2 MoE network.

[0141] The 2D output vectors are 0.23 and 0.45, which are the logistic distances calculated in steps 620 and 630. Expert 1 has the smallest logistic distance (0.23) and is therefore the closest expert to cluster 1, and its centroid density vector is the closest to the composite density vector derived from the given text input.

[0142] Therefore, if the logistic distance of 0.23 is less than the logistic distance threshold, Source 1 associated with cluster 1 (weak-stress prosody) is identified as the most likely data source for a given text unit. A message indicating that Source 1 is the most likely source for a given text unit is generated, sent, and presented. The message is shown in Figure 8 and displayed on the display screen of the UI device set 123 of computer 101, which will be described later.

[0143] Figure 7 shows a computer system 90 according to an embodiment of the present invention.

[0144] The computer system 90 includes a processor 91, an input device 92 coupled to the processor 91, an output device 93 coupled to the processor 91, and memory devices 94 and 95, respectively, coupled to the processor 91. The processor 91 represents one or more processors and may indicate a single processor or multiple processors. The input device 92 may be, in particular, a keyboard, mouse, camera, touchscreen, etc., or a combination thereof. The output device 93 may be, in particular, a printer, plotter, computer screen, magnetic tape, removable hard disk, floppy disk, etc., or a combination thereof. The memory devices 94 and 95 may be, in particular, a hard disk, floppy disk, magnetic tape, optical storage such as a Compact Disc (CD) or Digital Video Disc (DVD), Dynamic Random Access Memory (DRAM), Read-Only Memory (ROM), etc., or a combination thereof. The memory device 95 includes computer code 97. The computer code 97 includes an algorithm for performing embodiments of the present invention. The processor 91 executes computer code 97. The memory device 94 contains input data 96. The input data 96 contains inputs required by computer code 97. The output device 93 displays the output from computer code 97. Either or both of the memory devices 94 and 95 (or one or more additional memory devices such as the read-only memory device 96) may be used as a computer-readable medium (or computer-readable medium or program storage device) in which algorithms, computer-readable program code is internally embodied and / or other data is stored internally, where computer-readable program code includes computer code 97. In general, the computer program product (or alternatively manufactured article) of the computer system 90 may include a computer-readable medium (or program storage device).

[0145] In some embodiments, rather than being stored in and accessed from a hard drive, optical disk, or other writable, rewritable, or removable hardware memory device 95, stored computer program code 99 (including, for example, algorithms) may be stored in a static, non-removable, read-only storage medium, such as a read-only memory (ROM) device 98, or may be accessed directly by the processor 91 from such a static, non-removable, read-only medium 98. Similarly, in some embodiments, stored computer program code 99 may be stored as computer-readable firmware, rather than from a more dynamic or removable hardware data storage device 95, such as a hard drive or optical disk, or may be accessed directly by the processor 91 from such firmware.

[0146] Furthermore, any of the components of the present invention may be created, integrated, hosted, maintained, deployed, managed, and serviced by a service provider that provides improvements to software technologies related to cross-referencing metrics associated with the plug-in components, generating software code modules, and enabling the operational functions of the target cloud components. Accordingly, the present invention discloses a process for deploying, creating, hosting, maintaining, and / or integrating computing infrastructure, which includes integrating computer-readable code into a computer system 90, where the code combined with the computer system 90 can implement methods for enabling the processes of cross-referencing metrics associated with the plug-in components, generating software code modules, and improving software technologies related to enabling the operational functions of the target cloud components. In another embodiment, the present invention provides a business method for implementing the process steps of the present invention on a subscription, advertising, and / or commission basis. That is, a service provider, such as a solution integrator, can provide enabling the processes for cross-referencing metrics associated with the plug-in components, generating software code modules, and improving software technologies related to enabling the operational functions of the target cloud components. In this case, the service provider can create, maintain, and support computer infrastructure that implements the process steps of the present invention for one or more customers. In return, the service provider may receive payments from customers based on subscription agreements and / or fee agreements, and / or the service provider may receive payments from the sale of advertising content to one or more third parties.

[0147] Figure 7 shows computer system 90 as a specific hardware and software configuration, but as is known to those skilled in the art, any hardware and software configuration may be used in combination with the specific computer system 90 in Figure 7 for the purposes described above. For example, memory devices 94 and 95 may not be separate memory devices but rather parts of a single memory device.

[0148] The computer program product of the present invention comprises one or more computer-readable hardware storage devices in which computer-readable program code is stored internally, and the program code includes instructions that can be executed by one or more processors of a computer system for implementing the method of the present invention.

[0149] The computer system of the present invention comprises one or more processors, one or more memories, and one or more computer-readable hardware storage devices, wherein the one or more hardware storage devices include program code executable by one or more processors via one or more memories for implementing the method of the present invention.

[0150] Various aspects of this disclosure are described by explanatory text, flowcharts, block diagrams of computer systems, and / or block diagrams of mechanical logic included in embodiments of a Computer Program Product (CPP). In any flowchart, depending on the technology involved, operations may be performed in an order different from the order shown in a given flowchart. For example, again, depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated stage, simultaneously, or in a manner that at least partially overlaps in time.

[0151] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of storage media (also called "mediums") that collectively comprise a set of storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. "Storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any preferred combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital purpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices (such as pits / lands formed on the main surface of punch cards or disks), or any preferred combination of those described above. When the term "computer-readable storage medium" is used in this disclosure, it shall not be construed as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media. As those skilled in the art will understand, data is typically moved at any given time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection; however, data is not transient while it is stored, and this does not make the storage device transient.

[0152] Figure 8 shows a computing environment 100 that includes an example of an environment for executing at least some of the computer code involved in carrying out the method of the present invention according to an embodiment of the present invention. Such computer code includes new code for generating and using a mixed expert (MoE) network 180. In addition to block 180, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including a processing circuit configuration 120 and a cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including an operating system 122 and block 180 as identified above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes the remote database 130. The public cloud 105 includes the gateway 140, the cloud orchestration module 141, the host physical machine set 142, the virtual machine set 143, and the container set 144.

[0153] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that is capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the implementation of a computer implementation method may be distributed among multiple computers and / or multiple locations. On the other hand, in this description of the computing environment 100, in order to make the description as concise as possible, the detailed discussion will focus on a single computer, specifically computer 101. Computer 101 may be located in the cloud, although it is not shown in the cloud in Figure 1. On the other hand, it is not necessary for computer 101 to be located in the cloud, except to the extent that it can be definitively shown.

[0154] The processor set 110 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuit configuration 120 may be distributed across multiple packages, for example, multiple cooperative integrated circuit chips. The processing circuit configuration 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuit configuration. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may operate using qubits and be designed to perform quantum computing.

[0155] Computer-readable program instructions are typically loaded onto computer 101, causing the processor set 110 of computer 101 to execute a series of operational steps, thereby enabling the computer implementation method. As a result, the instructions thus executed instantiate the method specified in the flowcharts and / or descriptions of the computer implementation method contained herein (collectively referred to as the "Method of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the implementation of the Method of the Invention. In computing environment 100, at least some of the instructions for implementing the Method of the Invention may be stored in block 180 in persistent storage 113.

[0156] The communication fabric 111 is a signal conduction path that enables various components of the computer 101 to communicate with one another. Typically, this fabric consists of switches and conductive paths, such as buses, bridges, physical input / output ports, and switches and conductive paths that make up similar components. Other types of signal communication paths, such as optical fiber communication paths and / or wireless communication paths, may be used.

[0157] The volatile memory 112 is any type of volatile memory that is currently known or may be developed in the future. Examples include dynamic random-access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, but this is not required unless explicitly stated. In computer 101, the volatile memory 112 is located in a single package and is internal to computer 101, but alternatively or additionally, the volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0158] The persistent storage 113 is any form of non-volatile storage for computers that is currently known or may be developed in the future. Non-volatility of this storage means that the stored data is maintained whether or not power is supplied to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 can take multiple forms, such as various known proprietary operating systems or open-source portable operating system interface operating systems using a kernel. The code contained in block 180 typically includes at least a portion of computer code involved in carrying out the method of the present invention.

[0159] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the computer 101's peripheral devices and other components can be implemented in various ways, including Bluetooth® connections, Near-Field Communication (NFC) connections, connections made by cables (such as universal serial bus (USB) type cables), insert-type connections (e.g., secure digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the internet. In various embodiments, the UI device set 123 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic devices. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing memory device that stores data in the form of qubits. In embodiments where computer 101 needs to have large-capacity storage (for example, when computer 101 locally stores and manages a large database), this storage can then be provided by peripheral storage devices designed to store very large amounts of data, such as a Storage Area Network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0160] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, thereby allowing the control function to manage multiple different network hardware devices. Computer-readable program instructions for carrying out the method of the present invention can typically be downloaded from an external computer or external storage device to computer 101 via a network adapter card or network interface included in the network module 115.

[0161] WAN102 is any wide area network (e.g., the Internet) that can transmit computer data over non-local distances using any technology currently known or to be developed for transmitting computer data. In some embodiments, WAN102 may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi network, designed to transmit data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0162] An end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101) and can take any of the forms described above in relation to computer 101. EUD 103 typically receives useful and valuable data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end-user, these recommendations are typically communicated from computer 101's network module 115 to EUD 103 via WAN 102. Thus, EUD 103 can display or otherwise present recommendations to the end-user. In some embodiments, EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0163] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may then be provided to computer 101 from the remote database 130 of the remote server 104.

[0164] A public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without direct, active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct and active management of the computing resources of the public cloud 105 is carried out by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by a virtual computing environment running on various computers that make up the host physical machine set 142, which is a universe of physical computers located within and / or available to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs can be stored as images and transferred between various physical machine hosts, either as images or after VCE instantiation. The cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instantiations, and manages active instantiations of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate through the WAN 102.

[0165] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature in which the kernel allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave as actual computers from the perspective of the programs running within them. Computer programs running on a normal operating system can utilize all the resources of that computer, including connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container. This is a feature known as containerization.

[0166] The private cloud 106 is similar to the public cloud 105, except that its computing resources are available for use only by a single enterprise. Although the private cloud 106 is shown communicating with the WAN 102, in other embodiments the private cloud may be completely isolated from the internet and accessible only through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often each implemented by a different vendor. Each of the multiple clouds remains a separate, discrete entity, but larger hybrid cloud architectures are coupled together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configuration clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0167] Cloud computing services and / or microservices (not shown separately in Figure 1): Private and public clouds 106 are programmed and configured to deliver cloud computing services and / or microservices (unless otherwise indicated, the term “microservices” should be interpreted as including larger “services” of any size). Cloud services are typically infrastructure, platforms, or software hosted by a third-party provider and made available to users over the internet. Cloud services facilitate the flow of user data from front-end clients (e.g., user-side servers, tablets, desktops, laptops) through the internet to the provider’s systems and back. In some embodiments, cloud services may be configured and orchestrated according to the “as a service” technology paradigm, where something is presented to internal or external customers in the form of cloud computing services. As a service typically provides endpoints that various customers interface with. These endpoints are typically based on a set of APIs. One category of as-a-service offerings is Platform as a Service (PaaS), where a service provider provisions, instantiates, runs, and manages modular bundles of code that customers can use to instantiate a computing platform and one or more applications without the complexity of building and maintaining the infrastructure typically associated with them. Another category is Software as a Service (SaaS), where software is centrally hosted and allocated on a subscription basis. SaaS is also known as on-demand software, web-based software, or web-hosted software. The four technical subfields involved in cloud services are: deployment, integration, on-demand, and virtual private networks.

[0168] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Those skilled in the art will see many modifications and variations without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best describe the principles of the embodiments, their practical application to market technologies, or technical improvements, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. The first step involves training a transformer machine learning model using each of the N clusters n as training data to generate each trained transformer n, which is a machine learning model that is an expert n (n=1, ..., N), where the N experts are each associated with the N clusters, where N is at least 2, where the N trained transformers together form a mixed expert (MoE) network, where the N clusters are derived from input text. A computer implementation method comprising the above.

2. The step of segmenting the input text into text units, The step of generating a semantic dense vector by extracting semantic features from each text unit and encoding the semantic features into a semantic dense vector for each text unit using the encoder portion of a generative language machine learning model; A step of generating a related dense vector by extracting related features from each text unit and, for each text unit, encoding the related features into a related dense vector using the encoder portion of the generative language machine learning model; The step of generating a prosodic dense vector is to extract prosodic features from each text unit, generate a one-hot vector for each prosodic feature of each text unit, and then generate a prosodic dense vector for each text unit using the one-hot vector by a fully connected network; For each text unit, the steps include: calculating a composite dense vector as a weighted sum of the semantic dense vector, the related dense vector, and the prosodic dense vector; and The step of clustering the composite dense vectors of all text units into the N clusters and associating the N clusters with the number of each data source, where the number of unique data sources is equal to or less than N; The computer implementation method according to claim 1, further comprising the step of generating the N clusters via

3. The MoE network is an external adapter for a generative language machine learning model, and the computer implementation method is (i) The step of training the generative language machine learning model using the input text as training data, and (ii) The step of training the transformer model using each cluster n as training data (n=1, ..., N), are performed together. The computer implementation method according to claim 1 or 2, further comprising:

4. The aforementioned stage of training together is A step of determining the first loss of the generative language machine learning model and the MoE loss of the MoE network; The step of calculating the total loss equal to the first loss + α × MoE loss, where α is a specified constant in the range of 0.005 to 0.10; and The step of incorporating the total loss into both training the generative language machine learning model and training the transform model. A computer implementation method according to claim 3, comprising:

5. The stage of receiving a given text unit; A step of generating a composite dense vector of features present in a given text unit from the given text unit; A step of calculating the distance between the composite dense vector and the centroid of each of the N clusters, thereby generating N distances associated with each of the N experts; A step of performing a logistic operation on each of the N distances, thereby generating N logistic distances associated with each of the N experts, referred to as Expert 1, ..., Expert N; and From the N experts, the smallest logistic distance D among the N logistic distances. ｌｏｇｉｓｔｉｃ＿ｍｉｎ A step in selecting an expert n1 having, where n1 is 1, ..., or N. The computer implementation method according to claim 1 or 2, further comprising:

6. The aforementioned clusters 1, ..., N are associated with data sources 1, ..., N respectively, and the computer implementation method is D ｌｏｇｉｓｔｉｃ＿ｍｉｎ The defined logistic distance threshold distance D ｌｏｇｉｓｔｉｃ＿ｔｈ The step of determining that it is smaller than and, accordingly, identifying the data source n1 associated with the cluster n1 as the most likely data source for the given text unit. The computer implementation method according to claim 5, further comprising:

7. The computer implementation method according to claim 1, wherein the text data collectively comprises a plurality of text units, and the training data within each cluster n (n = 1, ..., N) comprises a plurality of dense vectors associated with each of the plurality of text units, each dense vector is based on the presence or absence of a feature within the plurality of text units to which the plurality of dense vectors are associated, and the feature comprises a plurality of prosodic features.

8. The computer implementation method according to claim 7, wherein the aforementioned features further include one or more closely related features, one or more semantic features, or a combination thereof.

9. On the computer, A procedure to train a transformer model using each of N clusters n as training data to generate a trained transformer n, which is a machine learning model with expert n (n = 1, ..., N), where the N experts are each associated with the N clusters, where N is at least 2, where the MoE network includes the N experts, where the N clusters are derived from input text. A computer program designed to execute something.

10. To the aforementioned computer, A procedure for segmenting the aforementioned input text into text units; A procedure for generating semantic dense vectors by extracting semantic features from each text unit and, for each text unit, encoding the semantic features into semantic dense vectors using the encoder portion of a large-scale language model; A procedure for generating a related dense vector by extracting related features from each text unit and, for each text unit, encoding the related features into a related dense vector using the encoder portion of the large-scale language model; A procedure for generating a prosodic dense vector by extracting prosodic features from each text unit, generating a one-hot vector for each prosodic feature of each text unit, and then generating a prosodic dense vector for each text unit using the one-hot vectors via a fully connected network; A procedure for calculating a composite dense vector for each text unit as a weighted sum of the semantic dense vector, the related dense vector, and the prosodic dense vector; and A procedure for clustering the composite dense vectors of all text units into N clusters, and associating the N clusters with the number of each data source, where the number of unique data sources is equal to or less than N. A computer program according to claim 9 for further execution of the above.

11. The aforementioned MoE network is an external adapter for the Large-Scale Language Model (LLM) network, To the aforementioned computer, A procedure that combines the following steps: (i) the procedure for training the LLM network using the input text as training data, and (ii) the procedure for training the transformer model using each cluster n as training data (n=1, ..., N). A computer program according to claim 9 or 10 for further execution of the above.

12. The aforementioned procedure for training together is: A procedure for determining the LLM loss of the LLM network and the MoE loss of the MoE network; A procedure for calculating the total loss equal to the LLM loss + α × MoE loss, where α is a specified constant in the range of 0.005 to 0.10; and A procedure for incorporating the total loss into both the training of the LLM network and the training of the transformer model. A computer program according to claim 11, having the following characteristics.

13. To the aforementioned computer, Procedure for receiving a given text unit; A procedure for generating a composite dense vector of features present in a given text unit from the given text unit; A procedure for calculating the distance between the centroids of each cluster among the composite dense vector and the N clusters, thereby generating N distances associated with each of the N experts; A procedure for performing a logistic operation on each of the N distances, thereby generating N logistic distances associated with each of the N experts, referred to as Expert 1, ..., Expert N; and From the N experts, the smallest logistic distance D among the N logistic distances. ｌｏｇｉｓｔｉｃ＿ｍｉｎ A procedure for selecting an expert n1 having, where n1 is 1, ..., or N. A computer program according to claim 9 or 10 for further execution of the above.

14. Clusters 1, ..., N are associated with data sources 1, ..., N respectively. To the aforementioned computer, D ｌｏｇｉｓｔｉｃ＿ｍｉｎ The defined logistic distance threshold distance D ｌｏｇｉｓｔｉｃ＿ｔｈ A procedure to determine that it is smaller than and, accordingly, to identify the data source n1 associated with the cluster n1 as the most likely data source for the given text unit. A computer program according to claim 13 for further execution of the above.

15. Processor set; One or more computer-readable storage media; and Program instructions stored in the one or more computer-readable storage media mentioned above, A computer system comprising the following, wherein the program instructions are provided to the processor set: A procedure to train a transformer model using each of N clusters n as training data to generate a trained transformer n, which is a machine learning model with expert n (n = 1, ..., N), where the N experts are each associated with one of the N clusters, where N is at least 2, where the MoE network includes the N experts, where the N clusters are derived from input text. A computer system that performs an action having the following characteristics.

16. The aforementioned operation is, A procedure for segmenting the aforementioned input text into text units; A procedure for generating semantic dense vectors by extracting semantic features from each text unit and, for each text unit, encoding the semantic features into semantic dense vectors using the encoder portion of a large-scale language model; A procedure for generating a related dense vector by extracting related features from each text unit and, for each text unit, encoding the related features into a related dense vector using the encoder portion of the large-scale language model; A procedure for generating a prosodic dense vector by extracting prosodic features from each text unit, generating a one-hot vector for each prosodic feature of each text unit, and then generating a prosodic dense vector for each text unit using the one-hot vectors via a fully connected network; A procedure for calculating a composite dense vector for each text unit as a weighted sum of the semantic dense vector, the related dense vector, and the prosodic dense vector; and A procedure for clustering the composite dense vectors of all text units into N clusters, and associating the N clusters with the number of each data source, where the number of unique data sources is equal to or less than N. The computer system according to claim 15, further comprising the above.

17. The aforementioned MoE network is an external adapter for a Large-Scale Language Model (LLM) network, and its operation is as follows: A procedure that combines the steps of (i) training the LLM network using the input text as training data, and (ii) training the transformer model using each cluster n as training data (n=1, ..., N). The computer system according to claim 15 or 16, further comprising the above.

18. The aforementioned procedure for training together is: A procedure for determining the LLM loss of the LLM network and the MoE loss of the MoE network; A procedure for calculating the total loss equal to the LLM loss + α × MoE loss, where α is a specified constant in the range of 0.005 to 0.10; and A procedure for incorporating the total loss into both the training of the LLM network and the training of the transformer model. The computer system according to claim 17, having the following features.

19. The aforementioned operation is, Procedure for receiving a given text unit; A procedure for generating a composite dense vector of features present in a given text unit from the given text unit; A procedure for calculating the distance between the centroids of each cluster among the composite dense vector and the N clusters, thereby generating N distances associated with each of the N experts; A procedure for performing a logistic operation on each of the N distances, thereby generating N logistic distances associated with each of the N experts, referred to as Expert 1, ..., Expert N; and From the N experts, the smallest logistic distance D among the N logistic distances. ｌｏｇｉｓｔｉｃ＿ｍｉｎ A procedure for selecting an expert n1 having, where n1 is 1, ..., or N. The computer system according to claim 15 or 16, further comprising the above.

20. Clusters 1, ..., N are associated with data sources 1, ..., N respectively, and the operation described above is: D ｌｏｇｉｓｔｉｃ＿ｍｉｎ is determined to be smaller than the defined logistic distance threshold distance D ｌｏｇｉｓｔｉｃ＿ｔｈ and accordingly, a procedure for identifying the data source n1 associated with the cluster n1 as the most likely data source of the given text unit The computer system according to claim 19, further comprising the above.

Images