Generating small language models through two-stage training

Abstract

Systems and methods for generating a small language model are provided. In particular, a computing device can obtain a general-purpose dataset comprising a plurality of general-purpose data, annotate a subset of the general-purpose dataset based on one or more classifier metrics indicative of a quality of the general-purpose dataset, train a classifier based on the annotated subset of the general-purpose dataset and the one or more classifier metrics, analyze each general-purpose data of the general-purpose dataset using the trained classifier to determine a score for each of the one or more classifier metrics associated with the respective general-purpose data, generate a filtered general-purpose dataset by filtering the general-purpose dataset based on one or more filters, train a small language model with the filtered general-purpose dataset, generate a synthetic dataset for refining the small language model, and train the small language model with the synthetic dataset.

Description

Background Technology

[0001] Over the past few years, Large Language Models (LLMs) have transformed the field of natural language processing. More broadly, they promise to drive a paradigm shift in human-computer interaction. These advancements have profound economic implications and the potential to redefine the conceptual framework of artificial intelligence and even potentially understand itself. LLMs achieve high levels of capability using massive amounts of parameters and training data lexicons. Improvements from one generation of LLMs to the next currently appear to stem primarily from increased scale, with the most powerful models approaching trillions of parameters and trillions of training data lexicons. However, the cost of training, deploying, and maintaining such large models can be substantial. From a responsible AI perspective, the energy consumption of large-scale models is increasingly concerning, as are the issues of controllability or manageability of these models.

[0002] Embodiments have been described in relation to these and other general considerations. Furthermore, although relatively specific problems have been discussed, it should be understood that the embodiments are not limited to solving the specific problems identified in the background section. Summary of the Invention

[0003] According to examples in this disclosure, a small language model (SLM) is generated via two-stage training to achieve capabilities similar to a large language model (LLM). A small language model is a generative machine learning model that, compared to an LLM, has fewer parameters and is trained with less training data. Specifically, the two-stage training comprises a first training phase and a second training phase. The first training phase involves generating a filtered general dataset from various sources and training the small language model using the filtered general dataset. The second training phase involves generating a synthetic dataset using prompts and training the small language model using the synthetic dataset. By curating training data from two separate sources through two-stage training, the small language model is trained with data of a specific quality suitable for understanding common-sense reasoning and general world knowledge, as well as performing specific application and reasoning tasks (e.g., common-sense or logical reasoning).

[0004] According to at least one example of this disclosure, a method for generating a small language model is provided. The method may include: obtaining a general dataset comprising multiple general data sets; annotating a subset of the general dataset based on one or more classifier metrics indicating the quality of the general dataset, the subset representing the general dataset; training a classifier based on the annotated subset of the general dataset and one or more classifier metrics; analyzing each general data set in the general dataset using the trained classifier to determine a score for each classifier metric among one or more classifier metrics associated with the corresponding general data set; generating a filtered general dataset by filtering the general dataset based on one or more filters indicating threshold scores for corresponding classifier metrics; training a small language model with the filtered general dataset; generating a synthetic dataset for refining the small language model; and training the small language model with the synthetic dataset after training it with the filtered general dataset.

[0005] According to at least one example of this disclosure, a computing device for generating a small language model is provided. The computing device may include a processor and a memory having a plurality of instructions stored thereon, which, when executed by the processor, cause the computing device to generate the small language model. The method includes: obtaining a general dataset comprising a plurality of general data; annotating a subset of the general dataset based on one or more classifier metrics indicating the quality of the general dataset, the subset representing the general dataset; training a classifier based on the annotated subset of the general dataset and the one or more classifier metrics; analyzing each general data point in the general dataset using the trained classifier to determine a score for each classifier metric among one or more classifier metrics associated with the corresponding general data point; generating a filtered general dataset by filtering the general dataset based on one or more filters, the one or more filters indicating threshold scores for corresponding classifier metrics; training the small language model with the filtered general dataset; generating a synthetic dataset for refining the small language model; and training the small language model with the synthetic dataset after training it with the filtered general dataset.

[0006] According to at least one example of this disclosure, a computer storage medium is provided. The computer storage medium stores computer-executable instructions that, when executed, cause at least one processor to perform operations. The operations include: obtaining a general dataset comprising multiple general data; annotating a subset of the general dataset based on one or more classifier metrics indicating the quality of the general dataset, the subset representing the general dataset; training a classifier based on the annotated subset of the general dataset and the one or more classifier metrics; analyzing each general data point in the general dataset using the trained classifier to determine a score for each classifier metric among one or more classifier metrics associated with the corresponding general data point; generating a filtered general dataset by filtering the general dataset based on one or more filters, the one or more filters indicating threshold scores for corresponding classifier metrics; training a small language model with the filtered general dataset; generating a synthetic dataset for refining the small language model; and training the small language model with the synthetic dataset after training it with the filtered general dataset.

[0007] This synopsis is provided to introduce some concepts in a simplified form, which will be further described in the detailed description below. This synopsis is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0008] Refer to the following figures to describe non-restrictive and non-exhaustive examples.

[0009] Figure 1 A block diagram depicts an example of two-stage training of a small language model according to the examples of this disclosure; Figure 2 A block diagram depicts an example of an operating environment in which a language model generator can be implemented, according to an example of this disclosure; Figure 3A and Figure 3B A flowchart depicts an example method for generating a small language model according to the examples of this disclosure; Figure 4A and Figure 4B An overview of example generative machine learning models that can be used based on the examples in this disclosure is shown; Figure 5 This is a block diagram illustrating example physical components of a computing device that can be used to practice various aspects of the present disclosure; Figure 6 A simplified block diagram of a computing device that can be used to implement various aspects of this disclosure; and Figure 7 This is a simplified block diagram of a distributed computing system in which the various aspects of this disclosure can be implemented. Detailed Implementation

[0010] In the following detailed description, reference is made to the accompanying drawings, which form a part of the description, and specific embodiments or examples are illustrated therein. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from this disclosure. Embodiments may be practiced as methods, systems, or devices. Therefore, embodiments may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Consequently, the following detailed description should not be considered limiting, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0011] Over the past few years, Large Language Models (LLMs) have transformed the field of Natural Language Processing. More broadly, they promise to drive a paradigm shift in human-computer interaction. These advancements have profound economic implications and the potential to redefine the conceptual framework of artificial intelligence and even potentially understand itself. LLMs achieve high levels of capability using massive amounts of parameters and training data lexicons. Improvements from one generation of LLMs to the next currently appear to stem primarily from increased scale, with the most powerful models approaching trillions of parameters and trillions of training data lexicons. However, the costs of training, deploying, and maintaining such large models can be substantial. From a responsible AI perspective, the capabilities of large-scale models are increasingly being recognized, along with concerns about their controllability or manageability.

[0012] According to examples in this disclosure, a small language model (SLM) is generated via two-stage training to achieve capabilities similar to a large language model (LLM). A small language model is a generative machine learning model that, compared to an LLM, has fewer parameters and is trained with less training data. Specifically, the two-stage training comprises a first training phase and a second training phase. Specifically, the two-stage training includes a first training phase for teaching the small language model general knowledge and language understanding, and a second training phase for teaching the small language model logical reasoning and various niche skills. The first training phase involves generating a filtered general dataset from various sources (e.g., web sources) and training the small language model with the filtered general dataset. The second training phase involves generating a synthetic dataset using prompts and training the small language model with the synthetic dataset. By curating training data from two separate sources through two-stage training, the small language model is trained using data of a specific quality suitable for understanding common-sense reasoning and general world knowledge, as well as performing specific application and reasoning tasks (e.g., common-sense or logical reasoning).

[0013] Now for reference Figure 1A block diagram illustrating an example of two-stage training of a small language model according to the present disclosure is provided. The two-stage training of the small language model 100 includes a first training phase 120 and a second training phase 130. As further described below, by curating training data from two separate sources through the two training phases, the small language model is trained using data of a specific quality suitable for understanding common-sense reasoning and general world knowledge, as well as performing specific application and reasoning tasks (e.g., common-sense or logical reasoning).

[0014] In some embodiments, small language model training can be warm-started by combining trained weights with new initialized weights before two-stage training. For example, trained weights from an existing trained model can be copied into the new small language model. In some embodiments, tiling can be used to copy weights across different dimensions of the model to ensure that the weights follow the magnitude distribution of the previous small language model and that the structure of the weight matrix is ​​maintained. By doing so, a warm start allows for rapid initiation of training of the new model and provides knowledge transfer between the existing and new models, enabling higher levels of performance with less training required for the new model. This saves computational resources and improves generalization ability when training small language models.

[0015] The first training phase 120 involves generating a filtered general dataset and training a small language model using the filtered general dataset. To generate the filtered general dataset, a general dataset comprising multiple general data points is obtained from various sources (e.g., web sources). Once the general dataset is obtained, a subset representing the general dataset is selected, and an annotated version of the subset is generated based on one or more classifier metrics indicating the quality of the subset. For this purpose, a generative transformer is used to annotate the quality of the subset of the general dataset based on one or more classifier metrics. For example, the generative transformer can be a language model (e.g., a large language model) or any generative machine learning model capable of annotating data using specific attributes or features. One or more classifier metrics are specific attributes or features associated with a subset of the general dataset. For example, one or more classifier metrics include, but are not limited to, factual knowledge, everyday knowledge, scientific knowledge, human behavior, toxicity, integrity, obscenity, ambiguity, commonality, reasoning, promotional content, and / or unwanted content. In an illustrative embodiment, each general data point in the subset of the general dataset is annotated with a score for each of the one or more classifier metrics using a generative transformer.

[0016] When annotating a subset of the general dataset, the annotated subset is used to train a classifier. The trained classifier is configured to predict the quality of the data based on one or more classifier metrics. For example, the quality of the data can be represented by scores for each classifier metric. Once the classifier is trained, each general data point in the general dataset is analyzed to determine a score for each classifier metric associated with the corresponding general data point using the trained classifier. Based on the scores of the general data points, one or more filters are generated to filter the general dataset. Each filter indicates a threshold score for the corresponding classifier metric. In other words, one or more filters can be used to filter the general dataset to select general data with specific attributes or features for training a small language model.

[0017] Once one or more filters are generated, a filtered general dataset is produced by filtering the general dataset using these filters. The filtered general dataset is a subset of the general dataset that meets a predefined quality level. As mentioned above, each filter is associated with a corresponding classifier metric. In many ways, one or more filters are selected to obtain those general data with specific attributes or features used to train the small language model. The filtered general dataset is used to train the small language model in the first training phase 120.

[0018] The second training phase 130 involves generating a synthetic dataset and using it to train a small language model. The synthetic dataset is "textbook-style" data created to teach common-sense reasoning and universal knowledge (e.g., science, everyday activities, theories of mind, etc.). For example, specific topics can be selected to seed the generation of the synthetic data.

[0019] To this end, one or more deficient skills in the small language model are identified. It should be understood that the term "deficient skill" means any skill or topic that can enhance the capabilities of the small language model. Once one or more deficient skills are identified, one or more data formats for addressing these deficient skills are determined, and one or more prompts are obtained to create these data formats. Additionally, sources of randomization and diversity can be injected into the one or more prompts. Based on the one or more prompts, a synthetic dataset is generated using a generative transformer. For example, the generative transformer can be any generative machine learning model capable of generating data based on prompts (e.g., a large language model). In all respects, the synthetic dataset is designed to provide and enhance specific skills in the small language model. A filtered general dataset is used to train the small language model in the second training phase 130.

[0020] Figure 2A block diagram depicts an example of an operating environment 200 in which a language model generator can be implemented, according to an example of this disclosure. For this purpose, the operating environment 200 includes a server 230 and a computing device 220 communicatively coupled to the server 230 via a network 250. The network 150 can include any kind of computing network, including but not limited to wired or wireless local area networks (LANs), wired or wireless wide area networks (WANs), and / or the Internet. The server 230 can be, but is not limited to, a computer, notebook computer, laptop computer, mobile device, smartphone, tablet computer, portable device, or any other suitable computing device capable of executing a language model generator 240. Additionally, the computing device 220 associated with the user 210 includes a processor 222, memory 224, and a communication interface 226. The computing device 220 can be, but is not limited to, a computer, notebook computer, laptop computer, mobile device, smartphone, tablet computer, portable device, wearable device, or any other suitable computing device capable of communicating with the server 230.

[0021] Server 230 includes a language model generator 240 configured to generate small language models that are trained to understand common-sense reasoning and universal knowledge to perform reasoning tasks (e.g., common-sense or logical reasoning). To this end, language model generator 240 also includes a general dataset manager 242, a synthetic dataset manager 244, and a small language model manager 246.

[0022] A general dataset manager 242 is configured to generate a filtered general dataset to be used during the first training phase of a small language model. To this end, the general dataset manager 242 is configured to obtain general datasets from various sources (e.g., web sources). The general dataset comprises multiple general datasets. Once the general dataset is obtained, the general dataset manager 242 is configured to select a subset of the general dataset representing the general dataset and generate an annotated version of the subset based on one or more classifier metrics indicating the quality of the subset. For this purpose, a generative transformer is used to annotate the quality of the subset of the general dataset based on one or more classifier metrics. For example, the generative transformer can be a language model (e.g., a large language model) or any generative machine learning model capable of annotating data with specific attributes or features. One or more classifier metrics are specific attributes or features associated with a subset of the general dataset. For example, one or more classifier metrics include, but are not limited to, factual knowledge, everyday knowledge, scientific knowledge, human behavior, toxicity, integrity, obscenity, ambiguity, commonality, reasoning, promotional content, and / or unwanted content. In an illustrative embodiment, the general dataset manager 242 is configured to annotate each piece of general data in a subset of the general dataset with scores from one or more classifier metrics using a generative transformer. For example, the scores could be on a scale of one to ten, where one is the lowest and ten is the highest. For instance, specific pieces of general data in a subset of the dataset could be annotated as a score of 2 / 10 for toxicity level and 3 / 10 for obscenity level.

[0023] When annotating a subset of the general dataset, the general dataset manager 242 is configured to train a classifier using the annotated subset of the general dataset. The trained classifier is configured to predict the quality of the data based on one or more classifier metrics. For example, the quality of the data can be represented by scores for each classifier metric. Once the classifier is trained, the general dataset manager 242 is further configured to analyze each general data point in the general dataset. Specifically, the general dataset manager 242 is configured to analyze each general data point using the trained classifier to determine a score for each classifier metric associated with the corresponding general data point.

[0024] Based on the scores of the general data, the general dataset manager 242 is configured to generate one or more filters for filtering the general dataset. Each filter indicates a threshold score for a corresponding classifier metric. In other words, one or more filters can be used to filter the general dataset to select general data with specific attributes or features for training small language models. For example, a toxicity filter can be defined with a threshold score of 2 / 10 for content toxicity. In other words, when the toxicity filter is applied, any general data with content having a toxicity level higher than 2 will be filtered out, and only those general data with content having toxicity levels of 1 and 2 will be retained.

[0025] Once one or more filters are generated, the general dataset manager 242 is configured to generate a filtered general dataset. Specifically, the general dataset manager 242 is configured to generate a filtered general dataset by filtering the general dataset using one or more filters. The filtered general dataset is a subset of the general dataset that meets a predefined quality level. As mentioned above, each filter is associated with a corresponding classifier metric. In each respect, one or more filters are selected to obtain those general data with certain properties or features used for training small language models.

[0026] The synthetic dataset manager 244 is configured to generate synthetic datasets to be used during the second training phase of the small language model. The synthetic datasets are "textbook-style" data created to teach common-sense reasoning and universal knowledge (e.g., science, everyday activities, theories of mind, etc.). For example, specific topics can be selected to seed the generation of synthetic data.

[0027] To this end, the synthetic dataset manager 244 is configured to identify one or more deficient skills in the small language model. It should be understood that the term "deficient skill" means any skill or topic that can improve the capabilities of the small language model. The synthetic dataset manager 244 is also configured to determine one or more data formats to address one or more deficient skills and to obtain one or more hints to create one or more data formats. Additionally, the synthetic dataset manager 244 is configured to inject sources of randomization and diversity into one or more hints.

[0028] The synthetic dataset manager 244 is configured to generate synthetic datasets based on one or more prompts using a generative transformer. For example, the generative transformer can be any generative machine learning model capable of generating data based on prompts (e.g., a large language model). In many ways, the synthetic dataset is designed to provide and enhance specific skills in a small language model. To this end, the prompt-based generative transformer synthesizes textbook-style content about the skills that need to be enhanced. For example, a detailed list of topics, subtopics, keywords, and content related to those skills, filtered from other sources such as web pages, can be provided and used as the seed for the generative transformer. The seed is randomly shuffled and combined with other sources of randomness, such as styles, use cases, user roles, and various ways of generating presented content. All of these can be combined in different random combinations and input into the prompts of the generative transformer. The output of the generative transformer can then be integrated to generate the synthetic dataset.

[0029] Small Language Model Manager 246 is configured to perform two-stage training to train a small language model. Specifically, Small Language Model Manager 246 is configured to train the small language model with a filtered general dataset during a first training stage and with a synthetic dataset during a second training stage. By curating the training data for the small language model, Small Language Model Manager 246 is configured to train the small language model using data that meets specific quality requirements suitable for a specific application. For example, Small Language Model Manager 246 can reduce toxic and biased content in both the general and synthetic datasets. In other words, two separate training data sources are used in the two-stage training to train the small language model to understand common-sense reasoning and universal knowledge in order to perform reasoning tasks (e.g., common-sense or logical reasoning).

[0030] Now for reference Figure 3A and Figure 3B This disclosure provides a method 300 for generating a small language model via two-stage training, according to examples. The general sequence of steps for method 300 is as follows: Figure 3A and Figure 3B As shown in the diagram. Typically, method 300 begins at 302 and ends at 332. Method 300 may include more or fewer steps, or may be combined with... Figure 3A and Figure 3BThe steps shown are arranged differently in order. In illustrative terms, method 300 is performed by a computing device (e.g., server 230). However, it should be understood that one or more steps of method 300 may be performed by another device (e.g., computing device 220). Specifically, in some aspects, method 300 may be performed by a language model generator (e.g., 240) executed on server 230. Server 230 may be, but is not limited to, a computer, laptop, mobile device, smartphone, tablet, portable device, or any other suitable computing device capable of executing language model generator 240 to generate and train small language models. Computing device 220 may be, but is not limited to, a computer, laptop, mobile device, smartphone, tablet, portable device, wearable device, or any other suitable computing device capable of communicating with server 230.

[0031] Method 300 can be executed as a set of computer-executable instructions executed and encoded by a computer system or stored on a computer-readable medium. For example, a set of computer-executable instructions stored on a storage device can cause a computing device (e.g., server 230) to execute method 300. Furthermore, method 300 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), neural processing unit (NPU), or other hardware device. References will be incorporated herein by reference. Figure 1 , Figure 2 and Figure 4 to Figure 7 Method 300 is explained by describing the system, components, modules, software, data structures, user interface, etc.

[0032] The method 300 for generating a small language model includes a first training phase (e.g., operations 304 to 316) and a second training phase (e.g., operations 318 to 330). However, it should be understood that, as described above... Figure 1 As described above, the language model generator 240 can warm-start the training of the small language model before the training phase of method 300.

[0033] Method 300 begins with operation 302, from which the process can proceed to 304. At operation 304, the language model generator 240 obtains a common dataset from various sources (e.g., web sources). The common dataset includes multiple general data sets.

[0034] At operation 306, the language model generator 240 annotates a subset of the general dataset based on one or more classifier metrics that indicate the quality of the subset. For this purpose, a subset of the general dataset is selected to represent the general dataset. Once the subset of the general dataset is selected, a generative transformer is used to annotate the quality of the subset of the general dataset based on one or more classifier metrics. For example, the generative transformer can be a language model (e.g., a large language model) or any generative machine learning model capable of annotating data with specific attributes or features. The one or more classifier metrics are specific attributes or features associated with a subset of the general dataset. For example, the one or more classifier metrics include, but are not limited to, factual knowledge, everyday knowledge, scientific knowledge, human behavior, toxicity, integrity, obscenity, ambiguity, commonality, reasoning, promotional content, and / or unwanted content. In an illustrative embodiment, the language model generator 240 uses the generative transformer to annotate each general data point of the subset of the general dataset with a score from each of the one or more classifier metrics. For example, the score can be a scale from one to ten, where one is the lowest and ten is the highest. For example, specific general data from a subset of the dataset can be annotated as toxicity level 2 / 10 and obscenity level 3 / 10.

[0035] When annotating a subset of the general dataset, the language model generator 240 uses the annotated subset of the general dataset, as indicated in operation 308, to train a classifier. The trained classifier is configured to predict the quality of the data based on one or more classifier metrics. For example, the quality of the data can be represented by scores for each classifier metric.

[0036] After training the classifier, the language model generator 240 analyzes each general data point in the general dataset, as indicated in operation 310. Specifically, the language model generator 240 uses the trained classifier to analyze each general data point to determine a score for each classifier metric associated with the corresponding general data point.

[0037] Based on the scores of the general data, the language model generator 240 generates one or more filters for filtering the general dataset, as indicated in operation 312. Each filter indicates a threshold score for the corresponding classifier metric. In other words, one or more filters can be used to filter the general dataset to select general data with specific attributes or features for training small language models. For example, a toxicity filter can be defined with a threshold score of 2 / 10 for content toxicity. In other words, when the toxicity filter is applied, any general data with content having a toxicity level higher than 2 will be filtered out, and only those general data with content having toxicity levels of 1 and 2 will be retained.

[0038] Once one or more filters are generated, the language model generator 240 generates a filtered general dataset, as indicated in operation 314. Specifically, the language model generator 240 generates a filtered general dataset by filtering the general dataset using one or more filters. The filtered general dataset is a subset of the general dataset that meets a predefined quality level. As mentioned above, each filter is associated with a corresponding classifier metric. In many ways, one or more filters are selected to obtain those general data with specific attributes or features used to train a small language model.

[0039] At operation 316, the language model generator 240 trains a small language model using a filtered general dataset. By curating the training data for the small language model, it utilizes application-specific data that meets certain quality requirements. For example, the language model generator 240 can reduce toxic and biased content in the general dataset. Once the first training phase is completed, the small language model possesses a general understanding of natural language.

[0040] After the first training phase, method 300 proceeds to the second training phase (e.g., operations 318-330). However, it should be understood that one or more operations from the second training phase used to generate the synthetic dataset (e.g., operations 318-328) can be performed in parallel with one or more operations from the first training phase used to generate the filtered general dataset (e.g., operations 304-314).

[0041] At operation 318, the language model generator 240 generates a synthetic dataset for refining the small language model. This synthetic dataset is "textbook-style" data created to teach common-sense reasoning and universal knowledge (e.g., science, daily activities, theories of mind, etc.). For example, a specific topic can be selected to seed the generation of the synthetic data.

[0042] Therefore, at operation 320, the language model generator 240 identifies one or more deficient skills in the small language model. It should be understood that the term "deficient skill" means any skill or topic that can improve the capabilities of the small language model.

[0043] At operation 322, the language model generator 240 determines one or more data formats to address one or more deficient skills. For example, the data format could be text, graphics, and / or images. At operation 324, the language model generator 240 receives one or more prompts to create one or more data formats. At operation 326, sources of randomization and diversity are injected into one or more prompts.

[0044] As described above, the generative transformer prompts to synthesize textbook-style content about the deficient skills that need improvement. For example, a detailed list of topics, subtopics, keywords, and content related to those skills, filtered from other sources such as web pages, can be provided and used as a seed for the generative transformer. The seed is randomly shuffled and combined with other sources of randomness, including styles, use cases, user roles, and various ways of generating presented content. All of these can be combined in different random combinations and input into the prompts of the generative transformer.

[0045] At operation 328, the language model generator 240 uses a generative transformer to generate a synthetic dataset based on one or more prompts. For example, the generative transformer can be any generative machine learning model capable of generating data based on prompts (e.g., a large language model). In many ways, the synthetic dataset is designed to provide and enhance specific skills in a small language model.

[0046] Subsequently, at operation 320, the language model generator 240 trains the small language model using a synthetic dataset. Once the small language model has undergone two-stage training, it is adapted to understand common-sense reasoning and universal knowledge to perform reasoning tasks (e.g., common-sense or logical reasoning).

[0047] Figure 4A and Figure 4B An overview of example generative machine learning models that can be used based on the aspects described in this article is shown. First, refer to... Figure 4A Conceptual diagram 400 depicts an overview of a pre-trained generative model package 404 according to the aspects described herein, which processes input 402 to generate model output for capturing and generatively transforming content items from generative model output 406.

[0048] In the example, the generative model package 404 is pre-trained on a variety of inputs (e.g., various human languages, various programming languages, and / or various content types) and therefore does not require fine-tuning or training for a specific scenario. Instead, the generative model package 404 can be pre-trained in a more general way, such that the input 402 includes a cue that is generated, selected, or otherwise curated to cause the generative model package 404 to produce a specific generative model output 406. It should be understood that the input 402 and the generative model output 406 can each include any type of content from various content types, including but not limited to text output, image output, audio output, video output, programming output, and / or binary output. In the example, the input 402 and the generative model output 406 can have different content types, which may occur when the generative model package 404 includes a generative multimodal machine learning model.

[0049] Therefore, generative model package 404 can be used in any scenario across a wide range of scenarios, and furthermore, different generative model packages can be used instead of generative model package 404 without substantially modifying other related aspects (e.g., similar to those discussed in this paper). Figure 1 , Figure 2 , Figure 3A and Figure 3B (Those aspects described). Therefore, the generative model package 404 operates as a tool for performing machine learning processing, wherein certain inputs 402 to the generative model package 404 are generated programmatically or otherwise determined, thereby enabling the generative model package 404 to produce model outputs 406, which can then be used for further processing.

[0050] Generative model package 404 can be provided or otherwise used according to any of the various paradigms. For example, generative model package 404 can be used on computing devices (e.g., Figure 1 The generative model package 404 may be used locally (e.g., by communication platform 102) or remotely (e.g., by client computing devices 104 and / or 106). In other examples, aspects of the generative model package 404 may be distributed across multiple computing devices. In some cases, the generative model package 404 may be accessible via an API, such as from the operating system of the computing device and / or from machine learning services.

[0051] Referring now to aspects of the generative model package 404, the generative model package 404 includes input tokenization 408, input embedding 410, model layer 412, output layer 414, and output decoding 416. In the example, input tokenization 408 processes input 402 to generate input embedding 410, which includes a sequence of symbol representations corresponding to input 402. Therefore, input embedding 410 is processed by model layer 412, output layer 414, and output decoding 416 to produce model output 406. Figure 4B The example architecture corresponding to Generative Model Package 404 is depicted below, which will be discussed in further detail. Even so, it should be understood that the architecture shown and described herein should not be considered limiting, and any of the various other architectures may be used in other examples.

[0052] Figure 4B This is a conceptual diagram depicting an example architecture 450 of a pre-trained generative machine learning model that can be used according to the aspects described herein. As mentioned above, any architecture and corresponding ML model from a variety of alternative architectures can be used in other examples without departing from the aspects described herein.

[0053] As shown in the figure, architecture 450 processes input 402 to produce generative model output 406, which are discussed above regarding... Figure 4ADiscussion. Architecture 450 is depicted as a transformer model comprising encoder 452 and decoder 454. Encoder 452 processes input embedding 458 (its aspects can be similar to...). Figure 4A The input embedding (410) includes a sequence of symbolic representations corresponding to input 456. In the example, input 456 includes input 402. Such aspects can be similar to those described above regarding... Figure 1 The aspects discussed in the language model generator 240, for example, by respectively in Figure 1 , 3A The various aspects of implementing methods 100 and / or 300 in 3B and 3B.

[0054] Furthermore, positional encoding 460 can incorporate information about the relative and / or absolute positions of the lexical units in the input embedding 458. Similarly, the output embedding 474 includes a sequence of symbolic representations corresponding to the output 472, and positional encoding 476 can similarly incorporate information about the relative and / or absolute positions of the lexical units in the output embedding 474.

[0055] As shown in the figure, encoder 452 includes example layer 470. It should be understood that any number of such layers can be used, and the architecture depicted is simplified for illustrative purposes. Example layer 470 includes two sub-layers: a multi-head attention layer 462 and a feedforward layer 466. In the example, residual connections are included around each layer 462 and layer 466, and normalization layers 464 and 468 are included after layers 462 and layer 466, respectively.

[0056] Decoder 454 includes example layer 490. Similar to encoder 452, any number of such layers may be used in other instances, and the depicted architecture of decoder 454 is simplified for illustrative purposes. As shown, example layer 490 includes three sublayers: a masked multi-head attention layer 478, a multi-head attention layer 482, and a feedforward layer 486. Aspects of multi-head attention layer 482 and feedforward layer 486 may be similar to those discussed above regarding multi-head attention layer 462 and feedforward layer 466, respectively. Additionally, masked multi-head attention layer 478 performs multi-head attention on the output of encoder 452 (e.g., output 472). In the example, masked multi-head attention layer 478 prevents each position from focusing on subsequent positions. This masking, combined with embedding offsets (e.g., offsetting by one position, as shown in multi-head attention layer 482), ensures that the prediction for a given position depends on the known outputs of one or more positions smaller than the given position. As shown in the figure, residual connections are also included around layers 478, 482 and 486, and normalized layers 480, 484 and 488 are included after layers 478, 482 and 486, respectively.

[0057] Multi-head attention layers 462, 478, and 482 can each use a set of linear projections to linearly project the query, key, and value onto their corresponding dimensions. Each linear projection can be processed using an attention function (e.g., dot product attention or summation attention), thus producing an n-dimensional output value for each linear projection. The resulting values ​​can be concatenated and projected again, such that subsequent... Figure 4B The values ​​are processed as shown (e.g., by the corresponding normalization layers 464, 480, or 484).

[0058] Feedforward layers 466 and 486 can each be fully connected feedforward networks, applicable to each location. In the example, feedforward layers 466 and 486 each include multiple linear transformations with rectified linear unit activation between them. In the example, each linear transformation is identical at different locations, while different parameters can be used compared to other linear transformations of the feedforward network.

[0059] Furthermore, aspects of the linear transformation 492 can be similar to the linear transformations discussed above regarding the multi-head attention layers 462, 478, and 482, and the feedforward layers 466 and 486. Softmax 494 can further transform the output of the linear transformation 492 into the predicted next lexical probability, as indicated by the output probability 496. It should be understood that the architecture shown is provided as an example, and in other examples, any of various other model architectures can be used based on the disclosed aspects.

[0060] Therefore, according to the aspects described herein, the output probability 496 can thus form the model output 406, such that the output of the generative ML model (e.g., which may therefore include generative content) is used as, for example, an opening or closing segment of one or more communication participants according to the aspects described herein.

[0061] Figures 5-7 The associated description provides a discussion of various operating environments in which the aspects of this disclosure can be practiced. However, regarding Figures 5-7 The devices and systems shown and discussed are for illustrative purposes only and are not intended to limit the wide range of computing device configurations that may be used to practice the aspects of this disclosure described herein.

[0062] Figure 5 This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 500 that can be used to practice various aspects of this disclosure. The computing device components described below can be adapted to the aforementioned computing device, including one or more devices associated with the communication platform 102, and those mentioned above. Figure 1The client computing devices 104 and / or 106 are discussed. In a basic configuration, computing device 500 may include at least one processing unit 502 and system memory 504. Depending on the configuration and type of the computing device, system memory 504 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memory.

[0063] System memory 504 may include an operating system 505 and one or more program modules 506 suitable for running software application 520, such as one or more components supported by the system described herein. As an example, system memory 504 may include a language model generator 521, which further includes a general dataset manager 522, a synthetic dataset manager 523, and / or a small language model manager 524. Operating system 505 may, for example, be adapted to control the operation of computing device 500.

[0064] Furthermore, embodiments of this disclosure can be practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. This basic configuration is... Figure 5 The components within the dashed line 508 are shown. The computing device 500 may have additional features or functions. For example, the computing device 500 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage... Figure 5 The image is shown by removable storage device 509 and non-removable storage device 510.

[0065] As described above, multiple program modules and data files can be stored in system memory 504. When executed on processing unit 502, program module 506 (e.g., application 520) can perform processes including, but not limited to, the aspects described herein. Other program modules that can be used according to aspects of this disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, PowerPoint presentation applications, drawing or computer-aided applications, etc.

[0066] Furthermore, embodiments of this disclosure can be practiced on circuits including discrete electronic components, packages containing logic gates or integrated electronic chips, circuits utilizing microprocessors, or single chips containing electronic components or microprocessors. For example, embodiments of this disclosure can be practiced via a system-on-a-chip (SOC), wherein... Figure 5Each or more of the components shown can be integrated onto a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “burned in”) as a single integrated circuit onto a chip substrate. When operating via the SoC, the capabilities described herein regarding the client switching protocol can be operated via dedicated logic integrated with other components of the computing device 500 on the single integrated circuit (chip). Embodiments of this disclosure can also be practiced using other techniques capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, embodiments of this disclosure can be practiced within a general-purpose computer or in any other circuit or system.

[0067] The computing device 500 may also have one or more input devices 512, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. It may also include output devices(s) 514, such as a display, speaker, printer, etc. The above devices are examples, and other devices may be used. The computing device 500 may include one or more communication connections 516 that allow communication with other computing devices 550. Examples of suitable communication connections 516 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0068] As used herein, the term computer-readable medium can include computer storage media. Computer storage media can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 504, removable storage device 509, and non-removable storage device 510 are examples of computer storage media (e.g., memory storage). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and can be accessed by computing device 500. Any such computer storage medium may be part of computing device 500. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0069] Communication media can be embodied in computer-readable instructions, data structures, program modules, or other data in modulated data signals (such as carrier waves or other transmission mechanisms), and include any information transmission medium. The term "modulated data signal" can describe a signal having one or more characteristics set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0070] Figure 6 The illustration shows a system 600, which can be, for example, a mobile computing device such as a mobile phone, smartphone, wearable computer (such as a smartwatch), tablet computer, laptop computer, etc., using which embodiments of the present disclosure can be practiced. In one embodiment, the system 600 is implemented as a "smartphone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 600 is integrated as a computing device, such as integrating a personal digital assistant (PDA) and a wireless phone.

[0071] In a basic configuration, such a mobile computing device is a handheld computer with both input and output elements. System 600 typically includes a display 605 and one or more input buttons that allow the user to input information into system 600. The display 605 can also be used as an input device (e.g., a touchscreen display).

[0072] If included, optional side input elements allow for additional user input. For example, a side input element could be a rotary switch, a button, or any other type of manual input element. Alternatively, system 600 can incorporate more or fewer input elements. For example, in some embodiments, display 605 may not be a touchscreen. In another example, an optional keyboard 635 may also be included, which could be a physical keyboard or a “soft” keyboard generated on a touchscreen display.

[0073] In various embodiments, output elements include a display 605 for displaying a graphical user interface (GUI), a visual indicator (e.g., a light-emitting diode 620), and / or an audio transducer 625 (e.g., a speaker). In some aspects, a vibration transducer is included to provide tactile feedback to the user. In another aspect, input and / or output ports are included, such as audio inputs (e.g., a microphone jack), audio outputs (e.g., a headphone jack), and video outputs (e.g., an HDMI port) for sending signals to or receiving signals from external devices.

[0074] One or more applications 666 may be loaded into memory 662 and run on or associated with operating system 664. Examples of applications include telephone dialers, email programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, etc. System 600 also includes a non-volatile storage area 668 within memory 662. The non-volatile storage area 668 may be used to store persistent information that should not be lost if system 600 is powered off. Applications 666 may use and store information in the non-volatile storage area 668, such as emails or other messages used by email applications. A synchronization application (not shown) also resides on system 600 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep the information stored in the non-volatile storage area 668 synchronized with the corresponding information stored on the host computer. It should be understood that other applications may be loaded into memory 662 and run on system 600 as described herein.

[0075] System 600 has a power supply 670, which can be implemented as one or more batteries. The power supply device 670 may also include an external power source, such as an AC adapter or power docking bracket for replenishing or recharging the batteries.

[0076] System 600 may also include a radio interface layer 672 that performs functions of transmitting and receiving radio frequency communications. Radio interface layer 672 facilitates wireless connectivity between system 600 and the "outside world" via a communications operator or service provider. Transmissions to and from radio interface layer 672 are conducted under the control of operating system 664. In other words, communications received by radio interface layer 672 can be propagated to application 666 via operating system 664, and vice versa.

[0077] A visual indicator 620 can be used to provide visual notifications, and / or an audio interface 674 can be used to generate auditory notifications via an audio transducer 625. In the illustrated embodiment, the visual indicator 620 is a light-emitting diode (LED), and the audio transducer 625 is a speaker. These devices can be directly coupled to a power supply device 670 such that when activated, they remain on for the duration specified by the notification mechanism, even if the processor 660 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 674 is used to provide and receive audible signals to and from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 674 can also be coupled to a microphone to receive audible input, such as to facilitate telephone conversations. According to embodiments of this disclosure, the microphone can also be used as an audio sensor to facilitate control of notifications, as described below. The system 600 may also include a video interface 676, which enables the operation of the onboard camera 630 to record still images, video streams, etc.

[0078] It should be understood that system 600 may have additional features or functions. For example, system 600 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage... Figure 6 The non-volatile storage region 668 is shown in the middle.

[0079] As described above, data / information generated, captured, and stored via system 600 can be stored locally, or the data can be stored on any number of storage media accessible by the devices via radio interface layer 672 or via a wired connection between system 600 and a separate computing device associated with system 600 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via radio interface layer 672 or via a distributed computing network. Similarly, such data / information can be easily transferred between computing devices for storage and use according to any of various data / information transmission and storage means, including email and collaborative data / information sharing systems.

[0080] Figure 7 One aspect of the architecture of a system for processing data received at a computing system from a remote source (such as a personal computer 704, a tablet computing device 706, or a mobile computing device 708), as described above, is illustrated. The content displayed at server device 702 can be stored in different communication channels or other storage types. For example, various documents can be stored using a directory service 724, a portal website 725, an email service 726, an instant messaging repository 728, or a social networking site 730.

[0081] Application 720 (e.g., similar to application 520) can be adopted by a client communicating with server device 702. Additionally or alternatively, server device 702 may employ a language model generator 791, a general dataset manager 792, a synthetic dataset manager 793, and / or a small language model manager 794. Server device 702 can provide data to and from client computing devices such as personal computer 704, tablet computing device 706, and / or mobile computing device 708 (e.g., smartphone) via network 715. As an example, the aforementioned computer system can be embodied in personal computer 704, tablet computing device 706, and / or mobile computing device 708 (e.g., smartphone). In addition to receiving graphical data that can be preprocessed at the graphics initiating system or post-processed at the receiving computing system, any of these examples of computing devices can obtain content from storage 716.

[0082] It should be understood that the aspects and functions described herein can operate on distributed systems (e.g., cloud-based computing systems), where application functions, memory, data storage and retrieval, and various processing functions can operate remotely to each other via distributed computing networks (such as the Internet or intranets). Various types of user interfaces and information can be displayed via onboard computing device displays or via remote display units associated with one or more computing devices. For example, various types of user interfaces and information can be displayed and interacted with on a wall surface on which various types of user interfaces and information are projected. Interaction with multiple computing systems on which embodiments of the invention can be practiced includes keystroke input, touchscreen input, voice or other audio input, gesture input, wherein the associated computing device is equipped with detection (e.g., camera) functions for capturing and interpreting user gestures used to control the functions of the computing device, etc.

[0083] For example, the foregoing references to block diagrams and / or operational illustrations of methods, systems, and computer program products according to various aspects of this disclosure describe aspects of the disclosure. Functions / actions mentioned in the blocks may not occur in the order shown in any flowchart. For example, depending on the functions / actions involved, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order.

[0084] The description and illustrations of one or more aspects provided in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey ownership and enable others to make and use the claimed aspects of this disclosure. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Various features (structural and methodological features) are intended to be selectively included or omitted, whether shown and described in combination or separately, to produce aspects having a particular set of features. Given the descriptions and illustrations provided in this application, those skilled in the art can contemplate variations, modifications, and alternatives falling within the spirit of the broader aspects of the overall inventive concept embodied in this application without departing from the broader scope of the claimed disclosure.

[0085] Furthermore, the aspects and functions described herein can operate on distributed systems (e.g., cloud-based computing systems), where application functions, memory, data storage and retrieval, and various processing functions can remotely operate on each other via distributed computing networks (such as the Internet or intranets). Various types of user interfaces and information can be displayed via onboard computing device displays or via remote display units associated with one or more computing devices. For example, various types of user interfaces and information can be displayed and interacted with on a wall surface on which various types of user interfaces and information are projected. Interaction with multiple computing systems on which aspects of this disclosure can be practiced includes keystroke input, touchscreen input, voice or other audio input, gesture input, wherein the associated computing device is equipped with detection (e.g., camera) functions for capturing and interpreting user gestures used to control the functions of the computing device, etc.

[0086] The phrases “at least one,” “one or more,” “or,” and “and / or” are open-ended expressions that possess both conjunction and disjunctive properties in their use. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” “A, B, and / or C,” and “A, B, or C” indicates a single A, a single B, a single C, A and B together, A and C together, B and C together, or A, B, and C together.

[0087] The term "a" or "an" entity refers to one or more of the entities. Therefore, the terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" are used interchangeably.

[0088] As used herein, the term "automatic" and its variations refer to any process or operation that is generally continuous or semi-continuous, performed without substantial human input. However, a process or operation can be automatic even if its execution uses substantial or non-substantial human input, if input is received prior to its execution. Human input is considered substantial if it influences how the process or operation will be performed. Human input consenting to the execution of a process or operation is not considered "substantial."

[0089] Any steps, functions, and operations discussed in this article can be performed continuously and automatically.

[0090] Example systems and methods of this disclosure have been described with respect to computing devices. However, to avoid unnecessarily obscuring this disclosure, several known structures and devices have been omitted from the foregoing description. Such omissions should not be construed as limiting. Specific details have been set forth to provide an understanding of this disclosure. However, it should be understood that this disclosure can be practiced in various ways beyond the specific details set forth herein.

[0091] Furthermore, while the examples shown herein illustrate various components of a co-located system, some components of the system may be located remotely in distant parts of a distributed network (such as a LAN and / or the Internet), or within a dedicated system. Therefore, it should be understood that system components may be combined into one or more devices, such as servers, communication equipment, or co-located on specific nodes of a distributed network, such as analog and / or digital telecommunications networks, packet-switched networks, or circuit-switched networks. As will be understood from the foregoing description, and for computational efficiency reasons, system components may be placed anywhere within the distributed network of components without affecting the operation of the system.

[0092] Furthermore, it should be understood that the various links of the connecting element can be wired or wireless links, or any combination thereof, or any other known or later-developed element(s) capable of providing data to and / or transmitting data from the connecting element. These wired or wireless links can also be secure links and capable of transmitting encrypted information. For example, the transmission medium used as the link can be any suitable carrier for electrical signals, including coaxial cables, copper wires, and optical fibers, and can take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0093] Although flowcharts have been discussed and explained regarding specific event sequences, it should be understood that the sequence can be changed, added to, or omitted without substantially affecting the operation of the disclosed configuration and aspects.

[0094] Several variations and modifications of this disclosure may be used. Some features of this disclosure may be provided without providing others.

[0095] In another configuration, the systems and methods of this disclosure may be implemented in combination with a dedicated computer, a programmable microprocessor or microcontroller and (multiple) peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, hardwired electronic or logic circuitry (such as discrete component circuitry), a programmable logic device or gate array (such as a PLD, PLA, FPGA, PAL), a dedicated computer, any similar means, etc. Generally, any (multiple) means or apparatuses capable of implementing the methods described herein can be used to implement various aspects of this disclosure. Example hardware that may be used in this disclosure includes computers, handheld devices, telephones (e.g., cellular, internet-enabled, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage devices, input devices, and output devices. Furthermore, alternative software implementations, including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing, may also be constructed to implement the methods described herein.

[0096] In another configuration, the disclosed method can be readily implemented using software that provides portable source code for object-oriented or object-based software development environments usable on various computer or workstation platforms. Alternatively, the disclosed system can be implemented partially or entirely in hardware using standard logic circuitry or VLSI designs. Whether to implement the system according to this disclosure using software or hardware depends on the system's speed and / or efficiency requirements, specific functions, and the particular software or hardware system or microprocessor or microcomputer system utilized.

[0097] In another configuration, the disclosed methods can be implemented in part in software, which can be stored on a storage medium and executed on a programmed general-purpose computer in cooperation with a controller and memory, a dedicated computer, a microprocessor, etc. In these cases, the systems and methods of this disclosure can be implemented as programs embedded in a personal computer, such as applets, JAVA®, or CGI scripts; as resources residing on a server or computer workstation; or as routines embedded in a dedicated measurement system, system components, etc. The system can also be implemented by physically integrating the system and / or methods into a software and / or hardware system.

[0098] As described herein, this disclosure is not limited to standards and protocols. Other similar standards and protocols not mentioned herein exist and are included in this disclosure. Furthermore, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are periodically replaced by faster or more efficient equivalents having substantially the same function. Such replacement standards and protocols having the same function are considered equivalents included in this disclosure.

[0099] According to at least one example of this disclosure, a method for generating a small language model is provided. The method may include: obtaining a general dataset comprising multiple general data sets; annotating a subset of the general dataset based on one or more classifier metrics indicating the quality of the general dataset, the subset representing the general dataset; training a classifier based on the annotated subset of the general dataset and one or more classifier metrics; analyzing each general data set in the general dataset using the trained classifier to determine a score for each classifier metric among one or more classifier metrics associated with the corresponding general data set; generating a filtered general dataset by filtering the general dataset based on one or more filters indicating threshold scores for corresponding classifier metrics; training a small language model with the filtered general dataset; generating a synthetic dataset for refining the small language model; and training the small language model with the synthetic dataset after training it with the filtered general dataset.

[0100] According to at least one aspect of the above method, the method may include: wherein each general data point of the general dataset is associated with a score of each classifier metric in one or more classifier metrics.

[0101] According to at least one aspect of the above method, the method may include: one or more classifier metrics including factual knowledge, everyday knowledge, scientific knowledge, human behavior, toxicity, integrity, obscenity, ambiguity, commonality, reasoning, promotional content, and / or unwanted content.

[0102] According to at least one aspect of the above method, the method may include: generating a filtered general dataset by filtering a general dataset based on one or more filters includes: generating one or more filters for one or more classifier metrics, each filter corresponding to a corresponding classifier metric and indicating a threshold score assigned to the corresponding classifier metric; and filtering the general dataset based on one or more filters.

[0103] According to at least one aspect of the above method, the method may include: wherein generating a synthetic dataset for refining a small language model includes: identifying one or more deficient skills in the small language model; determining one or more data formats to address the one or more deficient skills; generating one or more cues for generating the one or more data formats; injecting sources of randomization and diversity into the one or more cues; and generating a synthetic dataset based on the one or more cues using a generative transformer, the synthetic dataset including the one or more data formats.

[0104] According to at least one aspect of the above method, the method may include: one or more of the defective skills include any skill or topic for improving the capabilities of small language models.

[0105] According to at least one aspect of the above method, the method may include: wherein the generative transformer is a multimodal large language model.

[0106] According to at least one aspect of the above method, the method may further include: performing a warm start by copying weights from an existing trained model to the small language model before training the small language model with a filtered general dataset.

[0107] According to at least one example of this disclosure, a computing device for generating a small language model is provided. The computing device may include a processor and a memory having a plurality of instructions stored thereon, which, when executed by the processor, cause the computing device to generate a small language model. The method includes: obtaining a general dataset comprising a plurality of general data; annotating a subset of the general dataset based on one or more classifier metrics indicating the quality of the general dataset, the subset representing the general dataset; training a classifier based on the annotated subset of the general dataset and the one or more classifier metrics; analyzing each general data point in the general dataset using the trained classifier to determine a score for each classifier metric among one or more classifier metrics associated with the corresponding general data point; generating a filtered general dataset by filtering the general dataset based on one or more filters, the one or more filters indicating threshold scores for corresponding classifier metrics; training a small language model with the filtered general dataset; generating a synthetic dataset for refining the small language model; and training the small language model with the synthetic dataset after training it with the filtered general dataset.

[0108] According to at least one aspect of the computing device described above, the computing device may include: wherein each general data point of the general dataset is associated with a score of each classifier metric in one or more classifier metrics.

[0109] According to at least one aspect of the aforementioned computing device, the computing device may include: one or more classifier metrics including factual knowledge, everyday knowledge, scientific knowledge, human behavior, toxicity, integrity, obscenity, ambiguity, commonality, reasoning, promotional content, and / or unwanted content.

[0110] According to at least one aspect of the computing device described above, the computing device may include: generating a filtered general dataset by filtering a general dataset based on one or more filters includes: generating one or more filters for one or more classifier metrics, each filter corresponding to a corresponding classifier metric and indicating a threshold score assigned to the corresponding classifier metric; and filtering the general dataset based on one or more filters.

[0111] According to at least one aspect of the computing device described above, the computing device may include: wherein generating a synthetic dataset for refining a small language model includes: identifying one or more deficient skills in the small language model; determining one or more data formats to address one or more deficient skills; generating one or more cues for generating the one or more data formats; injecting sources of randomization and diversity into the one or more cues; and using a generative transformer to generate the synthetic dataset based on the one or more cues, the synthetic dataset including one or more data formats.

[0112] According to at least one aspect of the aforementioned computing device, the computing device may include one or more of the deficient skills, including any skills or topics for improving the capabilities of small language models.

[0113] According to at least one aspect of the aforementioned computing device, the computing device may include, when executed, a plurality of instructions that also cause the computing device to: perform a warm start by copying weights from an existing trained model into a small language model before training the small language model with a filtered general dataset.

[0114] According to at least one example of this disclosure, a computer storage medium is provided. The computer storage medium stores computer-executable instructions that, when executed, cause at least one processor to perform operations. The operations include: obtaining a general dataset comprising multiple general data; annotating a subset of the general dataset based on one or more classifier metrics indicating the quality of the general dataset, the subset representing the general dataset; training a classifier based on the annotated subset of the general dataset and the one or more classifier metrics; analyzing each general data point in the general dataset using the trained classifier to determine a score for each classifier metric among one or more classifier metrics associated with the corresponding general data point; generating a filtered general dataset by filtering the general dataset based on one or more filters, the one or more filters indicating threshold scores for corresponding classifier metrics; training a small language model with the filtered general dataset; generating a synthetic dataset for refining the small language model; and training the small language model with the synthetic dataset after training it with the filtered general dataset.

[0115] According to at least one aspect of the aforementioned computer storage medium, instructions can cause a computing device to perform a method in which each general data point of a general dataset is associated with a score of each classifier metric in one or more classifier metrics.

[0116] According to at least one aspect of the aforementioned computer storage medium, the operation may further include: wherein generating a filtered general dataset by filtering a general dataset based on one or more filters comprises: generating one or more filters for one or more classifier metrics, each filter corresponding to a corresponding classifier metric and indicating a threshold score assigned to the corresponding classifier metric; and filtering the general dataset based on one or more filters.

[0117] According to at least one aspect of the aforementioned computer storage medium, the operation may further include: wherein generating a synthetic dataset for refining a small language model comprises: identifying one or more deficient skills in the small language model; determining one or more data formats to address one or more deficient skills; generating one or more cues for generating the one or more data formats; injecting sources of randomization and diversity into the one or more cues; and generating a synthetic dataset based on the one or more cues using a generative transformer, the synthetic dataset comprising one or more data formats, wherein the one or more deficient skills comprise any skills or topics for improving the capabilities of the small language model.

[0118] According to at least one aspect of the aforementioned computer storage medium, the operation may further include: performing a warm start by copying weights from an existing trained model to the small language model before training the small language model with a filtered general dataset.

[0119] For example, the foregoing references to block diagrams and / or operational illustrations of methods, systems, and computer program products according to various aspects of this disclosure describe aspects of the disclosure. Functions / actions mentioned in the blocks may not occur in the order shown in any flowchart. For example, depending on the functions / actions involved, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order.

[0120] The description and illustrations of one or more aspects provided in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey ownership and enable others to make and use the claimed aspects of this disclosure. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Various features (structural and methodological features) are intended to be selectively included or omitted, whether shown and described in combination or separately, to produce embodiments with a particular set of features. Having provided the description and illustrations of this application, those skilled in the art can contemplate variations, modifications, and alternatives falling within the spirit of the broader aspects of the overall inventive concept embodied in this application without departing from the broader scope of the claimed disclosure.

Claims

1. A method for generating small language models, the method comprising: Obtain a general dataset, which includes multiple general datasets; A subset of the general dataset is annotated based on one or more classifier metrics that indicate the quality of the general dataset, the subset representing the general dataset; The classifier is trained based on the annotated subset of the general dataset and the one or more classifier metrics; The trained classifier is used to analyze each general data point in the general dataset to determine a score for each classifier metric among the one or more classifier metrics associated with the corresponding general data point; A filtered general dataset is generated by filtering the general dataset based on one or more filters, where the one or more filters indicate threshold scores for corresponding classifier metrics; The small language model is trained using the filtered general dataset; Generate a synthetic dataset for refining the small language model; as well as After training the small language model with the filtered general dataset, the small language model is then trained with the synthetic dataset.

2. The method of claim 1, wherein each general data point in the general dataset is associated with a score of each classifier metric in the one or more classifier metrics.

3. The method of claim 1, wherein the one or more classifier metrics include factual knowledge, everyday knowledge, scientific knowledge, human behavior, toxicity, integrity, obscenity, ambiguity, commonality, reasoning, promotional content, and / or unwanted content.

4. The method of claim 1, wherein generating the filtered general dataset by filtering the general dataset based on the one or more filters comprises: Generate one or more filters for the one or more classifier metrics, each filter corresponding to a corresponding classifier metric and indicating a threshold score assigned to the corresponding classifier metric; as well as The general dataset is filtered based on one or more of the filters.

5. The method of claim 1, wherein generating the synthetic dataset for refining the small language model comprises: Identify one or more deficient skills in the small language model; Determine one or more data formats to address the one or more defective skills; Generate one or more prompts for generating the one or more data formats; Inject sources of randomization and diversity into one or more of the prompts; as well as The synthetic dataset is generated using a generative transformer based on one or more prompts, the synthetic dataset comprising one or more data formats.

6. The method of claim 5, wherein the one or more defective skills include any skill or topic for improving the capabilities of the small language model.

7. The method according to claim 5, wherein the generative transformer is a multimodal large language model.

8. The method according to claim 1, further comprising: Before training the small language model with the filtered general dataset, a warm start is performed by copying the weights from the existing trained model into the small language model (110).

9. A computing device for generating small language models, the computing device comprising: processor; as well as A memory storing a plurality of instructions, which, when executed by the processor, cause the computing device to: The method for generating a small language model includes: Obtain a general dataset, which includes multiple general datasets; A subset of the general dataset is annotated based on one or more classifier metrics that indicate the quality of the general dataset, the subset representing the general dataset; The classifier is trained based on the annotated subset of the general dataset and the one or more classifier metrics; The trained classifier is used to analyze each general data point in the general dataset to determine a score for each classifier metric among the one or more classifier metrics associated with the corresponding general data point; A filtered general dataset is generated by filtering the general dataset based on one or more filters, where the one or more filters indicate threshold scores for corresponding classifier metrics; The small language model is trained using the filtered general dataset; Generate a synthetic dataset for refining the small language model; and After training the small language model with the filtered general dataset, the small language model is then trained with the synthetic dataset.

10. The computing device of claim 9, wherein each general data point in the general dataset is associated with a score of each classifier metric in the one or more classifier metrics.

11. The computing device of claim 9, wherein the one or more classifier metrics include factual knowledge, everyday knowledge, scientific knowledge, human behavior, toxicity, integrity, obscenity, ambiguity, commonality, reasoning, promotional content, and / or unwanted content.

12. The computing device of claim 9, wherein generating the filtered general dataset by filtering the general dataset based on the one or more filters comprises: Generate one or more filters for the one or more classifier metrics, each filter corresponding to a corresponding classifier metric and indicating a threshold score assigned to the corresponding classifier metric; as well as The general dataset is filtered based on one or more of the filters.

13. The computing device of claim 9, wherein generating a synthetic dataset for refining the small language model comprises: Identify one or more deficient skills in the small language model; Determine one or more data formats to address the one or more defective skills; Generate the one or more prompts used to generate the one or more data formats; Inject sources of randomization and diversity into one or more of the prompts; as well as The synthetic dataset is generated using a generative transformer based on one or more prompts, the synthetic dataset comprising one or more data formats.

14. The computing device of claim 13, wherein the one or more defective skills include any skill or subject for improving the capabilities of the small language model.

15. The computing device of claim 9, wherein, when the plurality of instructions are executed, the computing device further causes the computing device to: Before training the small language model with the filtered general dataset, a warm start is performed by copying the weights from the existing trained model into the small language model (110).

16. A computer storage medium storing computer-executable instructions, which, when executed, cause at least one processor to perform operations, said operations including: Obtain a general dataset, which includes multiple general datasets; A subset of the general dataset is annotated based on one or more classifier metrics that indicate the quality of the general dataset, the subset representing the general dataset; The classifier is trained based on the annotated subset of the general dataset and the one or more classifier metrics; The trained classifier is used to analyze each general data point in the general dataset to determine a score for each classifier metric among the one or more classifier metrics associated with the corresponding general data point; A filtered general dataset is generated by filtering the general dataset based on one or more filters, where the one or more filters indicate threshold scores for corresponding classifier metrics; The small language model is trained using the filtered general dataset; Generate a synthetic dataset for refining the small language model; as well as After training the small language model with the filtered general dataset, the small language model is then trained with the synthetic dataset.

17. The computer storage medium of claim 16, wherein each general data point of the general dataset is associated with a score of each classifier metric in the one or more classifier metrics.

18. The computer storage medium of claim 16, wherein generating the filtered general dataset by filtering the general dataset based on the one or more filters comprises: Generate one or more filters for the one or more classifier metrics, each filter corresponding to a corresponding classifier metric and indicating a threshold score assigned to the corresponding classifier metric; as well as The general dataset is filtered based on one or more of the filters.

19. The computer storage medium of claim 16, wherein generating the synthetic dataset for refining the small language model comprises: Identify one or more deficient skills in the small language model; Determine one or more data formats to address the one or more defective skills; Generate the one or more prompts used to generate the one or more data formats; Inject sources of randomization and diversity into one or more of the prompts; as well as The synthetic dataset is generated using a generative transformer based on the one or more prompts, the synthetic dataset comprising the one or more data formats. The one or more deficient skills mentioned above include any skills or topics used to improve the capabilities of the small language model.

20. The computer storage medium of claim 16, wherein the instructions, when executed by the one or more processors, further cause the computing device to perform the method, the method comprising: Before training the small language model with the filtered general dataset, a warm start is performed by copying the weights from the existing trained model into the small language model (110).

Images