Data health assessment using generative language models

Abstract

The disclosed concepts relate to utilizing a language model to identify data health issues in a dataset. An example method involves accessing a data set. The example method also involves inputting cues to the generative language model using the automated evaluation planning agent to generate a data evaluation plan for the dataset, the cues including a context describing the dataset. The example method also involves receiving a data assessment plan generated by the generative language model and identifying one or more data health issues in the dataset by executing the data assessment plan using the automated assessment plan execution agent.

Description

Background Technology

[0001] Using data to make critical decisions across a wide range of scenarios. Advances in storage technology have made it feasible to store large amounts of data at a reasonable cost. Furthermore, technologies such as solid-state storage have made it possible to process large amounts of data rapidly. However, while these advancements enable organizations to quickly analyze large volumes of data and make decisions based on that analysis, the quality of those decisions still depends solely on the quality of the underlying data itself. Summary of the Invention

[0002] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed embodiments. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0003] This specification generally relates to using generative language models to assess data health. One example includes a method or technique. This method or technique may include a method performed on a computing device, comprising: accessing a dataset using an automated assessment planning agent; inputting cue words to a generative language model to generate a data assessment plan for the dataset, the cue words including context describing the dataset; receiving the data assessment plan generated by the generative language model; and executing the data assessment plan using an automated assessment plan execution agent to identify one or more data health issues in the dataset.

[0004] Another example includes a system comprising a hardware processing unit and a storage resource storing computer-readable instructions that, when executed by the hardware processing unit, cause the system to: access a dataset; input cue words to a generative language model to generate a data evaluation plan for the dataset, the cue words including a summary of the dataset as context; receive the data evaluation plan from the generative language model, the data evaluation plan including one or more data evaluation actions; and perform the one or more data evaluation actions received from the generative language model to identify one or more data health issues in the dataset.

[0005] Another example includes a computer-readable storage medium that stores computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform actions, including: accessing a dataset; inputting prompt words to a generative language model to generate a data evaluation plan for the dataset, the prompt words including context describing the dataset; receiving the data evaluation plan generated by the generative language model; and identifying one or more data health issues in the dataset by executing the data evaluation plan.

[0006] The examples listed above are intended to provide a quick reference to help the reader and are not intended to limit the scope of the concepts described in this article. Attached Figure Description

[0007] The specific embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost numeral of the reference numeral indicates the drawing in which the reference numeral first appears. The use of similar reference numerals in different instances in the specification and drawings may indicate similar or identical items.

[0008] Figure 1 Examples of generative language models consistent with some implementations of this concept are shown.

[0009] Figure 2 An example workflow for data evaluation is shown that is consistent with some implementations of this concept.

[0010] Figure 3A Examples of datasets consistent with some implementations of this concept are shown.

[0011] Figure 3B Examples of annotations for datasets generated by prompt-based generative language models, consistent with some implementations of this concept, are shown.

[0012] Figure 3C An example of a dataset containing annotations generated by a generative language model is shown, consistent with some implementations of this concept.

[0013] Figure 4 An example of a prompt-based generative language model generating a data analysis plan is shown, consistent with some implementations of this concept.

[0014] Figure 5A Examples of code generation using a prompting generative language model, consistent with some implementations of this concept, are shown to implement data evaluation actions of a data evaluation plan.

[0015] Figure 5B The first example shown is a sample of fields from a dataset that are consistent with some implementations of this concept.

[0016] Figure 5C A second example of a sample field from a dataset, consistent with some implementations of this concept, is shown.

[0017] Figure 5D Examples of prompt-based generative language models that are consistent with some implementations of this concept are shown, generating analyses of the results of data evaluation actions.

[0018] Figure 6 and Figure 7 An example graphical user interface consistent with some implementations of this concept is shown that can be used to convey data health.

[0019] Figure 8 Example systems consistent with some implementations of this concept are shown.

[0020] Figure 9 Example methods or techniques for data evaluation using generative language models are shown that are consistent with some implementations of this concept. Detailed Implementation

[0021] Overview As mentioned above, corrupted or otherwise incorrect data can lead to poor or even catastrophic decisions. Advances in hardware allow for the processing of massive amounts of data very quickly, and advancements in data science and machine learning allow for deeper understanding of trends within the data. However, even the most sophisticated data analysis tools and advanced hardware may fail to produce good results when the underlying data itself is faulty.

[0022] One way to detect data quality issues is for trained personnel to manually evaluate the data. For example, a database administrator can examine individual database tables to detect data health problems, such as incorrect formatting or out-of-range values. Automation techniques also exist, such as writing scripts to iterate over data to find specific problems. However, these techniques tend to be very resource-intensive because tools need to be developed or modified as different data sources, software, and / or hardware are used.

[0023] The disclosed implementation overcomes these shortcomings of existing techniques for assessing data health by leveraging generative language models to help identify and, in some cases, automatically correct errors in the dataset. Recent advances in language modeling, such as transformer-based generative language models (e.g., one or more versions of models such as ChatGPT, BLOOM, PaLM, and / or LLaMA), have enabled language models to achieve milestones such as passing the bar exam. These generative language models learn broad concepts from large amounts of training data and are able to formulate data evaluation plans given appropriate queries. Therefore, as discussed in more detail below, generative language models can formulate customized data evaluation plans for different datasets based on the type of data present within them.

[0024] Machine Learning Overview There are various types of machine learning architectures that can be trained to perform a given task. Support vector machines, decision trees, and neural networks are just a few examples of machine learning architectures that have been used in a wide variety of applications, such as image processing and natural language processing. Some machine learning architectures, such as neural networks, use multiple layers of nodes to perform specific operations.

[0025] In a neural network, nodes are connected to each other via one or more edges. A neural network may include an input layer, an output layer, and one or more intermediate layers. Each node can process its corresponding input according to a predefined function and provide its output to subsequent layers, or in some cases, to previous layers. The input to a given node can be multiplied by the corresponding weight value of the edge between the input and the node. Furthermore, nodes may have individual bias values ​​that are also used to produce the output. Various training procedures can be applied to learn the edge weights and / or bias values. When used without embellishment, the term "parameter" in this document refers to learnable values, such as edge weights and bias values ​​that can be learned by training a machine learning model, such as a neural network.

[0026] Neural network architectures can have different layers that perform different specific functions. For example, one or more layers of nodes can work together to perform specific operations such as pooling, encoding, or convolution. For the purposes of this document, the term "layer" refers to a set of nodes that share inputs and outputs (e.g., to or from external sources or other layers in the network). The term "operation" refers to a function that can be performed by one or more layers of nodes. The term "model architecture" refers to the overall architecture of a layered model, including the number of layers, the connectivity of the layers, and the types of operations performed by each layer. The term "neural network architecture" refers to the model structure of a neural network. The terms "trained model" and / or "tuned model" refer to the model structure along with the parameters of the model structure that has been trained or tuned. Note that two trained models can share the same model structure but have different parameter values, for example, if the two models are trained on different training data, or if there is an underlying stochastic process during training.

[0027] Many machine learning tasks suffer from a relative scarcity of training data. A common approach to training a model for a specific task using limited task-specific training data involves "transfer learning." In transfer learning, the model is first pre-trained on another task with a large amount of training data available, and then the model is tuned to suit the specific task using task-specific training data.

[0028] As used herein, the term "pre-training" refers to training a model on a pre-training dataset in such a way that the model parameters are tuned to allow subsequent adjustments to these parameters to adapt the model to one or more specific tasks. In some cases, pre-training can involve a self-supervised learning process on unlabeled pre-training data, where the "self-supervised" learning process involves learning from the structure of pre-training examples in the absence of explicit (e.g., manually provided) labels. Subsequent modifications to the model parameters obtained through pre-training are referred to herein as "tuning." Supervised learning from explicitly labeled training data can be used to perform tuning for one or more tasks, and in some cases, tuning may be performed using a different task than that used for pre-training.

[0029] the term For the purposes of this document, the term "language model" refers to any type of automated agent that communicates via natural language. For example, a language model can be implemented as a neural network, such as decoder-based generative language models like ChatGPT, Long Short-Term Memory models, etc. As used herein, the term "generative model" refers to a machine learning model used to generate new content. Generative models can be trained to predict items in a sequence of training data. When employed in inference mode, the output of a generative model can include a sequence of new items generated by the model. Therefore, a "generative language model" is a model that can generate a new sequence of text given a certain input cue word (e.g., a query that may have some additional context).

[0030] As used in this paper, the term "cue word" refers to the input text provided to a generative language model, which uses this input text to generate output text. Cue words can include queries, such as requests for information from the generative language model. Cue words can also include contextual or additional information that the generative language model uses in response to a query.

[0031] The term "data health issue" refers to any characteristic of a dataset that may affect the results of processing that dataset. Examples of data health issues include corrupted data, erroneous data, improperly formatted data, statistical outliers, etc. The term "data evaluation action" refers to any action performed on a dataset that identifies a data health issue. A "data evaluation plan" is one or more data evaluation actions that can be performed on a given dataset. A "data cleaning action" is an action that attempts to improve data quality by correcting at least one data health issue (e.g., by removing entries or values ​​from the dataset, changing values ​​in the dataset to different values, etc.).

[0032] A dataset's "summary" refers to a representation of the dataset as a whole. A dataset summary may include the data types of the dataset's fields, statistics for the dataset's fields, and / or annotations for the dataset as a whole, a set of fields in the dataset, or individual fields of the dataset. A "data health score" refers to any metric characterizing the presence of data health issues in a dataset. A "severity dictionary" is one or more indicators of the severity of a specific type of data health issue when it is present in the dataset. For example, a severity dictionary might indicate that missing values ​​are relatively more severe than statistical outliers, and may include weights specifying the relative severity of each value.

[0033] The term "machine learning model" refers to any model in a broad range of models that can learn to generate automated user input and / or application output by observing properties of past interactions between users and applications. For example, a machine learning model can be a neural network, support vector machine, decision tree, clustering algorithm, etc. In some cases, machine learning models can be trained using labeled training data, reward functions, or other mechanisms, and in others, machine learning models can learn by analyzing data without explicit labels or rewards. The term "user-specific model" refers to a model having at least one component that is at least partially trained or constructed for a specific user. Therefore, this term encompasses models that have been fully trained for a specific user, models initialized using multi-user data and tuned to suit a specific user, and models that have both general components trained for multiple users and one or more components trained or tuned for a specific user. Similarly, the term "application-specific model" refers to a model having at least one component that is at least partially trained or constructed for a specific application.

[0034] The term "pruning" refers to removing parts of a machine learning model while retaining others. For example, a large machine learning model can be pruned into a smaller model for a specific task by retaining weights and / or nodes that significantly contribute to the model's ability to perform that specific task, while removing other weights or nodes that do not significantly contribute to the model's ability to perform that specific task. Similarly, a large machine learning model can be distilled into a smaller model for a specific task by training the smaller model to approximate the output distribution of the larger model for a task-specific dataset.

[0035] Examples of decoder-based language models Figure 1An exemplary generative language model 100 that can be adopted using the disclosed implementation is shown. The generative language model 100 is an example of a machine learning model that can be used to perform one or more natural language processing tasks involving the generation of text, as discussed in more detail below. For the purposes of this document, the term "natural language" refers to language that is typically used by humans for writing or conversation.

[0036] Generative language model 100 can receive input text 110, such as prompts from a user. For example, the input text may include words, sentences, phrases, or other representations of language. The input text can be decomposed into lexical units and mapped to lexical embeddings and positional embeddings 101 representing the input text. Lexical embeddings can be represented in a vector space where semantically and / or syntactically similar embeddings are relatively close to each other, and semantically or syntactically dissimilar embeddings are relatively far apart. Positional embeddings represent the positional order of each lexical unit relative to other lexical units from the input text.

[0037] Term embeddings and positional embeddings 101 are processed in one or more decoder blocks 102. Each decoder block implements masked multi-head self-attention 103, which is a mechanism that correlates terms at different positions within the input text to compute similarity between these terms. Each term embedding is represented as a weighted sum of other terms in the input text. Attention is applied only to already decoded values, and future values ​​are masked. Layer normalization 104 normalizes the features to a mean of 0 and a variance of 1, producing smooth gradients. Feedforward layers 105 transform these features into representations suitable for the next iteration of decoding, after which another layer of normalization 106 is applied. Multiple instances of decoder blocks can operate sequentially on the input text, with each subsequent decoder block operating on the output of the previous decoder block. After the final decoder block, a text prediction layer 107 predicts the next word in the sequence, which is output as output text 120 in response to input text 110, and is also fed back into the language model. The output text can be a newly generated response to the prompt words provided as input text to the generative language model.

[0038] Example Workflow Figure 2 An example data evaluation workflow 200 is shown, which incorporates a generative language model into an evaluation dataset 202. In workflow 200, a series of automated agents can perform various interactions with the generative language model to complete data evaluation and / or cleaning tasks.

[0039] The dataset is input to a summary agent 204, which generates a data summary 206. For example, the data summary may characterize the dataset as a whole and / or individual fields of the dataset. In some cases, the data summary includes the atomic data types of the individual fields and statistics for certain fields, such as the number of maximum, minimum, and / or unique values. The data summary may also include annotations obtained from a generative language model, such as a semantic description of the entire dataset, the semantic types of the individual fields of the dataset, and textual descriptions of the individual fields of the dataset. Annotations may be obtained by prompting the generative model to generate annotations using information about the dataset, such as the name of the dataset, the names of the fields in the dataset, and / or selected values ​​from the dataset. In some cases, a domain expert context 208 is also employed to generate the summary.

[0040] Data summary 206 is input to evaluation planning agent 210, which generates data evaluation plan 212. For example, the data summary can be provided to a generative language model as context, with prompts requesting a list of data evaluation actions for dataset 202. The generative language model can respond by generating different data evaluation actions to be taken for each field of the dataset. For example, data evaluation actions could include testing for invalid values ​​in the dataset, testing for inconsistent formatting in the dataset, testing for inconsistent semantic types in the dataset, testing for missing values ​​in the dataset, testing for outliers in the dataset, testing for duplicate unique values ​​in the dataset (e.g., two different rows with the same primary key), or testing for inconsistent units in the dataset.

[0041] Data evaluation plan 212 is input to evaluation plan execution agent 214, which generates data evaluation result 216. For example, the evaluation plan execution agent may involve code that executes actions to obtain samples from the dataset and perform data evaluation on those samples. In some cases, the code is written by a human developer. In other cases, the code may be prompted by a generative language model, as described further below.

[0042] The data evaluation results are input into aggregation and scoring agent 218, which generates an overall health score 220 from the results. In some cases, a machine learning model (e.g., a regression model) is used to determine the overall health score. In other cases, a severity dictionary 222 is used. For example, the severity dictionary may include weights for different types of errors, and the overall score may be calculated based on a weighted sum of all errors identified in the data evaluation results 216.

[0043] Example Dataset Summary Figure 3A Table 300 is shown, with two columns: Date 302 and Price 304. Table 300 is an example of a dataset that can be evaluated using Data Evaluation Workflow 200. As mentioned above, summarization can involve determining the data types of the fields in the dataset. For example, the Date column is of the "Date" data type, and the Price column is of the "Number" data type. Summarization can also involve calculating statistics for different fields. For example, statistics can include the number of minimum values, maximum values, unique values, etc.

[0044] Furthermore, summaries can include annotations obtained from generative language models. For example, references Figure 3B The generative language model interface 310 has a prompt field 312 and a response field 314. The generative language model can be prompted via the prompt field 312 with a request such as, "Based on the following information related to the dataset, please provide a semantic description of the dataset, the semantic type of each field, and a textual description of each field." The dataset itself (or values ​​sampled from it) and / or the dataset filename, as well as the name of each field, can be provided to the generative language model as context along with the prompt. Statistics for any field can also be provided as part of the prompt.

[0045] Generative language models can generate descriptions output via response field 314. Here, the generative language model responds as follows: "This dataset contains ABC stock prices from 4 / 1 / 2000 to 11 / 1 / 2009. The date field has the semantic type 'date' and represents the date on which the ABC stock price was recorded. The price field has the semantic type 'number' and represents the ABC stock price on a given date." Because modern generative language models are trained on large-scale training datasets and can learn a wide range of concepts, they are able to identify the meaning of different fields in a dataset by reasoning about information such as the name of the dataset, the names of the fields in the dataset, other metadata, and / or the values ​​present in the dataset. Therefore, generative language models can provide annotations that distill the meanings obtained from these information sources into natural language annotations.

[0046] Note that this paper uses a graphical user interface (GUI) to describe interactions with a generative language model to convey certain concepts. However, in some implementations, the generative language model can be accessed programmatically, for example, using one or more application programming interfaces (APIs), without necessarily involving a user interface. In other cases, the techniques described in this paper can be implemented with human intervention at different times, in which case a GUI can be used to interact with the generative language model. For the purposes of this document, unless otherwise specifically stated, it is assumed that all automated agents operate without human intervention. Therefore, the prompts shown in this paper can be programmatically provided to the generative language model, and the generated responses can be programmatically received by the corresponding automated agents.

[0047] Figure 3C A dataset summary 320 is shown, which includes a description of each field of the dataset. Entries obtained through data type analysis or statistical calculations are shown in regular font, and entries obtained from the generative language model via annotations are shown in bold font. As described in more detail below, the dataset summary can provide sufficient information for the generative language model to generate a data evaluation plan for the dataset.

[0048] Example evaluation plan generation Figure 4 An example is shown of how a generative language model can be prompted to generate a data evaluation plan. For example... Figure 4 As shown, a generative language model can be prompted with queries such as, "Based on the following data summary of the dataset, and acting as a data analyst, please list the data analysis actions to be performed on the dataset according to best practices." The query can be accompanied by a previously generated data summary as context, which can also be input into the generative language model. In response, the generative language model can generate a data evaluation plan.

[0049] Return to reference Figure 3C The dataset summary 320, input along with the query into the generative language model, indicates that the dataset includes both date and numeric types. In this case, the data evaluation plan generated by the generative language model includes two suggested data evaluation actions. The first data evaluation action involves testing the date field against valid dates. The second data evaluation action involves testing the price field against statistical outliers.

[0050] As mentioned earlier, modern generative language models are trained on a very large range of training data and are thus able to infer a wide range of topics. Here, a generative language model trained on natural language sentences with date fields that may contain invalid dates can recognize that testing for valid dates is an appropriate data evaluation action for any dataset with date fields. Similarly, a generative language model trained on natural language sentences with numeric fields that may contain statistical outliers can recognize that testing for statistical outliers is an appropriate data evaluation action for any dataset with numeric fields.

[0051] Example assessment plan execution Figure 5A An example is shown of how a generative language model can be used to generate code for implementing a data evaluation plan. For instance, the generative language model can prompt with a query such as, “Based on the following summary, please write a script to (1) sample and test the date field of the dataset for valid dates, and (2) sample and test the price field of the following dataset for statistical outliers.” The prompt can be provided along with additional context such as the data type of the field. In response, the generative language model can generate two corresponding scripts with code that performs the requested data evaluation actions.

[0052] Modern generative language models have demonstrated the ability to write accurate code given natural language prompts describing the expected function of the code. When a generative language model has seen enough training examples of code along with natural language descriptions of how the code works, it can generate code in different contexts to perform similar functions. Therefore, for example, a generative language model already trained on training data showing code used to test the date field of a first dataset can, given a description of a second dataset, write code to test the date field of a second dataset, even if the generative language model has not been specifically trained on code written for a second dataset.

[0053] Figure 5B The data sample 510 that can be extracted by the first script is shown. The first script is able to identify the value 4 / 31 / 2002 as an invalid date because April does not have 31 days. In other words, the first script has identified a single data health issue in the first sample. Figure 5C The data sample 520 that can be extracted by the second script is shown. The second script is able to identify the values ​​-124.27 and -340.23 as statistical outliers because they are the only negative values ​​in the sample. In other words, the second script has identified two data health issues in the second sample.

[0054] like Figure 5DAs shown, the script's output can be fed into a generative language model as prompts for analysis. For example, a query such as "Please analyze the following results of testing these data samples given the following summary" can be input into the generative language model as context, along with a data health issue identified by the script. In response, the generative language model can generate responses such as, "10% of the sampled dates are invalid. This is a moderately serious error that could affect any analysis code processing the dataset. 20% of the sampled prices are out of range and have negative values. This is a very serious error because programs developed for processing stock prices are typically coded with the expectation that all stock prices have positive values." Here, a generative language model trained with natural language that characterizes the severity of data health issues can infer the severity of identified health problems from the input test results. For example, the generative language model may have seen training data such as an email from one developer to another: “There are some invalid dates in our sales spreadsheet, which may cause us some problems, but we should continue with the demonstration.” Giving enough examples of such natural language statements allows the generative language model to infer that invalid dates are a moderate problem. Similarly, the generative language model may have seen training data with natural language statements that characterize the presence of negative values ​​in fields that should only have positive values ​​as a very serious error. For example, the generative language model may have seen sentences such as “These negative prices for ZZZ stock will crash our trading platform!!” Training on sentences like these allows the generative language model to infer that negative values ​​in the price field may be a very serious error.

[0055] In some cases, evaluating plan execution may also involve performing one or more data cleaning actions. For example, data cleaning actions may involve executing code that attempts to correct individual errors. For instance, a generative language model might be provided with a prompt word requesting code to "correct any invalid dates in this dataset." The generative language model could then generate a script that removes all entries (e.g., rows) with invalid dates from the dataset.

[0056] However, data cleaning actions do not necessarily improve the health of a given dataset, depending on various factors. For example, in a dataset given stock prices, simply deleting any entries with invalid dates might be better, since it's impossible for a stock to have a given price on that date, and there are other valid dates with valid prices in the dataset. On the other hand, consider a dataset of students in a given high school classroom, where each student is an entry in the dataset, and a date field represents their birthday. The dataset may contain other valuable information about the students, such as gender, interests, special needs, etc. In this case, performing a different data cleaning action than deleting students from the dataset might be preferred. For example, a data cleaning action could involve adding a flag such as "INVALID" to the dataset. This might prompt teachers to ask students for their birthdays and ultimately facilitate a better solution than deleting the entry.

[0057] In some cases, evaluating the execution of a data cleaning plan can involve trying various data cleaning actions, testing the results, and iteratively continuing the process until a final set of data cleaning actions is identified. For example, if the first data cleaning action recommended by a generative language model involves removing students from the dataset, the generative language model might be able to detect, after being prompted by the results of the first data cleaning action, that the result of that action is problematic—for example, if the dataset also lists the total number of students and the total is incorrect after the deletion. The generative language model can indicate that the data health of the dataset has been compromised by the deletion of students. Subsequently, after replacing invalid birth dates with an "invalid" flag and prompting with the results of that data cleaning action, the generative language model can indicate that the data health of the dataset has improved compared to having an invalid birth date field. More generally, the ability of generative language models to reason about complex data relationships allows for the iterative generation and testing of data cleaning actions suitable for a dataset, given a summary of the dataset and / or the context of the results of performing individual data cleaning actions.

[0058] Example aggregation and rating Given the results of a given script and / or a textual description of the results provided by a generative language model, an overall health score can be calculated. If data cleaning actions that modify the dataset are performed, a health score can be calculated for both the original data and the data after the cleaning action. If only data evaluation actions that do not modify the data are performed, a single overall score can be calculated.

[0059] In some cases, an error weighting function can be used to calculate the overall health score, where errors are identified when the assessment plan is executed. For example, weights can be assigned to each type of error, and then a weighted sum can be calculated for each error. The weighted sum can be used as a penalty to calculate a normalized health score between 0 and 100, where 100 represents data with no errors and 0 represents data that is completely corrupted.

[0060] In other cases, a regression model can be trained to map the script's results to health scores. For example, a regression model can be trained using a training dataset and corresponding error results labeled with health scores. The regression model can then map the results of the evaluation plan to the health scores of the dataset being evaluated.

[0061] In some cases, aggregation and scoring can also involve generating visualizations representing the health of individual datasets. For example, Figure 6 An interface 600 shows bar charts with data health scores for three different datasets. The ABC stock price dataset has a health score of approximately 70, the XYZ stock price dataset has a health score of approximately 90, and the GGG stock price dataset has a health score close to 100.

[0062] In some cases, visualizations that provide more detail about errors in a given dataset can also be presented to the user. For example, suppose a user clicks on a bar representing the stock prices of ABC in interface 600. A visualization could be presented to the user... Figure 7 Another interface, 700, is shown. Interface 700 provides a more detailed view of the data health issues in the ABC stock price dataset. Here, approximately 70% of the data has no health issues. 10% of the dates in the dataset are invalid, and 20% of the prices are outliers.

[0063] Example System This implementation can be executed on various devices in various scenarios. Figure 8 An example system 800 in which this implementation can be adopted is shown, as discussed in more detail below.

[0064] like Figure 8 As shown, system 800 includes client devices 810, data evaluation server 820, generative language model server 830, and database server 840 connected by one or more networks 850. Note that client devices can be mobile devices such as smartphones or tablets, as well as fixed devices such as desktop computers, server devices, etc. Similarly, servers can be implemented using various types of computing devices. In some cases, Figure 8 Any device shown, but in particular servers, can be implemented in data centers, server farms, etc.

[0065] Figure 8 Some components of the devices shown may be referred to herein by reference numerals in parentheses. For the purposes of the following description, parentheses (1) indicate the presence of a given component on client device 810, (2) indicate the presence of a given component on data evaluation server 820, (3) indicate the presence on generative language model server 830, and (4) indicate the presence on database server 840. Unless identifying a specific instance of a given component, this document will generally refer to the component without parentheses.

[0066] Typically, devices 810, 820, 830, and / or 840 may have corresponding processing resources 801 and storage resources 802, which will be discussed in more detail below. The devices may also have various modules that use the processing and storage resources to perform the techniques discussed herein. Storage resources may include both persistent storage resources (such as magnetic or solid-state drives) and volatile storage devices (such as one or more random access memory devices). In some cases, modules are provided as executable instructions stored on a persistent storage device, loaded into a random access memory device, and read from the random access memory by the processing resources for execution.

[0067] Client device 810 may include a configuration module 811 that can interact with various modules on data evaluation server 820. Generally, the configuration module can provide the data evaluation server with certain configuration parameters for evaluating one or more datasets. For example, configuration parameters may identify the dataset that data evaluation server 820 should process, any domain expert context information describing the dataset, a severity dictionary indicating the weights of different types of errors, etc. Client device 810 may also include an interface module 812 that can be used to interact with a visualization representing the data evaluation of the data evaluation server.

[0068] The data evaluation server 820 can obtain the identified dataset from the database server 840 and process the identified dataset using a summary agent 204, an evaluation plan agent 210, an evaluation plan execution agent 214, and an aggregation and scoring agent 218. The summary agent can generate a summary of the given dataset, such as... Figure 3C As shown. The summary agent can analyze atomic data types in a given dataset and include these atomic data types in the summary. The summary agent can also populate the summary with expert domain context. The summary agent can also prompt the generative language model 831 on the generative language model server 830 to generate annotations to be included in the summary.

[0069] The evaluation planning agent 210 on the data evaluation server 820 can determine a data evaluation plan with one or more data evaluation actions. For example, the evaluation planning agent can prompt the generative language model 831 to request a data evaluation plan. The prompt words can be provided to the generative language model along with context, such as a data summary generated by the summarization agent.

[0070] The evaluation plan execution agent 214 on the data evaluation server 820 can execute each data evaluation action in the data evaluation plan. For example, the evaluation plan execution agent can receive code implementing the data evaluation plan via user input and execute the code. In other cases, the evaluation plan execution agent can prompt the generative language model 831 to write code to implement the data evaluation plan and execute the code. In a further case, the code is initially generated by the generative language model and then modified or tested by the user before execution. The code can obtain samples (e.g., rows) from different fields (e.g., columns) of the dataset and test for various errors. For example, the code can perform tests against one or more invalid values ​​in the samples, against inconsistent formats in the samples, against inconsistent semantic types in the samples, against missing values ​​in the samples, against outliers in the samples, against duplicate unique values ​​in the samples, or against inconsistent units in the samples.

[0071] The aggregation and scoring agent 218 on the data evaluation server 820 can obtain the results of the data evaluation plan and determine the aggregate health score of the dataset. As previously mentioned, a regression model can be used, or in other cases, a weighted sum can be used to determine the overall severity of errors in a given dataset. For example, a multiplier of 1 can be used for each formatting error, a multiplier of 3 can be used for each invalid value, and so on. The weighted sum can be normalized and converted into an aggregate health score. The aggregation and scoring agent can also generate one or more visualizations representing the results of the data evaluation plan. The visualizations can be sent from the data evaluation server 820 to the client device 810 for display by the interface module 812.

[0072] Note that different data evaluation plans can be executed for different datasets. For example, database server 840 can store datasets 841, 842, and 843. These datasets can be accessed via database module 844 (e.g., a database management system). Datasets can have different field names, values, etc. Therefore, the summaries generated for each dataset will be different, and the data evaluation plan for each dataset can also be different. For example, if a dataset is missing any numerical values, the generative language model might not recommend testing against outliers on that dataset. As another example, if only one dataset includes dates, the generative language model might only recommend testing against out-of-range dates on that particular dataset.

[0073] Example Method Figure 9 An example method 900 consistent with this concept is shown. As discussed in more detail below, method 900 can be implemented on many different types of devices, such as by one or more cloud servers, by client devices (such as laptops, tablets, or smartphones), or by a combination of one or more servers, client devices, etc.

[0074] Method 900 begins at box 902, where a dataset is accessed. For example, the dataset can be one or more tables from a relational database, spreadsheet, .csv or spreadsheet file, .json file, etc. In some cases, the data is received over a network, for example, retrieved from database server 840 via data evaluation server 820.

[0075] Method 900 continues at box 904, where cue words about the data are input into the generative language model. For example, as described above, the cue words may request the generative language model to generate one or more data evaluation actions. In general, the data evaluation actions may constitute a data evaluation plan. The cue words provided to the generative language model may include a summary of the dataset as context.

[0076] Method 900 continues at box 906, where a data evaluation plan generated by a generative language model is received. The data evaluation plan generated by the generative language model can be conditioned on a summary provided along with the dataset as context. In this way, the generative language model can generate data evaluation plans tailored or custom-made for each dataset.

[0077] Method 900 continues at box 908, where data health issues are identified. For example, as described above, data evaluation actions can be performed by executing user-generated code and / or code generated by a generative language model. The result of executing the code can identify various data health issues in samples taken from the dataset. In other cases, data from the dataset can be input into the generative language model using prompt words, and the generative language model can directly identify data health issues by analyzing the data without generating code to do so.

[0078] Method 900 continues at box 910, where the data evaluation results are aggregated to determine a data health score. As previously described, weights from a severity dictionary can be used to calculate a weighted sum for deriving the data health score, and / or a machine-trained regression model can determine the data health score. In the case of performing data cleaning actions, a data health score can be determined for both the raw and cleaned data to communicate the extent to which the data cleaning actions improved the health of the original dataset.

[0079] Method 900 continues at box 912, where one or more user interfaces can be output. For example, one or more graphical user interfaces can be generated to convey the health of the dataset. The graphical user interface can also convey information about specific types of errors present in samples obtained from various fields of the dataset.

[0080] Alternative implementation The examples provided above are intended to convey certain concepts, but the concepts described in this article are not limited to those examples. For instance, there are many other ways to use generative language models to evaluate and / or clean datasets beyond those specifically described above. Additionally, there are many different ways to compute data health scores.

[0081] For example, the above example handles invalid dates by deleting entries with invalid dates or adding flags to fields indicating that the date of a given entry is invalid. In other cases, it might be more appropriate to correct invalid dates by selecting the most recent valid date; for example, April 31 could be corrected to April 30 or May 1. As another example, instead of deleting entries with statistical outliers, data can be cleaned by interpolating between two adjacent valid values ​​in the dataset and replacing the outliers with the interpolated values. This might be reasonable, for example, for house values, which do not tend to fluctuate on a daily basis. This might be less reasonable for stock prices, which are likely more volatile than house values. Modern generative language models are able to recognize that house prices do not fluctuate as rapidly as stock prices and are therefore able to articulate different data cleaning actions accordingly.

[0082] As another example, consider two datasets with an "Age" field. The first dataset contains a list of registered voters, and the second dataset contains a list of high school students. Furthermore, consider a generative language model that identifies the meaning of other fields in the datasets (e.g., the "Voter ID" field in the first dataset and the "Grade Level" field in the second dataset). The generative language model can annotate a summary for each dataset accordingly. The generative language model can consider these annotations when prompted to generate a data evaluation plan for each dataset. For example, the generative language model could generate a data evaluation action for a test targeting voters under 18 years old on the registered voter dataset, since many states have laws requiring voters to be at least 18. However, this test would be unreasonable for high school students. Instead, the generative language model could generate a different data evaluation action that tests the age field of the student dataset against values ​​appropriate for high school students (e.g., 14-18).

[0083] Similarly, generative language models can identify the different severity of outlying values ​​in the two datasets. A 17-year-old voter who is ineligible to vote might be considered a very serious error, and generative language models can recognize this by training on natural language sentences that indicate voting by someone under 18 is illegal. On the other hand, occasionally older or younger students may attend high school; for example, some students might skip grades and be younger than the standard high school age range, or they might need extra years to complete high school due to health issues, etc. Therefore, generative language models can recognize that for the high school student dataset, being out of age is a less serious error than in the registered voter dataset. The resulting health score for each dataset reflects this, with the voter dataset being penalized more severely than the dataset of out-of-age students.

[0084] Despite the immense power of modern generative language models, there are some things they simply cannot know because they haven't been trained on the appropriate data. For example, consider a scenario where legislators from a legislative body want to understand the possible political outcomes of changing the voting age from 18 to 16. Generative language models don't inherently "know" what the legislators are trying to do, and the vast majority of training data for a general-purpose model would likely indicate that voting ages below 18 are serious errors. This is an instance where domain context can be useful; for example, a summary of the voter dataset could be annotated to state that "ages of 16 and above are acceptable" for the legislator or their team members. The generative language model could then generate a data evaluation plan that treats ages below 16 (instead of 18) as out-of-range errors that are problematic.

[0085] Technical effect Traditionally, data scientists manually examine data to identify problems and then take corrective action. However, this approach is not only laborious and time-consuming but also error-prone. The consequences can be disastrous when human experts misdiagnose data health issues or when their attempts to correct them fail.

[0086] Recent advances in language modeling have led to generative language models capable of learning vast amounts of information, to the point where they can even accomplish tasks such as passing the bar exam. The techniques disclosed can leverage these capabilities of modern generative language models to automate data analysis and cleaning. As mentioned above, for example, generative language models can be used to summarize data, identify specific data health issues in data samples, plan corrective actions for a given dataset, and even generate code to fix the identified problems. All of this can be performed in an automated manner that reduces or eliminates reliance on error-prone manual techniques.

[0087] Note that there are technical limitations of generative language models that can be overcome by the disclosed techniques. For example, generative models often have input character limitations; some generative models are limited to 4k, 8k, 32k, or 64k input characters. Therefore, for large datasets, inputting the entire dataset for generative language model analysis is not necessarily feasible. The disclosed implementation overcomes these memory limitations by decomposing the data evaluation or cleaning task into a series of smaller subtasks that can be performed by corresponding automated agents. Each agent has its own defined role throughout the task and can perform that role using smaller cue words from the generative model that conform to the model's memory limitations.

[0088] For example, Summary Agent 204 can generate a summary of a dataset using statistics, data types, field or dataset names, and model-generated annotations. The annotations themselves can be generated by the generative language model based on statistics, data types, and / or field or dataset names, without having to input the entire dataset into the generative language model. Therefore, the generative model can generate annotations using information that conforms to the model's memory limitations.

[0089] Similarly, the evaluation planning agent 210 can use the summary generated by the summarizing agent as context for subsequent prompts to generate a data evaluation plan. Likewise, the generative language model does not necessarily require the entire dataset as context. Instead, a summary including annotations previously generated by the generative language model provides sufficient context for the generative language model to generate appropriate data evaluation actions.

[0090] Furthermore, the evaluation plan execution agent 214 can use a summary (including annotations generated by the generative language model) and a data evaluation action generated by the generative language model as context to request code to perform the data evaluation action. The prompts conform to the memory limitations of the generative language model, but the generated code can test the entire dataset. Moreover, because the code tests samples of the dataset rather than having to test every entry in the dataset, the results of executing the code can conform to the memory limitations of the generative language model, and the aggregation and scoring agent 218 can utilize the generative language model without exceeding its memory limitations.

[0091] As mentioned above, decomposing data evaluation or cleaning tasks into corresponding subtasks that can be performed by different automated agents has the technical benefit of allowing the execution of each subtask without exceeding the memory limit of the generative language model. Furthermore, there are other benefits to doing so. In some cases, not only do individual cue words conform to the memory limit of the generative language model, but the entire sequence of cue words can also conform to the memory limit. Therefore, for example, the generative language model can focus on embeddings from previous cue words and responses, e.g., in a masked multi-head self-attention layer 103 (…). Figure 1 Therefore, for example, when responding to a prompt from a subsequently executed agent, a generative language model can consider both the prompt from a previously executed agent and the response.

[0092] In a sense, the disclosed technique can be viewed as a type of thought chain cue. Each agent performs a different part of a larger data evaluation or cleaning task. The generative language model is not required to evaluate or clean the dataset in a single step. Instead, the overall task is broken down into discrete steps, where additional information (such as data types, field names, statistics, etc.) is used to seed initial summary generation and then utilized in subsequent cue words. This helps the model reason in an ordered manner, enabling it to produce appropriately tailored responses to the characteristics of the individual datasets being evaluated and / or cleaned.

[0093] Furthermore, the disclosed techniques can save computational resources, such as processor cycles and / or memory, when performing data cleaning operations. For example, when a dataset has many entries with invalid fields, it may be beneficial to simply remove those entries before performing intensive data analysis operations on the dataset. By doing so, fewer entries can be processed using only processor cycles and memory. By customizing the data evaluation plan to fit the dataset using a generative language model, these entries may be safely removed without affecting subsequent data analysis. As another example, when a dataset has statistical outliers, this may lead to sorting or data binning routines rearranging a large number of data items, thus using processor and memory resources that can be preserved by removing these values ​​or replacing them with interpolation.

[0094] Equipment implementation As mentioned above Figure 8 The system 800 includes several devices, including client device 810, server 820, server 830, and client device 840. It is also noted that not all device implementations can be shown, and other device implementations will be obvious to those skilled in the art from the description above and below.

[0095] As used herein, the terms “device,” “computer,” “computing device,” “client device,” and / or “server device” can mean any type of device having a certain amount of hardware processing power and / or hardware storage / memory capacity. Processing power can be provided by one or more hardware processors (e.g., hardware processing units / cores) capable of executing data in the form of computer-readable instructions to provide functionality. Computer-readable instructions and / or data can be stored on storage devices, such as storage devices / memory and / or data storage devices. As used herein, the term “system” can mean a single device, multiple devices, etc.

[0096] Storage resources can be internal or external to their associated corresponding devices. Storage resources can include any one or more of volatile or non-volatile memory, hard disk drives, flash memory devices, and / or optical storage devices (e.g., CDs, DVDs, etc.). As used herein, the term "computer-readable medium" can include signals. Conversely, the term "computer-readable storage medium" does not include signals. Computer-readable storage media includes "computer-readable storage devices." Examples of computer-readable storage devices include volatile storage media (such as RAM) and non-volatile storage media (such as hard disk drives, optical discs, and flash memory, etc.).

[0097] In some cases, the device is configured with general-purpose hardware processors and storage resources. In others, the device may include a system-on-a-chip (SOC) design. In an SOC design implementation, the functionality provided by the device can be integrated onto a single SOC or multiple coupled SOCs. One or more associated processors can be configured to coordinate with shared resources (such as memory, storage devices, etc.) and / or one or more dedicated resources (such as hardware blocks configured to perform certain specific functions). Therefore, the terms “processor,” “hardware processor,” or “hardware processing unit” as used herein may also refer to a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a processor core, or other types of processing devices suitable for implementation in both conventional computing architectures and SOC designs.

[0098] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0099] In some configurations, any modules / code discussed herein may be implemented in software, hardware, and / or firmware. In any case, modules / code may be provided during the manufacture of the device or by an intermediary preparing to sell the device to the end user. In other cases, the end user may install these modules / code later, such as by downloading and installing the executable code on the corresponding device.

[0100] It's also important to note that devices typically have input and / or output capabilities. For example, computing devices can have various input mechanisms, such as keyboards, mice, touchpads, voice recognition, and gesture recognition (e.g., using depth cameras, such as stereo or time-of-flight camera systems, infrared camera systems, RGB camera systems, or using accelerometers / gyroscopes, facial recognition, etc.). Devices can also have various output mechanisms, such as printers, monitors, etc.

[0101] It should also be noted that the devices described herein can function independently or collaboratively to implement the described techniques. For example, the methods and functions described herein can be executed on a single computing device and / or distributed across multiple computing devices communicating via a network 850. Without limitation, the network 850 may include one or more local area networks (LANs), wide area networks (WANs), and the Internet, etc.

[0102] Additional examples The above describes various examples. Further examples are described below. One example includes a method executed on a computing device, comprising: accessing a dataset; using an automated evaluation planning agent to input prompt words into a generative language model to generate a data evaluation plan for the dataset, the prompt words including context describing the dataset; receiving the data evaluation plan generated by the generative language model; and executing the data evaluation plan using an automated evaluation plan execution agent to identify one or more data health issues in the dataset.

[0103] Another example may include any of the above and / or the following examples, wherein one or more data health issues include: one or more invalid values ​​in the dataset, inconsistent format in the dataset, inconsistent semantic type in the dataset, missing values ​​in the dataset, outliers in the dataset, duplicate unique values ​​in the dataset, or inconsistent cells in the dataset.

[0104] Another example may include any of the above and / or the following examples, wherein the method further includes: generating a summary of the dataset and inputting the summary of the dataset into a generative language model as a context describing the dataset.

[0105] Another example may include any of the above and / or the following examples, wherein the method further includes: prompting the generative language model to generate one or more annotations for the dataset using the name of the dataset, the name of a field in the dataset, or a value in the dataset as context by an automated summary agent; receiving one or more annotations for the dataset from the generative language model; and including one or more annotations in a summary, wherein the one or more annotations generated by the generative language model are used as context for generating a data evaluation plan.

[0106] Another example may include any of the above and / or the following examples, wherein the annotations include a semantic description of the dataset generated by the generative language model, the semantic type of the fields of the dataset generated by the generative language model, and a textual description of the fields generated by the generative language model.

[0107] Another example may include any of the above and / or the following examples, wherein the method further includes: determining a data health score for the dataset by an automated aggregation and scoring agent based at least on the one or more data health issues.

[0108] Another example may include any of the above and / or the following examples, wherein the data health score is determined by an automated aggregation and scoring agent using at least one of a severity dictionary or a regression model.

[0109] Another example may include any of the above and / or the following examples, wherein the method further includes: writing code by prompting the generative language model to implement one or more data evaluation actions of the data evaluation plan by the automated evaluation plan execution agent, receiving the code from the generative language model, and executing the code to implement the data evaluation plan.

[0110] Another example may include any of the above and / or the following examples, wherein code written by a generative language model obtains samples from a dataset and performs one or more data evaluation actions of a data evaluation plan on the samples.

[0111] Another example could include any of the examples above and / or below, where the code performs one or more data cleaning actions on the dataset.

[0112] Another example may include any of the above and / or the following examples, where one or more data cleaning actions involve removing values ​​from the dataset or changing values ​​in the dataset.

[0113] Another example may include any of the above and / or the following examples, wherein the method further includes: determining, by an automated evaluation plan execution agent prompting a generative language model, whether one or more data cleaning actions improve the data quality of the dataset, and performing different data cleaning actions on the dataset in response to a response from the generative language model indicating that a particular data cleaning action does not improve the data quality of the dataset.

[0114] Another example may include any of the above and / or the following examples, where the generative language model includes a transformer decoder.

[0115] Another example may include any of the above and / or the following examples, wherein the method further includes: performing pruning or distillation on another generative language model having another transformer decoder to obtain a generative language model having fewer parameters than the other generative language model.

[0116] Another example may include a system comprising a hardware processing unit and a storage resource storing computer-readable instructions that, when executed by the hardware processing unit, enable the system to access a dataset, input cue words to a generative language model to generate a data evaluation plan for the dataset, the cue words including a summary of the dataset as context, receive the data evaluation plan from the generative language model, the data evaluation plan including one or more data evaluation actions, and perform the one or more data evaluation actions received from the generative language model to identify one or more data health issues in the dataset.

[0117] Another example may include any of the above and / or the following examples, wherein computer-readable instructions, when executed by a hardware processing unit, cause the system to prompt a generative language model to annotate the dataset, and include annotations from the generative language model along with prompt words to generate a data evaluation plan.

[0118] Another example may include any of the above and / or the following examples, wherein the annotations include a semantic description of the dataset generated by the generative language model, the semantic type of the fields of the dataset generated by the generative language model, and a textual description of the fields generated by the generative language model.

[0119] Another example may include any of the above and / or the following examples, wherein computer-readable instructions, when executed by a hardware processing unit, cause the system to include, in a summary included in the prompt words as context, at least the data types of the dataset's fields and statistics for specific fields of the dataset.

[0120] Another example could include any of the examples above and / or below, where the statistics include the number of minimum, maximum, and unique values ​​for a specific field.

[0121] Another example may include a computer-readable storage medium storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform operations including: accessing a dataset; inputting prompt words to a generative language model to generate a data evaluation plan for the dataset, the prompt words including context describing the dataset; receiving the data evaluation plan generated by the generative language model; and identifying one or more data health issues in the dataset by executing the data evaluation plan.

[0122] Summarize Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms of implementing the claims, and other features and actions that a person skilled in the art will recognize are intended to fall within the scope of the claims.

Claims

1. A computer-implemented method, executed on a computing device, comprising: (A) Process signal frames at a frame rate based on step value using time-synchronized beam search technology; (B) Determine the hypothesis score for each hypothesis in the first set of information for the signal frame; (C) Determine the hypothesis score for each hypothesis in the second information set for the signal frame; (D) Compare the worst hypothesis score of the first information set with the sum of the best hypothesis score and the threshold of the second information set; as well as (E) When the worst hypothesis score of the first information set is greater than the sum of the best hypothesis score of the second information set and the threshold, the processing of the signal frame is stopped.

2. The computer-implemented method according to claim 1 further includes: When the worst hypothesis score of the first information set is not greater than the sum of the best hypothesis score of the second information set and the threshold, repeat tasks (B) to (D).

3. The computer-implemented method according to claim 1, wherein the step size is greater than 8.

4. The computer-implemented method according to claim 3, wherein the step value is at least 12.

5. The computer-implemented method of claim 1, wherein the first information set includes information from the frame, including a blank token.

6. The computer-implemented method of claim 1, wherein the second information set includes information from the frame including a non-blank token.

7. The computer-implemented method of claim 1, wherein the signal frame comprises a frame of encoded audio signal.

8. The computer-implemented method according to claim 1, wherein the step value is dynamically set according to the characteristics of the data being processed.

9. A computing system, comprising: Memory; as well as Processor, used for: (A) Processing frames of encoded audio signals at a frame rate based on step value using time-synchronized beam search technology; (B) Determine the hypothesis score for each hypothesis in the first information set for the frame of the encoded audio signal; (C) Determine the hypothesis score for each hypothesis in the second information set for the frame of the encoded audio signal; (D) Compare the worst hypothesis score of the first information set with the sum of the best hypothesis score and the threshold of the second information set; (E) When the worst hypothesis score of the first information set is greater than the sum of the best hypothesis score of the second information set and the threshold, processing of the frame of the encoded audio signal is stopped; as well as (F) When the worst hypothesis score of the first information set is not greater than the sum of the best hypothesis score of the second information set and the threshold, repeat tasks (B) to (D).

10. The computing system according to claim 9, wherein the step size is greater than 8.

11. The computing system of claim 10, wherein the step size is at least 12.

12. The computing system of claim 11, wherein the first information set includes information from the frame, including a blank token.

13. The computing system of claim 12, wherein the second information set includes information from the frame including a non-blank token.

14. A computer program product residing on a computer-readable medium having stored thereon a plurality of instructions, said instructions, when executed by a processor, causing the processor to perform operations, said operations including: (A) Processing frames of encoded audio signals at a frame rate based on step value using time-synchronized beam search technology; (B) Determine the hypothesis score for each hypothesis in the first information set for the frame of the encoded audio signal; (C) Determine the hypothesis score for each hypothesis in the second information set for the frame of the encoded audio signal; (D) Compare the worst hypothesis score of the first information set with the sum of the best hypothesis score and the threshold of the second information set; as well as (E) When the worst hypothesis score of the first information set is not greater than the sum of the best hypothesis score of the second information set and the threshold, repeat tasks (B) to (D).

15. The computer program product according to claim 14, further comprising: When the worst hypothesis score of the first information set is greater than the sum of the best hypothesis score of the second information set and the threshold, the processing of the audio signal frame is stopped.

Images