Non-natural language generation and correction

Abstract

In some examples, a method comprises receiving a data input comprising non-natural language text, processing the data input, whereby to acquire a processed data input, initialising a large language model, LLM, and training the LLM based on a part of the processed data input, whereby to select a prompting strategy for the LLM, pruning the LLM, utilising a multi-task learning process for the pruned language model by training the model on multiple related tasks, generating multiple corrected data inputs, each of the multiple corrected data inputs based on the received data input comprising the non-natural language text, using the LLM, combining the multiple corrected data inputs to obtain a final corrected data input, determining whether the final corrected data input meets a predefined quality threshold based on a comparison with a set of ground-truth data, in response to determining that the final corrected data input meets a predefined quality threshold, outputting the final corrected data input, and categorising the final corrected data input.

Description

[0001] NON-NATURAL LANGUAGE GENERATION AND CORRECTION

[0002] TECHNICAL FIELD

[0003] The present disclosure relates, in general, to non-natural language generation and correction. Aspects of the disclosure relate to non-natural language hierarchical categorisation and recommendation.

[0004] BACKGROUND

[0005] Content generation and content correction have evolved to the point where many research and commercial solutions are readily available for different natural language processing (NLP) applications. However, the performance of the available solutions degrades when they are applied to non-NLP data (e.g. specific mobile network vendor data, bioinformatics domain data, network protocols data, mobile applications data, physical circuit specifications, etc.).

[0006] Non-NLP tasks typically involve more structured and domain-specific data, which requires specialized parsing and understanding techniques that may not generalize well across different domains. Furthermore, the lack of contextual richness in non-NLP data, compared to natural language, makes it harder to infer meaning and resolve ambiguities. In addition, the complexity of symbolic representations in non-NLP, such as mathematical expressions or programming code, often requires more sophisticated processing algorithms and semantic interpretation techniques. Furthermore, the consequences of errors in non-NLP tasks can be more severe, necessitating robust error handling mechanisms and extensive testing, which can further impact performance.

[0007] SUMMARY

[0008] An objective of the present disclosure is to provide a mechanism for generating and correcting data comprising non-natural language text.

[0009] The foregoing and other objectives are achieved by the features of the independent claims.

[0010] Further implementation forms are apparent from the dependent claims, the description and the Figures.

[0011] A first aspect of the present disclosure provides a method comprising receiving a data input comprising non-natural language text, processing the data input, whereby to acquire a processed data input, initialising a large language model, LLM, and training the LLM based on a part of the processed data input, whereby to select a prompting strategy for the LLM, pruning the LLM, utilising a multi-task learning process for the pruned language model by training the model on multiple related tasks, generating multiple corrected data inputs, each of the multiple corrected data inputs based on the received data input comprising the non- natural language text, using the LLM, combining the multiple corrected data inputs to obtain a final corrected data input, determining whether the final corrected data input meets a predefined quality threshold based on a comparison with a set of ground-truth data, in response to determining that the final corrected data input meets a predefined quality threshold, outputting the final corrected data input, and categorising the final corrected data input.

[0012] Accordingly, complex non-natural language data can be processed to prepare it for further analysis. Advantageously, the method supports multilingual non-natural language data types. Furthermore, the selected prompting strategy has general applicability for non-natural language LLMs, creating cleaner, more universal prompts (which in turn leads to better content generation). Similarly, the proposed pruning strategy can also be applied to other non-natural LLMs, improving the content generation and training. By training the model on multiple related tasks, training efficiency can be improved. Additionally, the proposed correction approach is agnostic to different input data modalities and characteristics. The data input comprising the non-natural language text may comprise at least one sentence, and processing the data input, whereby to acquire the processed data input may comprise identifying and removing extra characters and special characters from the received data input, parsing the processed text input to extract individual sentences, and outputting the extracted individual sentences.

[0013] The data input comprising the non-natural language text may comprise text written in at least one language, and processing the data input, whereby to acquire the processed data input may further comprise analysing the data input to identify at least one specific language present therein, identifying a main language of the data input, wherein the main language comprises the language most prevalent in the data input, segmenting the data input into language-specific segments based on the identified language, and pruning each segment by identifying and removing portions of text that do not belong to the main language, and outputting the pruned text segments.

[0014] Processing the data input, whereby to acquire the processed data input may further comprise segmenting the pruned text segments into substrings, applying a similarity matching algorithm to each substring to generate similarity pair values for all the substrings above a certain threshold length value, comparing the generated similarity pair values of each substring, whereby to identify duplicate strings, removing the identified duplicate strings to generate a set of deduplicated data inputs, applying a weight to the set of deduplicated data inputs to calculate a set of weighted outputs, and combining the set of weighted outputs to arrive at a final data output.

[0015] The method may further comprise the step of applying a heuristics-based algorithm to the final data output to obtain the processed data input.

[0016] Training the LLM based on the part of the processed data input, whereby to select the prompting strategy for the LLM may comprise initialising a prompting strategy, whereby to provide a first prompt, utilising an output of the first prompt usage as an input of a subsequent prompt, whereby to create a prompt-chain, applying a weak supervision-based approach, WSBA, to the prompt chain, whereby to aggregate prompt predictions based on the prompt chain, and selecting the prompting strategy for the LLM based on the aggregated prompt predictions.

[0017] The method may further comprise the steps of receiving user feedback relating to the selected prompting strategy, and incorporating the received user feedback into both the initialisation of the prompting strategy and the aggregation of the prompt predictions.

[0018] Pruning the LLM may comprise defining a number of neurons to be pruned, wherein the number of neurons to be pruned comprises multiple batches of neurons, stacking a first batch of neurons in a transitional region, adding multiple neurons of a second batch of neurons to each other, and combining the added neurons with the stacked first number of neurons, and pruning a third batch of neurons by employing a series expansion technique while a validation loss is below a threshold.

[0019] Utilising the multi-task learning process for the pruned language model by training the model on multiple related tasks may comprise dividing the processed data input into multiple subsets of data based on characteristics of the data input, such that each subset of the multiple subsets of data comprises data sharing a predefined characteristic, and training the pruned language model using each of the multiple subsets of data as a separate task.

[0020] Combining the multiple corrected data inputs to obtain a final corrected data input may comprise applying seed-text correction, synthesised text correction and proximal policy optimisation to the data input comprising the non-natural language text, respectively, whereby to acquire multiple sets of predictions, and obtaining the final corrected data input based on a comparison between the multiple sets of predictions.

[0021] Obtaining the final corrected data input based on the comparison between the multiple sets of predictions may comprise normalising the predictions in the multiple sets of predictions, scaling the predictions based on a normalised frequency of appearance, selecting first and second ranked predictions of the multiple sets of predictions, and obtaining the final corrected data input based on comparing the first and second ranked predictions.

[0022] Applying the seed-text correction may comprise generating sentences based on the generated multiple data input, applying an adapted beam search algorithm for the generated sentences, and examining whether the generated sentences are valid, wherein a valid sentence comprises a non-empty sentence, an end-of-sequence token and follows a predefined expression.

[0023] The beam search algorithm may be associated with a beam width, and applying the beam search algorithm for the generated sentences may comprise expanding a set of candidate solutions by generating multiple successor states, wherein the successor states are sorted in a decreasing order of cost, for each successor state, calculating a score representing the likelihood of combining the successor state with each candidate sequence in the beam based on a predetermined scoring function, selecting a plurality of top candidates from the set of successor states based on the calculated scores, adding the selected top candidates to a final candidate list, and reducing the beam width by one for each top candidate added to the final candidate list.

[0024] A second aspect of the present disclosure provides a computer program stored on a non-transitory medium and including code instructions, which, when executed on more or more processors, cause the one or more processors to execute the steps of the method described herein.

[0025] A third aspect of the present disclosure provides an apparatus comprising a processor, a memory coupled to the processor, the memory configured to store program code executable by the processor, the program code comprising one or more instructions, whereby to cause the apparatus to receive a data input comprising non-natural language text, process the data input, whereby to acquire a processed data input, initialise a large language model, LLM, and train the LLM based on a part of the processed data input, whereby to select a prompting strategy for the LLM, prune the LLM, utilise a multi-task learning process for the pruned language model by training the model on multiple related tasks, generate multiple corrected data inputs, each of the multiple corrected data inputs based on the received data input comprising the non-natural language text, using the LLM, combine the multiple corrected data inputs to obtain a final corrected data input, determine whether the final corrected data input meets a predefined quality threshold based on a comparison with a set of ground-truth data, in response to determining that the final corrected data input meets a predefined quality threshold, output the final corrected data input, and categorise the final corrected data input.

[0026] These and other aspects of the invention will be apparent from the embodiment(s) described below.

[0027] BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order that the present invention may be more readily understood, embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

[0029] Figure 1 is a flow chart of a method according to an example;

[0030] Figure 2 is a flow chart of data processing according to an example;

[0031] Figure 3 is a flow chart of a prompting selection algorithm according to an example;

[0032] Figure 4 is a flow chart of multi-task learning according to an example;

[0033] Figure 5 is a flow chart of a post-correction method according to an example;

[0034] Figure 6 is a flow chart of an improved beam search according to an example; Figure 7 is a flow chart of a hybrid fusion function according to an example;

[0035] Figure 8 is a flow chart of a method according to another example; and

[0036] Figure 9 is a schematic representation of an apparatus according to an example.

[0037] DETAILED DESCRIPTION

[0038] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.

[0039] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.

[0040] The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and ‘The” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof.

[0041] Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein.

[0042] Content generation tends to create an improved form of an input text without sacrificing the meaning thereof (or sacrificing it in a non-significant manner). Content generation is a natural language processing task that involves condensing a lengthy text document into a shorter, more compact version, while still retaining the most important information and meaning. The aim is to produce meaningful content that accurately represents the contents of the original text in a concise form. Various approaches to text content generation exist, for example, extractive methods that identify and extract important sentences or phrases from the input text, and abstractive methods that generate new text based on the contents of the original text.

[0043] Models that perform extractive content generation essentially pick the most representative sentences and copy them into the generated content summary. In contrast, models that perform abstractive content generation generate new sentences that capture the general idea. While extractive content generation is a binary problem, i.e., the sentence can be either classified as one that should be in the generated content or should not be in there, abstractive content generation is a sequence-to-sequence text generation problem. Compared to extractive content generation, abstractive content generation is significantly more difficult since the machine has to synthesise the information it “reads” into a new form. Content correction involves automatically identifying and rectifying errors in written text, including grammar, spelling, punctuation, and style inconsistencies.

[0044] While content generation and correction are not as challenging for NLP scenarios, issues arise when the language models are to consider non-natural language inputs. The currently available solutions cannot effectively manage queries that are not part of the natural languages, either full or partially. Thus, while the existing libraries and models for spelling correction work well for languages such as English or Spanish, they are not adapted for correcting non-natural language inputs such as programming languages, mathematical expressions, music scores, and more. The presently available solutions are not able to handle distorted inputs (e.g., inputs that contain abbreviations, misspellings, autocomplete, spaces, and similar). Furthermore, as the current solutions tend to generate only “obvious” content, they are not suited towards dealing with more challenging scenarios, such as non-natural language ones. This is due to the fact that the existing solutions target only natural language scenarios to support other domains that are purely based on natural language.

[0045] Furthermore, the solutions based on LLMs cannot fully solve the problem for distorted input queries, slot filling problems and other non-NLP scenarios, such as those relating to programming languages and / or application execution and categorisation.

[0046] According to an example, there is provided a mechanism for content generation and correction of non-natural language inputs. Advantageously, in contrast to the prior art, the solution described herein is based on non-natural language inputs. The solution is applicable to domains such as applications, mobile network vendor data, bioinformatics domain data, network protocols data, mobile applications data, and physical circuit specification. Due to a large presence of non-natural language words and pseudo- mathematical expressions, these areas cannot be handled by prior art solutions. Furthermore, the mechanism described herein is able to automatically create generated content, independent of size and / or length. Compared to the prior art, in addition to the slot-filling task, the present solution handles more variations of non-correct input (for example, misspellings, abbreviations, autocomplete, space-related issues) in a multilingual setting.

[0047] Advantageously, the present solution can also be used to solve the problem of correctly categorising applications (apps) based on their corresponding app description by employing an approach based on text content generation and content correction. App descriptions do not represent natural language.

[0048] Examples in the present disclosure can be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware or the like. Such machine-readable instructions may be included on a computer readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon.

[0049] The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. It shall be understood that each flow and / or block in the flow charts and / or block diagrams, as well as combinations of the flows and / or diagrams in the flow charts and / or block diagrams can be realized by machine readable instructions.

[0050] The machine-readable instructions may, for example, be executed by a machine such as a general-purpose computer, user equipment such as a smart device, e.g., a smart phone, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine-readable instructions. Thus, modules of apparatus (for example, a module implementing a comparator unit, or a firewall structure and so on) may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate set etc. The methods and modules may all be performed by a single processor or divided amongst several processors. Such machine-readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. For example, the instructions may be provided on a non- transitory computer readable storage medium encoded with instructions, executable by a processor.

[0051] Figure 1 is a flow chart of a method according to an example. The method comprises, in block 101, receiving a data input comprising non-natural language text. The data input comprising non-natural language text may comprise, for example, mobile network vendor data, bioinformatics domain data, network protocols data, mobile applications data, physical circuit specifications, programming code (e.g., Python functions, Java classes), markup languages (e.g., HTML, XML), data formats (e.g., JSON, YAML), command-line interface commands (e.g., Is -1), mathematical notations (e.g., equations, formulas), and structured query language queries (e.g., SQL statements). The data input comprising the non-natural language text may also comprise app descriptions. Here, the term “app descriptions” refers to textual content or metadata that provides information about a software application. These descriptions typically include details such as the app's features, functionalities, purpose, compatibility, version history, developer information, user reviews, ratings, and other relevant information. App descriptions serve as a means for users to understand what an app offers and whether it meets their needs before downloading or purchasing it.

[0052] In block 102, the method comprises processing the data input, whereby to acquire a processed data input. To illustrate the processing better, reference is made to Figure 2. Figure 2 is a flow chart of data processing according to an example. The data processing may comprise, for example, a collection stage 201, a syntax cleaning stage 202, a deduplication stage 203, and a knowledge-based filtering stage 204. The collection stage 201 may comprise taking the data input comprising the non-natural language text and extracting meaningful sentences by identifying and removing extra characters, non-query inclusive characters, spaces and special characters from the sentences, and parsing the non-natural language text to extract individual sentences. Special characters may include symbols, punctuation marks, mathematical symbols, and other non-alphabetic or non-numeric characters. Non-query inclusive characters may comprise characters that are not included within a search query or a specific search pattern - in other words, characters that are disregarded or treated separately from the main search or processing criteria in a given context.

[0053] In order to extract the meaningful sentences, a union of special characters may be employed and applied to the input data text. Here, the term “union of special characters” may refer to a set that includes all the unique special characters found across different systems, applications, or languages, used in various contexts, such as programming, data processing, or textual representation. As part of the collection stage 201, the following rule may be applied: if a sentence found within the non-natural language text is larger than 2-word tokens (i.e., the smallest units of text that the LLM can process), digits and extra spaces may be removed, followed by simple deduplications in order to obtain “clean” versions of the pre-collected non-NLP text data points. Then, data collections cleaning may be obtained by stacking and removing collections using the SoftTFIDF method, i.e., by applying it to multiple two-pair sentences to acquire a similarity score between them. In a scenario where the pairs’ similarity score indicates that they are equally similar, the pair having a lower word mover’s distance (WMD; a technique to measure semantic word similarity, where the pair having the lower WMD signifies greater similarity between the text) may be selected to proceed with. Finally, the tokenised output may be used and cross-checked against dictionary forms.

[0054] During the syntax cleaning stage 202, multilingual data in the non-NLP text may be recognised and the specific languages may be identified. The language identification may be performed using any suitable method, for example, by using a “langdetect” library. Afterwards, pruning of the “minor” language parts of the text may be performed. That is, by employing language identification, the “main” language (i.e., the language most prevalent in the input text) may be identified, and the “minor” (i.e., languages other than the main language) language parts may be pruned. After the pruning, filtering on character level may be employed, during which bad / incorrect characters may be identified through the use of pre-defined thresholds on sub-token level and comparing those to a dictionary of clean slate tokens for different languages. In other words, they may be compared to a collection of special tokens used in fine-tuning LLMs across various languages.

[0055] During the deduplication stage 203 , firstly, all substrings of each sentence identified in the non-NLP input text may be extracted, and a similarity matching algorithm may be applied to each sentence. For example, a modified version of the Fuzzy Min-hash algorithm may be applied in order to create different sequences of different lengths without duplications and without minority languages. Afterwards, a linear weighted combination of the outputs may be obtained by applying a weight to the set of outputs. The weight(s) may be obtained during a training stage of the LLM. The final result may be obtained by using a linear weighted combination of the previously obtained outputs.

[0056] Finally, during the filtering stage 204, a heuristic-based algorithm may be employed. The heuristic-based algorithm may employ non-overlapping, contiguous parts of the non-NLP text by removing stop-words, stemming, splitting, and employing TF-IDF and Siamese LSTMs to detect and filter out similar objects to which the non-NLP text refers to. For example, if the non-NLP text comprises app descriptions, similar apps may be detected and filtered out. Afterwards, prominent words-based filtering may be applied, followed by adequate sampling-based filtering to obtain the processed data input. While similar to the syntax-cleaning stage 202 (during which pruning was performed), the filtering stage 204 may comprise the use of more radical filtering based on context and content, whereas the syntax-cleaning stage 202 may mostly relate to superficial filtering based on errors and minor language faults.

[0057] As a result of the stages 201-204, the LLM may become more and more resilient in the non-NLP language domains. Importantly, a “clean slate” may be created with respect to the data entry. The particular order of steps outlined above results in the best performance of the LLM. In order to eliminate over-confusing inputs, it is preferred that the stages 201-204 do not take place simultaneously. However, the stages 201-204 may be skipped during the interference stage, i.e., during a phase during the LLM’s training or operation where external factors or inputs disrupt the model's performance or output quality.

[0058] Referring back to Figure 1, the method comprises, in block 103, initialising the LLM and, in block 104, training the LLM on a part of the processed data input, whereby to select a prompting strategy for the LLM. For better understanding of the prompting strategy selection, reference will now be made to Figure 3, which is a flow chart of a prompting selection algorithm according to an example.

[0059] Within the context of LLMs, a prompting strategy may refer to providing specific input prompts or cues to guide the model's text generation process towards desired outcomes. These prompts typically consist of partial sentences, keywords, or instructions tailored to the task or objective at hand. In order to select the most suitable prompting strategy for the LLM, a learning algorithm that dynamically selects the optimal strategy based on pre-trained user feedback and corresponding automated evaluation may be applied.

[0060] Figure 3 is a flow chart of a prompting selection algorithm according to an example. In block 310, the prompting selection algorithm may perform data collection. The data collection may comprise, for example, scraping data from many different sources that are available in various European languages, as well as those available in Chinese language. Advantageously, the data may relate to different data types in which non-natural language data is present, for example, specific mobile network vendor data, bioinformatics domain data, network protocols data, mobile applications data, physical circuit specifications, and others. Block 320 may comprise preprocessing. The preprocessing may comprise the processing shown in Figure 2 (and described above in relation thereto). Block 330 may comprise labelling, while block 340 may comprise refinement. In particular, blocks 330 and 340 may relate to the ground-truth data information and may be performed semi-manually (or semi- automatically). This is due to fact that preprocessing sometimes “distorts” the real meaning of the data, and thus needs to be revisited and relabelled accordingly. Block 350 may refer to an evaluation dataset, i.e., an independent dataset split. Block 360 may comprise prompt generation via aggregation. To better illustrate the nature of the prompt generation via aggregation, a breakdown of the block 360 is provided below the blocks 310-360. For the prompt generalisation via aggregation, block 301 may comprise a prompt initialisation block in which the prompting may be initiated / updated, and a refinement strategy (i.e., iteratively adjusting and improving the input prompts to guide the model 302 towards generating more desirable outputs) and a variety of patterns may be defmed / provided to the algorithm. The prompting may start by providing a few examples with the associated reasoning process. The output of one prompt may be utilised as an input of a subsequent prompt in the chain, resulting in a more advanced prompting approach. Advantageously, the algorithm preserves context in a discussion (in order to support generalisation and correction). A set of prompts may be produced using different patterns, to be applied to different generation and correction tasks.

[0061] The LLM base model (block 302) may be recursively applied using a chain of functional prompts called a prompt-chain. These prompts may employ a task-agnostic generation and correction template to all inputs in the tasks. Different prompt-chains may be constructed, with each unique prompt-chain providing a different view of the task and emphasising different aspects thereof. To aggregate the prompt predictions reliably, in the response block 303, a weak supervision-based approach (WSBA) may be applied by learning the model without labelled data.

[0062] The WSBA may include a rule attention teacher-student network. The rule attention teacher-student network may refer to a type of neural network architecture designed for knowledge distillation, particularly in scenarios where interpretability and adherence to predefined rules are crucial. In this setup, the "teacher" network, typically a complex model, may provide supervision to a simpler "student" network by emphasizing certain parts of the input data based on predefined rules or patterns. The rule attention mechanism may guide the student network's attention towards relevant features or information. The WSBA may combine multiple rules and model predictions with per-instance specific weights dependent on the corresponding generation and correction context (supported by categorisation). A semi-supervised learning approach may be employed for training without any knowledge about the nature or structure of the weak prompt rules. Thus, given the varied dependencies among the prompt-chains, some of the last outputs may be aggregated, while considering the accuracy and dependency properties (without relying on labelled data). As such, the algorithm may be able to scalably obtain multiple effective prompts given a generation and correction input, and combine the intermediate answers to these prompts into a final answer.

[0063] Finally, in block 304, a user feedback refinement check may be performed. The user feedback may relate to the prompting strategy selected as a result of the outputs of the algorithm and may be incorporated into both the initialisation of the prompting strategy (block 301), as well as the aggregation of the prompt predictions performed in the response block 303.

[0064] Referring back to Figure 1, in block 105, the method comprises pruning the LLM. Pruning the LLM may comprise applying a combination of fixed pruning and learned pruning. Firstly, a number of neurons to be pruned may be defined, followed by pruning the neurons in batches and finetuning the LLM after the pruning has been performed. Initially, some neurons may be stacked in the transitional regions, while other neurons may be added firstly and then combined with the stacked neurons to achieve fixed pruning. After k epochs, a different pruning strategy may be applied - the neurons may be continuously pruned as long as a validation loss is below a threshold (thus called “learned pruning”). For the learned pruning, a series expansion technique may be utilised. In an example, for learning pruning, an expansion of a Maclaurin series may be employed by utilising a weighted combination of a first-order and a second-order Maclaurin expansion and performing iterative pruning. In contrast to a Taylor series that approximates a function with a polynomial around an arbitrary point different to 0, the Maclaurin series represents an approximation of a function with a polynomial around 0.

[0065] For the validation loss, different threshold values may be utilised for every layer of the LLM. The threshold values may be trainable, i.e., a training procedure may find optimal thresholds alongside the layer weights during the finetuning. A normalised histogram may be used for the thresholds as an estimate of the probability function in which for each threshold value a mean and a variance can be obtained and utilised later on. The upper and lower weights limits of the thresholds may be set to the minimum and the maximum values of a gradient descent optimisation to minimise the difference between the obtained and the projected results. The threshold interval may be set at, for example, 0.1, so as to ensure precision in identifying the optimal threshold.

[0066] In block 106, the method comprises utilising a multi-task learning process for the pruned language model by training the model on multiple related tasks. The multi-task learning process is illustrated in Figure 4. The process may involve training the LLM 402 by utilising multiple related tasks 401-1, 401-2, ..., 401-n simultaneously.

[0067] The multi-task learning process may employ both hard (in which hidden layers are shared between the tasks 401-1, 401-2, ..., 401-n and the output layers, which are specific to a task, fixed at the same time) and soft parameter sharing of hidden layers (where each task 401-n has its own model with its own parameters). Employing both hard and soft parameter sharing of hidden layers is advantageous because it helps more complex data samples and scenarios (i.e., it can better separate relevant features from irrelevant ones).

[0068] The approach may jointly optimise the network parameters and a categorical random variable (uni, n) for each layer and each task in Tn, where Tn represents a set of n=7 tasks (however, the invention is not limited thereto). Here, the categorical random variable CRV may represent which tasks are shared, skipped, or done individually by a particular block for the multiple tasks 401-n. As the task variable is non-differentiable, a modified version of Gumbel-Softmax sampling (MGS) may be employed in order to optimise it.

[0069] Gumbel-Softmax sampling is a method used for generating text probabilistically during text generation tasks. Gumbel-Softmax sampling introduces a temperature parameter that controls the degree of randomness in the sampling process. Lower temperatures lead to more deterministic sampling, where the most likely token is selected with higher probability, while higher temperatures result in more exploratory sampling, where less likely tokens have a higher chance of being selected. This method allows for more diverse and varied text generation while still maintaining control over the sampling process. Gumbel distribution may be sampled from a uniform distribution, normalised by taking in a low-min and high-max normalisation approach, and finally interpolated. Low-min and high-max normalisation ensures that all values are scaled proportionally, allowing for better comparison and interpretation of data. Specifically, in a "low-min and high-max" normalization approach, the minimum value in the original range is transformed to 0, and the maximum value is transformed to 1 , while all other values are scaled accordingly based on their relative positions within the original range.

[0070] Interpolation may be achieved between discrete one-hot-encoded distributions (i.e., distributions in which each category is represented by a vector of binary values, where only one element is "hot" or set to 1, indicating the presence of that category, and all other elements are "cold" or set to 0, indicating the absence of those categories). During training for small temperature values (e.g., r <= 0.5), the expected value of the distributions may approach the expected value of the corresponding CRV variable with the same logits (i.e., the raw outputs of the model before it is transformed into a probability distribution over the vocabulary). Thus, this approximation may be used to train / infer on logits. For each task, a corresponding range of temperatures may be defined. The loss function may be modelled as a sum of two losses: a sparsity loss to achieve model compactness, and a sharing loss that achieves sharing across multiple of tasks 401-n. The sequential order of tasks presented in Figure 4 follows an adapted correlation covariance coefficient (ACC) order powered by a normalized dataset size, followed by sorting in a descending order.

[0071] This approach helps the LLMs to generalize better to different tasks, since a hypothesis space that performs well for a sufficiently large number of training tasks will also perform well for learning new tasks, as long as the new tasks share the same “context”. Since different tasks have different noise patterns, an LLM that learns multiple tasks simultaneously is able to learn a more general representation by averaging the noise patterns. In contrast, learning a single task only would come with a risk of overfitting the LLM to that specific task.

[0072] Tasks 401-n may refer to content generation data of different complexity, to be used in finetuning process of the LLM 402. The content generation data may comprise, for example, the input data comprising the non-natural language text. The data may be split into a plurality of subsets based on their inherent features. For example, the data may be split into the plurality of subsets, wherein each of the subsets comprises data of similar complexity. Each of the subsets may relate to a separate task. As described above, the subsets of data (i.e., the tasks) may then be used for training the LLM 402.

[0073] Referring once again back to Figure 1, in block 107, the method comprises generating, using the LLM, multiple corrected data inputs, each of the multiple corrected data inputs based on the received data input comprising the non-NLP text. In other words, now that the received data input has been processed and the LLM has been initialised and finetuned, the LLM is ready to produce outputs. At this point, the post-correction stage may begin, as described in more detail below.

[0074] The method comprises, in block 108, combining the multiple corrected data inputs to obtain a final corrected data input (i.e., hybrid fusion described in more detail below). In block 109, the method comprises determining whether the final corrected data input meets a predefined quality threshold based on a comparison with a set of ground-truth data. Here, the term “groundtruth data” refers to the authentic, manually labeled or annotated data that serves as the authoritative reference for evaluating the performance of models or algorithms. Ground truth data provides the correct answers or labels for the given inputs, enabling the assessment of model accuracy and other performance metrics.

[0075] The method comprises, in block 110, in response to determining that the final corrected data input meets a predefined quality threshold, outputting the final corrected data output. In block 111 , the method comprises categorising the final corrected data input. This categorisation process enables efficient and accurate placement of the entity the input data relates to within relevant sections or categories. For example, if the data input comprising the non-natural text relates to an app description, based on the final corrected data input, the app may be placed within relevant sections or categories of an app distribution platform, enhancing discoverability and user engagement within the app marketplace.

[0076] Figure 5 is a flow chart of a post-correction method according to an example. The post-correction may comprise a seed-text correction stage 510, a synthesised text correction stage 520, as well as rejection sampling, proximal policy optimisation, and hybrid fusion stages (shown in Figure 6). Here, the term “seed-text correction” refers to the process of refining or correcting a given initial text input using the capabilities of the language model. In this approach, a user provides an initial text prompt, often containing errors or imperfections, and the LLM analyses it to generate a corrected or improved version.

[0077] For the seed-text correction stage 510, a string “correct:” may be added to the end of the obtained string (i.e., the processed data input) in order to trigger the LLM to generate corrected content. Following this, sentences may be generated for the obtained text. Then, a beam search algorithm 520 may be adopted for text generation together with disjunctive positive constraints to include delimiters and between the corrected generated contents. The post-correction method may comprise examining whether the generated corrected content strings are valid. A valid string may comprise a string that ends with “< / s>”, an end-of-sequence token, and, for English language, follows the regular expression “[0-9A-z(), While English language was mentioned, the skilled person would appreciate that the invention is not limited thereto, and languages such as German, Italian, French and / or Spanish can also be considered. For Chinese language, the valid string may follow the regular expression A beam search algorithm is a heuristic search technique used for generating sequences of tokens, such as words or characters, based on the probabilities assigned by the model. The beam search starts with an initial sequence, typically consisting of a start token or a few words of input. At each step, the algorithm generates a set of possible next words or tokens based on the probabilities assigned by the language model. It evaluates these candidates and selects the top-k most promising ones, where k is the beam width. After expanding the candidates, the algorithm prunes the less promising ones, retaining only the top-k candidates for further expansion. Figure 6 is a flow chart of an improved beam search according to an example.

[0078] The improved beam search may employ, at each level, the successors’ states (i.e., potential next states) and sort them in a decreasing order of cost. The states may be expanded with the beam increase: the greater the beam width, the fewer states may be filtered out. In block 601, the beam search may select the best N sentences so far and consider the probabilities of the combination of all of the preceding keywords, along with the keyword in the current position. The process may be repeated until the beam is zero - i.e., when the search has stopped - and the keyword with the highest probability of combination has been selected out of the final candidate list. After that, the corrected generated content may be stored into a final candidate list and the beam width may be reduced by one.

[0079] After the final candidate list has been reduced, a new beam search may be initiated in block 602. During this beam search, only a specific number of candidates having the “same history” may be allowed. The maximum number of candidates C may be denoted using num_can (i.e., the number of candidate sequences retained at each step of the search process). A candidate (end) may be removed from the candidate list if a candidate with a “better” history (i.e., a candidate containing the same predecessor words) is present.

[0080] Following this, in block 603, the candidate(s) that are worse by a specific threshold when compared to the best active candidate may be discarded. Given a pruning threshold (PT) and an active candidate list C, a candidate (end) may be discarded if score(cnd) <= max{score(c)}-PT [c e C], After this, blocks 602 and 603 may be performed once more (in a sequence); the beam search may stop once a predefined number of corrected generated contents has been reached. Otherwise, the procedure goes back to block 601.

[0081] Figure 7 is a flow chart of a hybrid fusion function according to an example. In order to fuse the seed-text correction 701, synthesised text correction 702 and proximal policy optimisation 703, top g predictions may be considered. Firstly, the predictions may be normalised (for example, in a [0,1] interval) and scaled with the normalised frequency of appearance. The first and second ranked guesses may be considered, based on the threshold difference between them. In other words, by observing the differences between the first top two ranked predictions of the joint seed-text correction 701, synthesised text correction 702 and proximal policy optimisation 703 in many confidence pairs, it may be concluded that, for these differences (beyond reasonable large thresholds), the correct prediction is the first ranked prediction, based on either the seed-text correction, synthesised text correction and proximal policy optimisation or both. If the difference between the first and the second ranked guesses is below the predefined threshold, a union (i.e., sum) of the normalised prediction from both parts may be taken, and the prediction having the most occurrences may be selected. The predictions may then be aggregated in block 704, resulting in an output 705.

[0082] In the case of a “tie-break”, low-rank multimodal fusion may be employed. Predictions from the seed-text correction, synthesised text correction and proximal policy optimisation may be observed as vectors. The predictions vectors may be decomposed (using an appropriate scaling) into multiple (e.g., two) sets, denoted in the equation below as A, B and C. In the above equation, the parametrisation may be specified by the rank of the vector which makes the sets’ decomposition valid. The generated fused vectors may be defined in the above equation, where II is the element- wise product over a sequence of vectors, is the proposed product solution, w_m is the weight matrix for modality m G (A, B, C), p is the rank, and f_m are the input vector representations. Since the fusion explores the interdependency between the predictions, eventually, the first prediction may be selected as the final prediction. “w_m” may be obtained during the training stage by creating a Hadamard product of the decomposed vector predictions for each modality.

[0083] Figure 8 is a flow chart of a method according to another example. In particular, Figure 8 aims to show the above-described individual modules of the invention working together and to show the generation loop for the LLM finetuning and correction. In other words, Figure 8 shows the journey of the input data 801 comprising a non-natural language text to becoming a correctly generated input 802. Firstly, the input data 801 subjected to processing 803, described above in relation to Figure 2. Following that, model initialisation 804 may take place. Model initialisation 804 may comprise initialising the large language model to be used, as well as configuring and setting up the model. Followed by the model initialisation 804, finetuning 805 may take place. The adjustable prompt storage may comprise a pool of prompts used to train the prompt algorithm (i.e., perform the learnable adapter prompting, described above in relation to Figure 3). The LLM finetuning for generation and correction may comprise a generation loop 807. The output of finetuning 805 may be subjected to post-correction 806 (described above in relation to Figure 5). The result of the post-correction 806 may comprise a correctly generated input 802.

[0084] In particular, as part of the post-correction 806, the results of the seed-text correction, synthesised text correction and proximal policy optimisation may be merged into a single semantic interpretation, as discussed above in relation to Figure 7. By combining the strengths of these complementary approaches, accuracy and robustness to the problems that affect individual modalities can be improved. In this late fusion, the predictions may be combined on a semantic level. As each modality is trained separately, the joint probability learning of the modalities may not be done explicitly.

[0085] Figure 9 is a schematic representation of an apparatus according to an example. The apparatus 900 may comprise a processor 903, and a memory 905 coupled to the processor 903 and configured to store instructions or program code 907, executable by the processor 903. The apparatus 900 may comprise the program code 907 arranged to cause the apparatus to perform the method described herein.

[0086] According to an example, machine-readable instructions can be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operations to produce computer-implemented processing, thus the instructions executed on the computer or other programmable devices provide an operation for realizing functions specified by flow(s) in the flow charts and / or block(s) in the block diagrams.

[0087] Further, the teachings herein may be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, the computer software or product being stored in a storage medium and comprising a plurality of instructions, e.g., machine readable instructions, for making a computer device implement the methods recited in the examples of the present disclosure.

[0088] In some examples, some methods can be performed in a cloud-computing or network-based environment. Cloud-computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a web browser or other remote interface of the user equipment for example. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment. While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these exemplary embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable-storage media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the exemplary embodiments disclosed herein. In addition, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another.

[0089] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.

Claims

CLAIMS1. A method comprising : receiving a data input comprising non-natural language text (101); processing the data input, whereby to acquire a processed data input (102); initialising a large language model, LLM (103), and training the LLM based on a part of the processed data input, whereby to select a prompting strategy for the LLM (104); pruning the LLM (105); utilising a multi-task learning process for the pruned language model by training the model on multiple related tasks (106); generating multiple corrected data inputs, each of the multiple corrected data inputs based on the received data input comprising the non-natural language text, using the LLM (107); combining the multiple corrected data inputs to obtain a final corrected data input (108); determining whether the final corrected data input meets a predefined quality threshold based on a comparison with a set of ground-truth data (109); in response to determining that the final corrected data input meets a predefined quality threshold, outputting the final corrected data input (110); and categorising the final corrected data input (111).

2. The method of claim 1 , wherein the data input comprising the non-natural language text comprises at least one sentence, wherein processing the data input, whereby to acquire the processed data input (102) comprises: identifying and removing extra characters and special characters from the received data input; parsing the processed text input to extract individual sentences; and outputting the extracted individual sentences.

3. The method of claim 2, wherein the data input comprising the non-natural language text comprises text written in at least one language, wherein processing the data input, whereby to acquire the processed data input (102) further comprises: analysing the data input to identify at least one specific language present therein; identifying a main language of the data input, wherein the main language comprises the language most prevalent in the data input; segmenting the data input into language-specific segments based on the identified language, and pruning each segment by identifying and removing portions of text that do not belong to the main language; outputting the pruned text segments.

4. The method of claim 3, wherein processing the data input, whereby to acquire the processed data input (102) further comprises: segmenting the pruned text segments into substrings; applying a similarity matching algorithm to each substring to generate similarity pair values for all the substrings above a certain threshold length value; comparing the generated similarity pair values of each substring, whereby to identify duplicate strings; removing the identified duplicate strings to generate a set of deduplicated data inputs; applying a weight to the set of deduplicated data inputs to calculate a set of weighted outputs; and combining the set of weighted outputs to arrive at a final data output.

5. The method of claim 4, further comprising the step of: applying a heuristics-based algorithm to the final data output to obtain the processed data input.

6. The method of any preceding claim, wherein training the LLM based on the part of the processed data input, whereby to select the prompting strategy for the LLM (104) comprises: initialising a prompting strategy, whereby to provide a first prompt; utilising an output of the first prompt usage as an input of a subsequent prompt, whereby to create a prompt-chain; applying a weak supervision-based approach, WSBA, to the prompt chain, whereby to aggregate prompt predictions based on the prompt chain; and selecting the prompting strategy for the LLM based on the aggregated prompt predictions.

7. The method of claim 6, further comprising the steps of: receiving user feedback relating to the selected prompting strategy; and incorporating the received user feedback into both the initialisation of the prompting strategy and the aggregation of the prompt predictions.

8. The method of any preceding claim, wherein pruning the LLM comprises: defining a number of neurons to be pruned, wherein the number of neurons to be pruned comprises multiple batches of neurons; stacking a first batch of neurons in a transitional region; adding multiple neurons of a second batch of neurons to each other, and combining the added neurons with the stacked first number of neurons; and pruning a third batch of neurons by employing a series expansion technique while a validation loss is below a threshold.

9. The method of any preceding claim, wherein utilising the multi-task learning process for the pruned language model by training the model on multiple related tasks (106) comprises: dividing the processed data input into multiple subsets of data based on characteristics of the data input, such that each subset of the multiple subsets of data comprises data sharing a predefined characteristic; and training the pruned language model using each of the multiple subsets of data as a separate task.

10. The method of any preceding claim, wherein combining the multiple corrected data inputs to obtain a final corrected data input comprises: applying seed-text correction, synthesised text correction and proximal policy optimisation to the data input comprising the non-natural language text, respectively, whereby to acquire multiple sets of predictions; and obtaining the final corrected data input based on a comparison between the multiple sets of predictions.

11. The method of claim 10, wherein obtaining the final corrected data input based on the comparison between the multiple sets of predictions (108) comprises: normalising the predictions in the multiple sets of predictions; scaling the predictions based on a normalised frequency of appearance; selecting first and second ranked predictions of the multiple sets of predictions; and obtaining the final corrected data input based on comparing the first and second ranked predictions.

12. The method of any one of claims 10 or 11, wherein applying the seed-text correction comprises: generating sentences based on the generated multiple data input;applying an adapted beam search algorithm for the generated sentences; and examining whether the generated sentences are valid, wherein a valid sentence comprises a non-empty sentence, an end-of-sequence token and follows a predefined expression.

13. The methods of claim 12, wherein the beam search algorithm is associated with a beam width, wherein applying the beam search algorithm for the generated sentences comprises: expanding a set of candidate solutions by generating multiple successor states, wherein the successor states are sorted in a decreasing order of cost; for each successor state, calculating a score representing the likelihood of combining the successor state with each candidate sequence in the beam based on a predetermined scoring function; selecting a plurality of top candidates from the set of successor states based on the calculated scores; adding the selected top candidates to a final candidate list; and reducing the beam width by one for each top candidate added to the final candidate list.

14. A computer program stored on a non-transitory medium and including code instructions, which, when executed on more or more processors, cause the one or more processors to execute the method of any of claims 1 to 13.

15. An apparatus (900) comprising: a processor (901); a memory (902) coupled to the processor (901), the memory (902) configured to store program code (903) executable by the processor (901), the program code (903) comprising one or more instructions, whereby to cause the apparatus to: receive a data input comprising non-natural language text; process the data input, whereby to acquire a processed data input; initialise a large language model, LLM, and train the LLM based on a part of the processed data input, whereby to select a prompting strategy for the LLM; prune the LLM; utilise a multi-task learning process for the pruned language model by training the model on multiple related tasks; generate multiple corrected data inputs, each of the multiple corrected data inputs based on the received data input comprising the non-natural language text, using the LLM; combine the multiple corrected data inputs to obtain a final corrected data input; determine whether the final corrected data input meets a predefined quality threshold based on a comparison with a set of ground-truth data; in response to determining that the final corrected data input meets a predefined quality threshold, output the final corrected data input; and categorise the final corrected data input.

Images