Device and method for end-to-end summarization and keyword extraction for non-natural language input

Abstract

Described is a data processing device for performing end-to-end summarization and keyword extraction of input descriptions, the data processing device having one or more processors configured to: receive (1201) an input description (301) comprising non-natural language content; process (1202) the input description to extract one or more keywords from the input description; process (1203) the input description to form a summarized output of the input description in dependence on the one or more keywords extracted from the input description; and based on the extracted keywords and the summarized output, classify (1204) the input description (301) into one or more of a plurality of classes. A corresponding method (1200) is also disclosed. This may allow for the processing of input descriptions comprising non-natural language content to perform summarization and keyword extraction in an end-to-end manner, and for the final summarization result to be used to classify the input descriptions. (3)

Description

[0001] DEVICE AND METHOD FOR END-TO-END SUMMARIZATION AND KEYWORD EXTRACTION FOR NONNATURAL LANGUAGE INPUT

[0002] FIELD OF THE INVENTION

[0003] This invention relates to natural language processing operations, in particular to text summation and keyword extraction of input descriptions.

[0004] BACKGROUND

[0005] Text summarization and keyword extraction have evolved to the point where many research and commercial solutions are readily available for different natural language processing (NLP) applications. Although this has led to some promising results, performance is degraded once applied to non-natural language data, such as specific mobile network vendor data, bioinformatics domain data, network protocols data, mobile applications data and physical circuit specifications, thus making such approaches generally unusable for these applications.

[0006] The aim of summarization is to create a shorter form of an input text without sacrificing the meaning, or sacrificing it to a minimum. It is an NLP task that involves condensing a lengthy text document into a shorter, more compact version while still retaining the most important information and meaning. The goal is to produce a summary that accurately represents the content of the original text in a concise form.

[0007] There are different approaches to text summarization. Extractive summarization methods, as schematically illustrated in Figure 1(a), identify and extract important sentences or phrases from the text of the original document 101 to form the summary 102. Abstractive summarization methods, as schematically illustrated in Figure 1(b), generate new text for the summary 104 based on the content of the original text in the document 103. Models that perform extractive summarization essentially pick the best most representative sentences and copy them into a summary. Models that perform abstractive summarization generate new sentences that capture general ideas. Extractive summarization is a binary classification problem in which the sentence is either classified as “should be in the summary” or “should NOT be in the summary”. Abstractive summarization is a sequence-to- sequence text generation problem. This is significantly more difficult than extractive summarization, since the machine has to synthesize the information it “reads” into a new form.

[0008] Keyword extraction identifies keyworks from the input text.

[0009] The main drawbacks of prior approaches lie in the fact they are purely target for natural languages applications (i.e. language that occurs naturally in a human community, such as English, French or Spanish). Another disadvantage of the prior art relates to inability to automatically create summaries (dependent on size).

[0010] Prior approaches based on solving keyword extraction problems do not reflect application distribution platform (for example, the Huawei AppGallery) domain classes and also do not deal with distorted inputs (for example, inputs that contain abbreviations, misspellings, autocomplete, spaces, etc.). Moreover, current solutions usually detect only “obvious” summaries while those more challenging ones that do not belong to the natural language are usually poorly handled. Intuition behind this lies in the fact that the solutions so far are usually target to the natural language scenarios to support other domains that are purely based on natural language. Previous solutions based on large language models (LLMs) cannot fully solve the problem either for distorted input queries and for application domain target scenarios.

[0011] Furthermore, one problem of approaches that correct input text is that they cannot generally manage queries that are not part of the natural languages, either in full or partially. Thus, the existing libraries and approaches for spelling correction work well for general English, French, Spanish, Italian, etc. but are not adapted for a particular domains, such as the application name domain.

[0012] It is desirable to develop an approach that can overcome at least some of the above issues.

[0013] SUMMARY OF THE INVENTION

[0014] According to one aspect there is provided a data processing device for performing end-to-end summarization and keyword extraction of input descriptions, the data processing device having one or more processors configured to: receive an input description comprising non-natural language content; process the input description to extract one or more keywords from the input description; process the input description to form a summarized output of the input description in dependence on the one or more keywords extracted from the input description; and based on the extracted keywords and the summarized output, classify the input description into one or more of a plurality of classes.

[0015] This may allow for the processing of input descriptions comprising non-natural language content to perform summarization and keyword extraction in an end-to-end manner, and for the result to be used to classify the input descriptions. This approach has applications in a variety of fields, including in mobile networks and mobile data applications, bioinformatics and for the processing of physical circuit specifications.

[0016] The input description may comprise natural language (language that occurs naturally in a human community, such as English, French or Spanish) and non-natural language (language that does not occur naturally in a human community) content. The input description may comprise non-natural language words. The input description may comprise natural language words. The non-natural language content may comprise application domain language content.

[0017] The one or more processors may be further configured to pre-process the input description by performing one or more of the following:

[0018] (i) extracting one or more sentences from the input description by removing one or more of the following from the input description: extra characters, non-query inclusive characters, spaces, special characters and duplicated content;

[0019] (ii) identifying content corresponding to a primary language and one or more minor languages in the input description and remove content corresponding to the one or more minor languages and / or bad characters from the input description;

[0020] (iii) extracting one or more sub-strings from the input description, apply a Fuzzy Minhash algorithm to each substring to form respective outputs, and take a linear weighted combination of the outputs; and

[0021] (iv) filtering the input description based on content and / or context of the input description.

[0022] These pre-processing steps may provide a viable way to process complex non-natural language data and make it ready for further analysis. The approach can support multilingual non-natural language data types.

[0023] The step of processing the input description to form a summarized output of the input description may comprise inputting the input description to a large language model, wherein the large language model comprises a neural network finetuned to form the summarized input from the input description. The neural network may comprise output layers and hidden layers. Finetuning of the neural network may comprise performing multi-task training in which hidden layers of the neural network are shared and output layers of the neural network are fixed specific to a task of multiple tasks in the multi-task training. The LLM finetuning method including multitask learning is generalizable to a multitude of complementary tasks. The approach is applicable and extendable to a variety of tasks of different kinds of data to be understood and summarized. At the same time, it may contribute to training efficiency.

[0024] The step of processing the input description to form a summarized output of the input description may comprise implementing a prompting strategy, wherein the one or more processors are configured to select the prompting strategy from a plurality of possible prompting strategies. The one or more processors may be configured to select the prompting strategy by applying a selection algorithm to determine which of the plurality of possible prompting strategies are best-performing for the input description. This approach for prompting has general applicability for non-natural language large language models. This can result in the use of cleaner, more universal prompts which leads to better summarization performance.

[0025] The one or more processors may be configured to prune the neural network by applying one or more fixed pruning steps and one or more learned pruning steps. In the one or more learned pruning steps, the one or more processors may be configured to continuously prune the neural network when a validation loss is below a threshold. The one or more learned pruning steps may comprise applying a weighted combination of first-order and second-order Maclaurin expansion series. This pruning strategy has general applicability for LLMs. This can improve training and allow for improved summarization.

[0026] The one or more processors may be configured to: append a data string to the end of the received input description, wherein the appended data string acts to trigger a further large language model to generate keywords from the input description; and apply the further large language model to the input description to generate one or more sentences from which the keywords are generated.

[0027] The one or more processors may be further configured to determine whether a generated keyword is valid by determining whether the following are true: the string comprising the generated keyword is non-empty string ending with < / s>; the string comprising the generated keyword is an end-of-sequence token; and for non-Chinese languages, the string comprising the generated keyword follows the regular expression [0-9A-zQ,.- ;’]*, or for Chinese language, the string comprising the generated keyword follows the regular expression [0-9\p{Han}(),.-

[0028] The one or more processors may be further configured to apply a beam search algorithm to the generated keywords to refine the keywords to be output. This keyword extraction method can support multilingual non-natural language data types, and may improve the performance of the overall framework.

[0029] The one or more processors may be configured to: receive refined keywords; receive keywords generated by applying a further keyword selection method to the input description; and combine the refined keywords and the keywords generated by applying the further keyword selection method in one or more fusion stages to form final output keywords. The further keyword selection method may comprise applying a TextRank algorithm (for example as described in Rada Mihalcea and Paul Tarau, “TextRank: Bringing Order into Text”, Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics) to the input description. The fusion function may be agnostic to different input data modalities and characteristics. Additionally, the function may be applied to different data types. The fusion may outperform each individual keyword extraction modality on its own.

[0030] The step of processing the input description to form a summarized output of the input description in dependence on the one or more keywords extracted from the input description may comprise filtering an initial summarized output (which may be the output of the finetuned large language model) using keywords formed from the input description to form a final summarized output. The final summarized output may be used for the classification. The initial input description may be filtered using the final output keywords generated from the one or more fusion stages as described above. Filtering the initial summarized output based on the keywords may improve the accuracy of the final summarized output.

[0031] The non-natural language content may comprise content corresponding to multiple languages. The input description may also comprise natural language content from multiple natural languages. This may allow the approach to be used to process complex non-natural language input.

[0032] The input description may further comprise pseudo-mathematical and / or code content. This may allow input descriptions for further applications to be processed.

[0033] The input description may comprise mobile network data, bioinformatic domain data, network protocol data, mobile application data or a physical circuit specification. This may allow input descriptions for further applications to be processed. The input descriptions may comprise other data types than those listed herein, which are merely examples.

[0034] The input description may be a description of a software or mobile application. The data processing device may be communicatively connected to an application distribution platform. The one or more processors may be configured to classify the input description into one or more categories of the application distribution platform. This may allow for hierarchical categorization in application distribution platforms.

[0035] According to another aspect, there is provided a computer-implemented method for performing end-to-end summarization and keyword extraction of input descriptions, the method comprising: receiving an input description comprising non-natural language content; processing the input description to extract one or more keywords from the input description; processing the input description to form a summarized output of the input description in dependence on the one or more keywords extracted from the input description; and based on the extracted keywords and the summarised output, classifying the input description into one or more of a plurality of classes.

[0036] This method may allow for the processing of input descriptions comprising non-natural language content to perform summarization and keyword extraction in an end-to-end manner, and for the result to be used to classify the input descriptions. This approach has applications in a variety of fields, including in mobile networks and mobile data applications, bioinformatics and for the processing of physical circuit specifications.

[0037] According to another aspect, there is provided a computer program which, when executed by a computing device, causes the computing device to perform the method described above.

[0038] According to a further aspect, there is provided a data carrier storing in non-transient form the computer program described above. BRIEF DESCRIPTION OF THE FIGURES

[0039] The present invention will now be described by way of example with reference to the accompanying drawings.

[0040] In the drawings:

[0041] Figure 1(a) schematically illustrates extractive summarization;

[0042] Figure 1(b) schematically illustrates abstractive summarization;

[0043] Figure 2 shows an example of an application description predominantly in English with different parts of non-natural language content and special character “words” misspelled or distorted;

[0044] Figure 3 schematically illustrates an example of an overall system for end-to-end summarization and keyword extraction;

[0045] Figure 4 schematically illustrates an example of a pre-processing block;

[0046] Figure 5 schematically illustrates an example of a learnable adaptive prompting method;

[0047] Figure 6 schematically illustrates an example of an adaptable pruning strategy;

[0048] Figure 7 schematically illustrates finetuning of a large language model for summarization;

[0049] Figure 8 schematically illustrates an exemplary approach for keyword extraction;

[0050] Figure 9 schematically illustrates an example of a beam search and viable pruning strategy;

[0051] Figure 10 schematically illustrates an example of a late normalised fusion approach;

[0052] Figure 11 schematically illustrates an approach for comparing predictions in the late normalised fusion approach;

[0053] Figure 12 shows an example of a data processing method in accordance with embodiments of the present invention;

[0054] Figure 13 schematically illustrates an apparatus for performing the method and some of its associated components;

[0055] Figures 14(a) and 14(b) show results for the approach described herein and a known model respectively;

[0056] Figures 15(a) and 15(b) show results for the approach described herein and a known model respectively;

[0057] Figures 16(a)-16(c) show tables with performance metrics for various implementations of the method described herein.

[0058] DETAILED DESCRIPTION

[0059] Embodiments of the present invention relate to an approach for end-to-end summarization and keyword extraction for input descriptions. The approach can advantageously deal with non-natural-language input, such as application domain language data, in addition to content comprising different natural languages. The approach can also handle more variations of non-correct input (for example, misspellings, abbreviations, autocomplete, space-related issues) in multilingual settings.

[0060] The approach employs summarization and keyword extraction for input descriptions that can be used to categorize the input descriptions. In one example, the approach may be used in order to obtain application categorization for application distribution platforms, such as the Huawei AppGallery. This approach supports multilinguality for application domain descriptions (not only in the natural languages but also in the application domain language). The approach can thus be used to solve a problem of correctly categorizing software applications based on their corresponding application descriptions. This problem is difficult, as application descriptions often do not represent natural language. Moreover, since application descriptions are available in different (non-natural) languages and could also represent combination of different (non-natural) languages at once in one description, the approach is robust in handling all those scenarios. Input description lengths can vary in different cases and can also contain special characters and symbols. This also increases complexity in terms of finding an appropriate solution that would work in all of these scenarios. One example of an application description predominantly in English can be seen in Figure 2. One can observe different parts of different non-natural languages which resemble Italian, Korean, English, special character “words” misspelled or distorted.

[0061] The proposed categorization framework employs a deep learning technique in addition to some machine learning techniques. The framework can support multilingualism and has been tested on 5 European languages but may in theory support many (for example, 100) different languages.

[0062] The approach can also be applied to a variety of other applications. In many different types of input descriptions, beside natural language there is a presence of non-natural language words, and often pseudo-mathematical and code descriptions.

[0063] An example of the overall system 300 is schematically illustrated in Figure 3. The system comprises a plurality of functional blocks, starting with an input point at 301 in which an input description is received and then processed until the end point of the pipeline. The input descriptions comprise text. The input descriptions may be in a string format. The input descriptions comprise non-natural language content, such as application domain language content, and in some examples the non-natural language content may include pseudo-mathematical and / or code content.

[0064] The input descriptions can be first processed in a pre-processing block 400. This block comprises four parts corresponding to four sub-blocks: namely, collection 401, initial cleaning (also referred to as light pruning) 402, deduplication 403 and filtering 404.

[0065] The pre-processed data can be applied to the input descriptions and / or used in the training of a large language model (LLM). As shown in Figure 3, the model initialization stage 311 comprises three mutually dependent blocks: namely, model initialization 312, data preparation 313 and model configuration 314. These blocks may have any suitable conventional format.

[0066] Next, a finetuning stage 321 separately addresses prompt engineering, supported through a learnable adapter prompting approach 323, and a neural network pruning approach, supported through an adaptable pruning strategy 324, and LLM finetuning for summarization 325 on its own.

[0067] Initial summarized outputs 326 formed by the finetuned LLM can be filtered at 342 based on keywords output from a late fusion approach, shown at 334. The late fusion approach fuses the output of two different keyword extraction methods. In this example, these methods are extractLLM 332 and TextRank, 333. An example of the TextRank algorithm is described in Rada Mihalcea and Paul Tarau, “TextRank: Bringing Order into Text”, Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics. The final keywords can be subsequently transformed into standardized keywording (using a transformation engine 341). The final summarized output of the input description can be formed by filtering the initial summarized output 326 (the output of the finetuned LLM) using keywords formed from the input description to form a final summarized output 343. The final summarized output (which is the output of the step-filtering stage at 342) may be used for the classification at 350. The initial input description 326 may be filtered using the final output keywords generated from the one or more fusion stages (optionally transformed by a transformation engine 341, as in this example), as described above. Filtering the initial summarized output 326 based on the keywords may improve the accuracy of the final summarized output 343.

[0068] Based on training data, an iterative quality assessment procedure 344 can also be set up in which it is checked whether the summarized input descriptions are “good enough11. Once one or more quality assessment criteria are met, the procedure is finalised. In one implementation, the quality assessment 344 is performed in terms of statistical significance based on the confidence scores for the output. The statistical significance may be measured based on p-value (p=0.05) and if within those boundaries the quality may be considered to be good.

[0069] The output is correctly summarized versions of the input descriptions. The output summaries can be employed for more precise hierarchical categorization (i.e. for obtaining hierarchical-level categories for application distribution platforms). The summaries may additional be used for description creation 351, search and retrieval 352 and recommendation 353.

[0070] In this example, the system 300 also comprises the following blocks: i. A model initialization block 311 comprises the following sub-blocks (which in this example are mutually related): model initialization 312, model configuration 313 and data preparation 314. ii. The adaptable prompt storage 322 comprises of a pool of prompts used to train the prompt algorithm (learnable adapter prompting 323). iii. TextRank 333 is a further keyword extraction algorithm that is complementary to the cxtractLLM method 332 described herein. iv. The transformation engine 341 converts the fusion output form into an output employed by the step-filtering process. v. Step-filtering 342 using keywords (from the fusion output) filters out summarization sentences to get a more appropriate output summary. vi. A classifier is employed for hierarchical categorization at 350.

[0071] Figure 4 shows an example of the pre-processing block 400 and some of its components and sub-components. In the example shown in Figure 4, there are four different parts of the pre-processing stage: collection 401, light pruning 402 (or initial cleaning), deduplication 403 and filtering 404.

[0072] In the collection block 401, the input descriptions are received. In this example, the input descriptions are application descriptions. However, other types of input descriptions may be used. The collection block is configured to extract meaningful sentences from the input description. This may be done by, for example, removing extra characters, non-query inclusive characters, spaces and special characters from sentences of the input description. To achieve this, a union of special characters can be employed and then applied on the input descriptions. The following rule can then be applied: if a sentence is larger than 2 -word tokens, digits and extra spaces are removed, followed by applying simple deduplications in order to obtain clean versions of the pre-collected input description data points. The data collections cleaning can be obtained by stacking and removing collections using a Soft Term Frequency - Inverse Document Frequency (TFIDF) method (for example, as described in W. W. Cohen, P. D. Ravikumar and S. E. Fienberg, “A comparison of string distance metrics for name-matching tasks,” in IlWeb, pp. 73-78, 2003), i.e., by applying it on multiple two-pair sentences to get their similarity score. In case of equally similar pairs, one that has the lower Word Movers’ Distance (WMD) is chosen to proceed with. Finally, the tokenized output is taken and those collections are crosschecked with the dictionary forms.

[0073] In the light pruning block 402, multilingual data is recognized. Specific languages are identified. Language identification may be achieved using a langdetect library (see, for example https: / / pypi.org / project / langdetect). Pruning of the text is then performed based on minor languages by employing language identification (for example using langdetect) at the token level and identifying which parts don’t belong to the “main” language. Then, filtering can be employed at the character level by identifying bad characters by using predefined thresholds at the sub-token level and comparing those to a dictionary of clean slate tokens for different languages.

[0074] In order to perform deduplication of the input descriptions, in the deduplication block 403 , firstly all of the substrings of each sentence are extracted and subsequently a modified Fuzzy Minhash algorithm is applied to the substrings. The algorithm creates different sequences of different lengths without duplications and without minority languages. A linear weighted combination of outputs is then taken, where weights are obtained during the training stage. The final result is created by using a linear weighted combination of the previously obtained outputs.

[0075] The final stage of the pre-processing block is filtering. In the filtering block 404, a heuristic -based algorithm is employed. This algorithm employs non-overlapping, contiguous parts of input descriptions based on removing stop-words, stemming and splitting firstly and then it employs TF-IDF and Siamese Long Short-Term Memory (LSTM) networks to detect and filter out similar descriptions. It then performs prominent words-based filtering and finally adequate sampling-based filtering to obtain the processed input descriptions. Here, modified nucleus sampling may be employed from the smallest possible set of words whose cumulative probability exceeds a threshold, for example 0.4. The main difference between the light pruning and filtering stages lies in the fact that in the latter stage more radical filtering is applied based on context and content, while in the former stage filtering is performed superficially for errors and minor language faults.

[0076] This eventually makes the model become more and more resilient in non-natural language and natural language domains. This part creates “clean slate” with respect to the data entry. This particular order of steps was found to be the most performant. This block can be skipped in the inference stage. In order to make the model learn profoundly but also to eliminate over-confusing inputs, the pre-processing stages may be applied sequentially rather than simultaneously.

[0077] Figure 5 shows an example of a learnable adapter prompting approach. A prompt in machine learning is a piece of text that is used to guide the learning process. The prompt can be used to provide the model with information about the task that it is trying to solve. An LLM can provide several outputs based on how the prompt is phrased. A prompt can be as simple as a word or as complex as a paragraph. The prompt's objective is to provide the model with sufficient information so it can produce output pertinent to the prompt.

[0078] In this stage, a learning algorithm is used which dynamically selects the optimal prompting strategy. In this example, the optimal prompting strategy is selected based on pretrained user feedback and corresponding automated evaluation metrics.

[0079] The prompting starts as shown at 501 by providing a few exemplary prompts with the associated reasoning process. The LLM base model, before finetuning, is shown at 502. The present method uses a prompt chain at 503. The output of one prompt is utilized as the input of the subsequent prompt in the chain. This method preserves context in a discussion (in order to support summarization). It is also beneficial when aiming to delve further into a subject without adjusting and organizing each prompt.

[0080] The prompt strategy produces a set of prompts using different patterns, to be applied to different summarization tasks at scale. These patterns are preselected based on the task.

[0081] The LLM 502 is recursively applied using the chain of functional prompts (the prompt-chain 503). The prompts in the chain are functional because they employ a task-agnostic summarization template to all inputs in the tasks. Different prompt-chains may be constructed where each unique prompt-chain is a different view of the task and can emphasize a different aspect of the task.

[0082] To aggregate the prompt predictions reliably, a weak supervision-based approach (WSBA), shown at 504, may be applied by learning the prompt selection model without labelled data. The WSBA 504 can include a rule attention teacher-student network for combining multiple rules and model predictions with per-instance specific weights dependent on the corresponding summarization context (supported by categorization).

[0083] The user feedback automated check block 505 can verify the prompt output with respect to some existing ground truth prompts and corresponding feedback data. Based on the result of that verification, the block 505 can determine whether the prompt should be further refined (by sending it to the starting block 501, updating it and applying the refinement strategy through some varied prompt patterns) or not.

[0084] The lower part of Figure 5 depicts how the overall process is conducted: starting from data collection 551, applying preprocessing 552, followed by iterative relabelling using a labelling block 553 and a refinement block 554 which mainly depends on success of the prompt generation block 555 by verifying its performance on an evaluation dataset 556. The success of the prompting part is achieved by improving the prompt itself: by doing so, one essentially refines the model and then the labels, so later on, it can be used to create a new model with improved performance.

[0085] A semi-supervised learning approach can be employed for training without any knowledge about the nature or structure of the weak prompt rules. Thus, given the varied dependencies among prompt-chains, some last outputs are aggregated while taking into account the accuracy and dependency properties (without relying on the labelled data). Hence this approach is able to scalably obtain multiple effective prompts given a summarization input and combine the intermediate answers (to these prompts) for the final answer.

[0086] Figure 6 shows an example of an adaptable pruning strategy. Pruning of the neural network of the LLM by reducing the number of parameters it uses can improve efficiency without significantly impacting prediction performance. The pruning strategy described herein is based on a combination of fixed adapted and learned pruning of the neural network, schematically illustrated at 601 and 602 respectively in Figure 6. Finetuning of the neural network may then be performed after the pruning.

[0087] In the fixed pruning stage 601, the number of neurons to be removed from the neural network may be defined. The neurons may be pruned from the network in batches.

[0088] The fixed pruning may be carried out as follows. Initially, multiple neurons are stacked in a transitional region. For other neurons, the other neurons are first added together and then combined with the stacked neurons to achieve fixed pruning. After k epochs, a different pruning strategy can be applied where one or more neurons of the neural network are continuously pruned as long as the validation loss is below a threshold. If the validation loss is above a threshold, no pruning of the network is performed. This different pruning strategy is therefore referred to as learned pruning, as opposed to the fixed pruning.

[0089] In one implementation, forthe learned network pruning, the expansion of the Maclaurin series is applied. This can be performed by utilizing a weighted combination of the first- and second-order Maclaurin expansion to perform iterative pruning. A Maclaurin series represents an approximation of a function with a polynomial around 0 (in contrary to a Taylor series, that approximates a function with a polynomial around an arbitrary point different to 0). This approach uses different thresholds for every layer and makes all those thresholds trainable, by letting the training procedure find optimal thresholds alongside the layer weights during finetuning. For the thresholds, a normalized histogram may be employed as an estimate of the probability function, in which, for each threshold, mean and variance canbe obtained and employed later on. The upper and lower weights’ limits of the threshold can be set to the maximum and minimum values of the gradient descent optimization to minimize the difference between the obtained and projected results. The threshold interval canbe set at 0.01 to ensure precision in identifying the optimum threshold.

[0090] Hence, the combined pruning approach of fixed and learned pruning of the neural network can lead the optimization into a minima and final results can be improved.

[0091] To support the intuition behind this approximation, for the vast majority of weights that are outside the transitional region, pruning gives almost the same effect. On the other hand, although values do differ for weights within the transitional region, these weights remain there for a very small amount of the training time. This means that they would acquire their correct values through back-propagation once they are out of the transitional region (denoted with arrows in Figure 6). This pruning makes the process end-to-end trainable and allows to achieve improved performance at a lower cost. Pruning of the network effectively reduces the number of neurons (weights) and makes the network simpler and thus faster to train, without losing significant performance (i.e. pruning makes the process more efficient).

[0092] As shown in Figure 3, the approach described herein can also comprise LLM finetuning for summarization. The LLM canbe finetuned using a multi-task learning process.

[0093] Figure 7 schematically illustrates a multi-tasking learning process in which both hard parameter sharing (in which parameters of hidden layers are shared between tasks and parameters of output layers, that are specific to a task, are fixed at the same time) and soft parameter sharing of hidden layers (where each task has its own LLM model 701, 702, 703, 704 with its own parameters) are employed simultaneously. <!’ represents the LLM model’s network model function (referring to its weights, biases and also its hyperparameters). O1, <I’2. <!’ ’ and <I’4refer to the network functions of the LLM models 701, 702, 703 and 704 respectively. The models 701, 702, 703, 704 can be jointly finetuned, for example in a Sequential Summarization Finetuning framework.

[0094] In Figure 7, taskl 705, task2 706, task3 707 and task4 708 denote summarization data of different complexity that are being used in the finetuning process. Each task of the multi-task training corresponds to summarization training data of different complexities. This can make the summarization process more coherent and robust. This can be performed in a seq-to-seq way by taking into account that in every round, each sequence is represented at least once. In theory, this process could be extended to a N (where N > 5) number of tasks, which makes the process more generalizable. Employing hard and soft parameter sharing at the same time can improve performance for more complex data samples and scenarios (i.e. it can better separate relevant from irrelevant features). Hence the approach jointly optimizes the network parameters and a categorical random variable (uni, n) for each layer 1 and each task in Tn, where Tn represents a set of n=4 tasks. Here, the categorical random variable (crv) represents which tasks are shared, skipped, or performed individually by a particular block for the multiple tasks. Since the task variable is non-differentiable, in one example a modified Gumbel-Softmax (MGS) sampling is employed to optimize the task variable. A Gumbel distribution is sampled from a uniform distribution, then normalized by taking in low-min and high-max normalization approach and finally interpolated. In here, the interpolation is achieved between discrete one-hot-encoded distributions. During training for small temperature values (r <= 0.5), the expected value of the distributions approaches the expected value of corresponding crv variable with the same logits. Thus, this approximation is used to train / infer on logits. For each task there is a corresponding range of temperature values. The loss function is modelled as the sum of the two losses: a sparsity loss to achieve model compactness, and a sharing loss that achieves sharing across multiple of tasks. Also, the sequential order of tasks can follow adapted correlation covariance coefficient (ACCC) order powered by normalized dataset size.

[0095] This may also help the models to generalize better to different tasks, as a hypothesis space that performs well for a sufficiently large number of training tasks will also perform well for learning new tasks as long as they are from the same “context”. As different tasks have different noise patterns, a model that learns four tasks simultaneously is able to learn a more general representation. Learning just a single task may in some implementations result in overfitting to that specific task, while learning more tasks jointly enables the model to obtain a better representation by averaging the noise patterns.

[0096] An approach for keyword extraction that can be used in the present framework will now be described.

[0097] The extractLLM block, shown at 332 in Figure 3, takes the input description as input. The TextRank block also takes the input description as input. This further keyword extraction method, and the subsequent fusion of results, will be described later. extractLLM is an approach for keyword extraction that is based on the synergy of LLM and machine learning techniques. An overview of the approach is shown in Figure 8. This self-contained process comprises the following steps.

[0098] The application descriptions 801 are input to the pipeline. At 802, the string “keywords:” is attached to the end of the obtained string to trigger an LLM to generate keywords (with equivalents for different European languages and an equivalent in Chinese used for Chinese datasets). At 803 is a text generation step, where sentences are generated for the given text obtained from the previous step. At 804 the keywords are output. At 805, an adapted beam search algorithm is used along with viable pruning. The beam search algorithm is adopted for text generation together with disjunctive positive constraints to include delimiters “,” and between keywords. It can be examined whether the generated keyword strings are valid. A valid string is a nonempty one that ends with “< / s>”, an end-of-sequence token and follows the regular expression “[0-9A-z (),.—;’]*”. A corresponding expression may be used for Chinese natural language. At 806, the final keywords are output.

[0099] The adapted beam search approach and viable pruning are depicted in further detail in Figure 9.

[0100] In step 1, shown at 901, at each level the successors’ states are employed and sorted in a decreasing order of the cost. Thus, the states are expanded with the beam increase. That is, the greater the beam width the fewer states are filtered out. Then, the beam search picks the B best sequences so far and considers the probabilities of the combination of all of the preceding keywords, along with the keyword in the current position. The process is repeated until the beam is zero, when the search has stopped and the keyword with the highest probability is picked (normalized by the number of keywords) out of the final candidate list. Then the beam is reduced by one after the keyword is stored into a final candidate list. In step 2, shown at 902, after the list of keywords is reduced to the final candidate list, the beam search algorithm is applied to the keywords in the final candidate list. Allowed here are a specific number of candidates from the “same history” at the time, denoted by num_can, maximum number of candidates C, and a candidate end is removed in case there is already a candidate with a “better” history i.e. a candidate that contains the same predecessor words.

[0101] At step 3, shown at 903, the method discards those candidate(s) that are worse than the best active candidate by a specific threshold. Given a pruning threshold, PT, and an active candidate list C, a candidate end e C may be discarded if score(cnd) <= max{score(c)}-PT [c e C],

[0102] At the decision block shown at 904, the procedure is stopped if the predefined number of keywords num_can is reached. Otherwise, the process returns to step 1.

[0103] The fusion function indicated at 334 in Figure 3 will now be described further.

[0104] The fusion function is designed to merge the outputs of two different keyword extraction methods into one semantic interpretation. The fusion of the outputs of the cxtractLLM and TextRank methods is illustrated in Figures 10 and 11.

[0105] In Figure 10, 1001 indicates the cxtractLLM predictions and 1002 indicates the TextRank keyword predictions.

[0106] By combining the strengths of these two complementary approaches, accuracy and robustness to the problems that affect individual modalities canbe achieved. In this late fusion, the predictions are combined on a semantic level. Since each modality is trained separately, the joint probability learning of the modalities is not performed explicitly.

[0107] To fuse the extractLLM and TextRank parts, the top g predictions are taken into account. These predictions are indicated at 1003 for the extractLLM predictions and at 1004 for the TextRank predictions. These are processed at 1005. The normalized predictions (normalized in [0, 1] interval) are taken and scaled with the normalized frequency of appearance. The predictions are aggregated at 1006 to form an output 1007.

[0108] Figure 11 schematically illustrates an approach for comparing predictions in the late normalised fusion approach and shows the fusion of predictions from each modality (i.e. the predictions of extractLLM and TextRank).

[0109] The textured boxes on the left and right of the image (shown collectively at 1101 and 1102 respectively) represent different predictions (i.e. sorted rankings). Thus, on both the left and the right of the image the same ranking has been presented using the same texture. Above them, the rectangles shown generally at 1103 depict all of the predictions represented jointly that are of interest (regardless of the modality employed). The textured rectangles both on the left at 1101 and right at 1102 are connected with bidirectional arrows, meaning that during the fusion process, different predictions interact with each other (from both modalities, i.e., from left and right sides of figure 11).

[0110] The rectangles 1151, 1152, 1153 represent stages of the fusion process. In this example, the fusion process comprises three sequential stages and the stage indicated at 1151 is followed by stage 1152 and then stage 1153.

[0111] The first and second ranked guesses are taken and the difference between them is determined. In other words, differences between the first top two ranked predictions of the joint cxtractLLM and for the TextRank approaches in many confidence pairs are observed. In stage 1151, the difference is compared to a threshold. It is concluded that for differences beyond the threshold (which may be a predetermined threshold, and may be a reliably large threshold), one can be sure that the correct prediction was the first ranked prediction, based either on the extractLLM or on the TextRank (or both).

[0112] If the difference is below the threshold, a union (sum) of normalized predictions is found from both parts and the one with most occurrences is chosen in stage 1152.

[0113] In case of a “tie-break” (i.e. if there is no difference between the first and second ranked guesses), in stage 1153, an adapted low rank multimodal fusion can be employed.

[0114] Predictions from the two modalities are observed as vectors. The predictions vectors are decomposed (using an appropriate scaling) into two sets (denoted by A and B for the two modalities respectively). Here, the parametrization is specified by the rank of the vector which makes the sets’ decomposition valid.

[0115] The fused vectors generated in the aggregation process are defined below, where n is the element-wise product over a sequence of vectors, ‘ ’ is the product solution, w_m is the weight matrix for modality m e (A, B), p is the rank, and f_m are the input vector representations. Since this fusion explores interdependency between the predictions, the first ranked prediction is eventually taken as the final one. w_m has been obtained in the training stage by creating a Hadamard product of the of decomposed vector predictions for each modality.

[0116] Figure 12 shows an exemplary method for performing end-to-end summarization and keyword extraction for input descriptions comprising non-natural language content. At step 1201, the method comprises receiving an input description comprising non-natural language content. At step 1202, the method comprises processing the input description to extract one or more keywords from the input description. At step 1203, the method comprises processing the input description to form a summarized output of the input description in dependence on the one or more keywords extracted from the input description. At step 1204, the method comprises, based on the extracted keywords and the summarised output, classifying the input description into one or more of a plurality of classes.

[0117] In one implementation, the input description is a description of a software or mobile application and the data processing device is communicatively connected to an application distribution platform. In this example, the step of classifying the input description into one or more of a plurality of classes comprises classifying the input description into one or more categories of the application distribution platform. This may allow for more efficient and accurate hierarchical categorization in application distribution platforms for mobile computing devices.

[0118] Figure 13 shows an example of a data processing apparatus 1300 configured to implement the method described herein. The apparatus comprises a device 1301. The device 1301 comprises a processor 1302 and a memory 1303.

[0119] The device 1301 may in some implementations also comprise a transceiver 1304 that is capable of communicating over a network with other entities 1310, 1311. Those entities may be physically remote from the device 1301. The network may be a publicly accessible network such as the internet. The entities 1310, 1311 may be based in the cloud. Entity 1310 is a computing entity. Entity 1311 is a command and control entity. These entities are logical entities. In practice they may each be provided by one or more physical devices such as servers and data stores, and the functions of two or more of the entities may be provided by a single physical device. Each physical device implementing an entity comprises a processor and a memory. The devices may also comprise a transceiver for transmitting and receiving data to and from the transceiver 1304 of device 1301. The memory stores in a non-transient way code that is executable by the processor to implement the respective entity in the manner described herein.

[0120] Therefore, the method may be deployed in multiple ways, for example in the cloud, on the device, or alternatively in dedicated hardware. As indicated above, the cloud facility could perform training to develop new algorithms or refine existing ones. Depending on the compute capability near to the data corpus, the training could either be undertaken close to the source data, or could be undertaken in the cloud, e.g. using an inference engine. The method may also be implemented at the device, in a dedicated piece of hardware, or in the cloud.

[0121] Described in the following are results based on different analysis and ablation studies performed to express the potential of the solution.

[0122] In the table shown in Figure 14(a) the present method was applied using different pretrained models tested on different app- description-language domain datasets (denoted as Lang), in which the majority of words belong to a specific “main” language i.e. with application descriptions in English (EN), German (DE), French (FR), Italian (IT), Spanish (ES) languages. Figure 14(b) shows results obtained using the OPT-6.7B based model only (“OPT : Open Pre-trained Transformer Language Models”, S. Zhang et al., https: / / arxiv.org / abs / 2205.01068). The following evaluation metrics are used: precision, recall and flscore.

[0123] The table in Figure 15(a) shows results for the approach described herein used with the ChatGML-6B model tested on an app- description-language domain dataset in which the majority of words belong to the Chinese language (ZH). Figure 15(b) shows results obtained using the OPT-6.7B based model only.

[0124] Figures 16(a)- 16(c) show results obtained using other data types. Figure 16(a) shows results for network protocol data (NPD), Figure 16(b) shows results for mobile network vendor data (MNVD) and Figure 16(c) shows results for network vendor data (NVD) using the present approach with OPT-6.7B.

[0125] The approach is versatile and can be used for different data types and application scenarios. It provides a way to bridge the gap between a user and application distribution platforms or other systems whose applicability is beyond the standard use of natural language(s) and the application domain. The framework can support multilingualism (for example, supporting up to 100 different languages).

[0126] The data pre-processing block creates a viable way to process complex non-NL data and make it ready for further analysis. The methods have general application for different kinds of data and also support multilingual non-natural language data types.

[0127] The prompting strategy has general applicability for non-NL LLMs and can create cleaner, more universal prompts which can lead to a better summarization.

[0128] The pruning strategy has general applicability for LLMs and improves the training and may help to obtain a better summarization. The pruning strategy is compatible with and complementary to the above prompting strategy.

[0129] The LLM finetuning method including multitask learning is also generalizable to a multitude of complementary tasks. It is applicable and extendable to a variety of tasks of different kinds of data to be understood and summarized. At the same time, this can contribute to training efficiency. The keyword extraction method described herein supports multilingual non-natural language data types and can improve the performance of the overall framework.

[0130] The fusion function is agnostic to different input data modalities and characteristics. Additionally, the function could be applied to different data types too. The fusion outperforms each modality on its own.

[0131] The approach described herein provides a method applicable to a range of natural / non-natural language services such as application / mobile / networking search and retrieval, recommendation, categorization, description creation and marketing, application / mobile / networking description creation and marketing, app / mobile / networking recommendation, circuit classification, genome and genomics classification.

[0132] The main drawbacks of prior approaches he in the fact they are purely target for natural languages applications, while the present approach is based on non-natural language control, which allows the approach to be used in application domain applications for application distribution platforms and for different languages in the application distribution platform as well (which may also contain non-natural languages).

[0133] Another disadvantage of the prior art relates to inability to automatically create summaries (dependent on size on the input descriptions). The present approach can create summaries that are dependent on the size of the input description, while further allowing for the generation of a predefined length of summary.

[0134] This solution can achieve end-to-end input text summarization and keyword extraction. The approach can also handle input descriptions having autocomplete, abbreviations, space-related issues and misspellings in more than 100 languages and for application domain language content. The summarization approach is able to create summaries that could be both dependent and independent of input text size.

[0135] As well as being suitable for different types of domain application language, as well as for natural language data, the approach is generally applicable to other domains, such as mobile network vendor data, bioinformatics domain data, network protocols data, mobile applications data and physical circuit specifications, to name a few. These examples are non-limiting and other input description data types are possible.

[0136] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A data processing device (1301) for performing end-to-end summarization and keyword extraction of input descriptions, the data processing device having one or more processors (1302) configured to: receive (1201) an input description (301) comprising non-natural language content; process (1202) the input description to extract one or more keywords from the input description; process (1203) the input description to form a summarized output of the input description in dependence on the one or more keywords extracted from the input description; and based on the extracted keywords and the summarized output, classify (1204) the input description into one or more of a plurality of classes.

2. The data processing device (1301) as claimed in claim 1, wherein the one or more processors (1302) are further configured to pre-process the input description (301) by performing one or more of the following:(i) extracting one or more sentences from the input description by removing one or more of the following from the input description: extra characters, non-query inclusive characters, spaces, special characters and duplicated content;(ii) identifying content corresponding to a primary language and one or more minor languages in the input description and remove content corresponding to the one or more minor languages and / or bad characters from the input description;(iii) extracting one or more sub-strings from the input description, apply a Fuzzy Minhash algorithm to each substring to form respective outputs, and take a linear weighted combination of the outputs; and(iv) filtering the input description based on content and / or context of the input description.

3. The data processing device (1301) as claimed in claim 1 or claim 2, wherein the step of processing the input description (301) to form a summarized output of the input description comprises inputting the input description to a large language model, wherein the large language model comprises a neural network finetuned to form the summarized input from the input description.

4. The data processing device (1301) as claimed in claim 3, wherein the neural network comprises output layers and hidden layers, and wherein finetuning of the neural network comprises performing multi-task training in which hidden layers of the neural network are shared and output layers of the neural network are fixed specific to a task of multiple tasks.

5. The data processing device (1301) as claimed in any preceding claim, wherein the step of processing the input description to form a summarized output of the input description comprises implementing a prompting strategy, wherein the one or more processors (1302) are configured to select the prompting strategy from a plurality of possible prompting strategies.

6. The data processing device (1301) as claimed in claim 5, wherein the one or more processors (1302) are configured to select the prompting strategy by applying a selection algorithm to determine which of the plurality of possible prompting strategies are best-performing for the input description.

7. The data processing device (1301) as claimed in claim 3 or claim 4, or as claimed in claim 5 or claim 6 as dependent on claim 3 or claim 4, wherein the one or more processors (1302) are configured to prune the neural network by applying one or more fixed pruning steps and one or more learned pruning steps.

8. The data processing device ( 1301 ) as claimed in claim 7, wherein in the one or more learned pruning steps, the one or more processors (1302) are configured to continuously prune the neural network when a validation loss is below a threshold.

9. The data processing device (1301) as claimed in claim 7 or claim 8, wherein the one or more learned pruning steps apply a weighted combination of first-order and second-order Maclaurin expansion series.

10. The data processing device (1301) as claimed in any preceding claim, wherein the one or more processors (1302) are configured to: append a data string to the end of the received input description, wherein the appended data string acts to trigger a further large language model to generate keywords from the input description; and apply the further large language model to the input description to generate one or more sentences from which the keywords are generated.

11. The data processing device (1301) as claimed in claim 10, wherein the one or more processors (1302) are further configured to determine whether a generated keyword is valid by determining whether the following are true: the string comprising the generated keyword is non-empty string ending with < / s>; the string comprising the generated keyword is an end-of-sequence token; and for non-Chinese languages, the string comprising the generated keyword follows the regular expression [0-9A-z(),.- ;’]*, or for Chinese language, the string comprising the generated keyword follows the regular expression [0-9\p{Han}(),.-12. The data processing device (1301) as claimed in claim 10 or claim 11, wherein the one or more processors (1302) are further configured to apply a beam search algorithm to the generated keywords to refine the keywords to be output.

13. The data processing device (1301) as claimed in any' preceding claim, wherein the one or more processors (1302) are configured to: receive refined keywords; receive keywords generated by applying a further keyword selection method to the input description; and combine the refined key words and the keywords generated by applying the further keyword selection method in one or more fusion stages to form final output keywords.

14. The data processing device (1301) as claimed in claim 13, wherein the further keyword selection method comprises applying a TextRank algorithm to the input description (301).

15. The data processing device (1301) as claimed m any preceding claim, wherein the step of processing the input description to form a summarized output of the input description in dependence on the one or more keywords extracted from the input description comprises filtering an initial summarized output using the one or more keywords formed from the input description to form a final summarized output.

16. The data processing device (1301) as claimed in claim 15, wherein the one or more processors are configured to classify the final summarized output into the one or more of the plurality of classes.

17. The data processing device (1301) as claimed in claims 15 or claim 16 as dependent on claim 13 or claim 14, wherein the one or more processors are configured to filter the initial input description is filtered using the final output keywords generated from the one or more fusion stages.

18. The data processing device (1301) as claimed m any preceding claim, wherein the non-natural language content comprises content corresponding to multiple languages.

19. The data processing device (1301) as claimed in any preceding claim, wherein the input description further comprises pseudo-mathematical and / or code content.

20. The data processing device (1301) as claimed in any preceding claim, wherein the input description (301) comprises mobile network data, bioinformatic domain data, network protocol data, mobile application data or a physical circuit specification.

21. The data processing device (1301) as claimed m any preceding claim, wherein the input description (301) is a description of a software or mobile application and wherein the data processing device is communicatively connected to an application distribution platform, wherein the one or more processors (1302) are configured to classify the input description into one or more categories of the application distribution platform.

22. A computer-implemented method (1200) for performing end-to-end summarization and keyword extraction of input descriptions, the method comprising: receiving (1201) an input description (301) comprising non-natural language content; processing (1202) the input description to extract one or more keywords from the input description; processing (1203) the input description to form a summarized output of the input description in dependence on the one or more keywords extracted from the input description; and based on the extracted keywords and the summarised output, classifying (1204) the input description (301) into one or more of a plurality of classes.

23. A computer program comprising instructions that when executed by a computer (1301) cause the computer to perform the method (1200) of claim 22.

Images