Method and system for generating processing logic

Abstract

The processing logic generation system is operable to generate processing logic that can automatically process textual system descriptions related to the system. The processing logic generation system may perform a method that includes performing an automatic marking on a textual description included in a training batch of textual descriptions to generate an annotated training batch, where performing the automatic marking may include performing a clustering technique to generate an annotated training batch. And annotating the text item included in the training batch with a tag in a set of tags based on a semantic distance of the text item included in the training batch from text items previously assigned to a plurality of tags in the set of tags.

Description

Technical Field

[0001] The present invention relates to methods, devices and systems for generating processing logic, which can be operated to automatically process text system descriptions related to systems (such as public service systems (such as power systems), industrial systems (such as industrial manufacturing plants), and / or computing and / or communication networks (such as computing and / or communication networks of public service systems or industrial systems)). The present invention relates to methods, devices and systems that can be operated to facilitate the processing of text system descriptions related to systems (such as public service systems (such as power systems), industrial systems (such as industrial manufacturing plants), and / or computing and / or communication networks (such as computing and / or communication networks of public service systems or industrial systems)). Background Art

[0002] Various technical systems, such as public service systems (e.g., power generation systems, distribution systems, transmission systems, power grids or other public service systems, such as gas systems, water systems or sewage systems), substations of public service systems (e.g., power system substations), industrial systems, and / or computing and / or communication networks (e.g., for public service systems), may include a large number of components. The textual descriptions of such systems may be long, complex and related. Processing textual descriptions of such systems (e.g., for the purpose of deploying appropriate equipment, debugging and / or configuring the system) is a complex task. This task is made more difficult by the fact that the textual descriptions are not standardized and may use different semantics to describe the same or related technical aspects.

[0003] US2021 / 0264111 A1 (hereinafter "D1") discloses a system and method for natural language processing (NLP). The system can be trained by: identifying training data including clean data and noisy data; using an artificial neural network (ANN) to predict annotation information; using a weighted loss function to calculate a loss value of the annotation information, the weighted loss function applies a first weight to the clean data and at least one second weight to the noisy data; and updating the ANN based on the loss value. The noisy data can be obtained by: identifying a set of unannotated sentences in a target domain; de-lexicalizing the set of unannotated sentences; finding similar sentences in a source domain; filling at least one arbitrary value in a similar de-lexicalized sentence; generating annotation information for similar de-lexicalized sentences using an annotation model of the source domain; and applying a heuristic mapping to generate annotation information for sentences in the target domain.

[0004] There is a need for techniques that provide assistance in the complex task of processing textual descriptions of technical systems. While natural language processing (NLP) is generally known, such techniques require training and / or testing sets that include a sufficient number of labeled text items. Generating such training sets is itself a complex, arduous, and time-consuming task. Summary of the invention

[0005] There is a need to provide enhanced techniques for processing textual descriptions of technical systems. There is a need to provide techniques that allow textual descriptions of technical systems to be annotated by processing logic while mitigating challenges associated with providing a training set for generating said processing logic.

[0006] According to the invention, a method and a system are provided as described in the independent claims. The dependent claims define preferred embodiments.

[0007] The systems and methods according to the present invention are operable to automatically label data using clustering based on the semantic distance of text items from previously labeled text items. The labeled text items can be used as (part of) a training batch for training an AI model, which can be used to generate processing logic for processing text descriptions of technical systems. Alternatively or in addition, the labeled text items can ultimately be used to determine a technical system configuration and / or automatically perform system debugging and / or configuration tasks for a technical system.

[0008] A method of generating processing logic operable to automatically process a textual system description related to a system is disclosed. The method includes performing automatic tagging on textual descriptions included in a training batch of textual descriptions to generate an annotated training batch, wherein performing the automatic tagging may include performing a clustering technique to annotate textual items included in the training batch with a label from a set of labels based on a semantic distance of the textual items included in the training batch from textual items previously assigned to a plurality of labels from the set of labels.

[0009] The method may be performed by a computing system.

[0010] The textual system description may be a request for proposal (RFP) or another technical system description.

[0011] The textual description may include a request for proposal (RFP) or other technical system description.

[0012] The method may include training an AI model using the annotated training batch to generate processing logic.

[0013] The processing logic may be operable to automatically process textual system descriptions relating to public service systems (such as power systems), industrial systems, or computing and / or communications systems.

[0014] The set of tags may include an intent category.

[0015] A set of intent categories may include intent categories related to different devices of the system.

[0016] The set of tags may include attributes of at least some of the intent categories.

[0017] The attributes of an intent class may define entities related to the device to which the intent class relates.

[0018] The entities can be annotated independently of the semantic distance.

[0019] Annotating the entity may include operating a user interface to allow information about the entity to be received via the user interface.

[0020] The method may further include executing the AI ​​model after training to process the textual system description into an annotated textual system description.

[0021] The method may further include iteratively repeating the steps of automatically labeling and training the AI ​​model until the performance criteria are met.

[0022] The method may further comprise modifying the set of labels or modifying labels assigned in the automatic labeling.

[0023] The method may further include retraining the AI ​​model after modifying the set of labels or modifying the labels assigned in the automatic labeling.

[0024] The method may include modifying the set of tags.

[0025] Modifying the set of tags may include one or more of the following: merging at least two tags in the set of tags into one tag; dividing a tag in the set of tags into at least two tags; and retaining a tag in the set of tags.

[0026] Modifying the set of tags may be performed automatically.

[0027] Alternatively or additionally, modifying the set of tags may be performed in response to user input received via a user interface.

[0028] The method may further comprise iteratively repeating at least one (i.e., one or more) of the following steps until the performance criterion is met:

[0029] - performing said automatic marking,

[0030] - training the AI ​​model,

[0031] - Modify the set of labels or modify the labels assigned in the automatic labeling and retrain the AI ​​model.

[0032] Performing automatic labeling may include annotating each text item included in the training batch with a corresponding label from the set of labels if a semantic distance to at least one cluster in the clustering technique satisfies a semantic distance threshold criterion.

[0033] Performing automatic labeling may include leaving text items included in the training batch unannotated if the semantic distances to all clusters in the clustering technique do not satisfy a semantic distance threshold criterion.

[0034] The semantic distance between the text item and the cluster may be an average semantic distance between the text item and all text items belonging to the corresponding cluster.

[0035] The AI ​​model may have an input layer that receives text items included in the text system description and an output layer that outputs annotations to the text items included in the text system description.

[0036] AI models can include natural language processing (NLP) transformers.

[0037] The NLP transformer may include an encoder-decoder model.

[0038] The NLP transformer can include an encoder-decoder model with a self-attention layer.

[0039] The encoder of an NLP transformer can include a stack of encoder layers.

[0040] Each encoder layer can include a self-attention layer and a feed-forward network.

[0041] The feed-forward network can be a fully connected 2-layer network.

[0042] The AI ​​model may include a conditional random field (CRF) layer that receives the output of the NLP transformer.

[0043] In all of the above and in all of the following, unless otherwise stated, textual descriptions (e.g., textual descriptions included in a training batch and / or textual descriptions included in annotated training batches and / or textual system descriptions being processed) may all relate to the same domain or be part of the same domain.

[0044] The term "domain" in this context may be understood to specify a particular kind of content that the textual description describes (e.g., a bid, tender, RFP regarding a particular project or system), but it may also specify a particular kind of technical system that the textual description relates to (e.g., a public service system such as a power system, an industrial system, a computing system, a communication system). The common domain of the textual description may even relate to a particular kind of equipment or classification of equipment of such a technical system (e.g., routers, switches, circuit breakers, etc.). The common domain may also be defined by a particular kind of action or effect that the textual description will have or invoke on the relevant system or equipment (e.g., configuration or commissioning of a system or equipment).

[0045] In any case, the need to transfer information from one (technical) domain to another domain can be avoided, contrary to the approach of, for example, document D1. In particular, for applications of technical systems and equipment of the kind mentioned (e.g. public service systems like power systems, industrial systems, computing systems, communication systems and related equipment), being able to avoid such transfer operations can significantly improve performance and accuracy, since similar or similar-sounding terms often have significantly different meanings in different technical contexts and can therefore be a source of errors and inaccuracies, for example in the heuristic mapping from the source domain to the target domain of D1 (see also the discussion further below on the technical advantages obtained by the present invention).

[0046] A method of debugging, configuring and / or operating a system may include executing generated processing logic to automatically process a textual system description related to the system, and automatically performing at least one debugging, configuring and / or operating action based on an output of the processing logic.

[0047] The method may be automatically performed by at least one circuit.

[0048] The method may further include executing the method of an embodiment to generate processing logic.

[0049] A method of automatically generating and / or evaluating a system configuration may include executing the method of an embodiment to generate processing logic operable to automatically process a textual system description associated with a system.

[0050] The method of generating and / or evaluating a system configuration may include: receiving a textual system description related to a system; processing the textual system description using the AI ​​model to generate an annotated textual system description; and generating an output based on the annotated textual system description.

[0051] Generating output may include one or more of the following: automatically configuring at least one device of the system based on the annotated text system description; automatically generating configuration data for at least one device of the system based on the annotated text system description, and storing the configuration data in the at least one device for field operation of the system, optionally, wherein the configuration data may include at least one substation configuration description SCD file, further optionally, wherein the SCD file is an SCD file compliant with IEC 61850; automatically selecting devices for deployment in the system based on the annotated text system description; automatically determining the system topology of the system based on the annotated text system description.

[0052] A computing system adapted to generate processing logic operable to automatically process textual system descriptions related to a system includes at least one circuit adapted to perform automatic tagging of textual descriptions included in a training batch of textual descriptions to generate an annotated training batch, wherein performing the automatic tagging may include performing a clustering technique to annotate textual items included in the training batch with a label from a set of labels based on a semantic distance of the textual items included in the training batch from textual items previously assigned to a plurality of labels from the set of labels.

[0053] The at least one circuit may be adapted to train the AI ​​model using the annotated training batch to generate processing logic.

[0054] The computing system may be adapted to perform the method of any embodiment.

[0055] The computing system may be adapted such that the textual system description may be a request for proposal (RFP) or another technical system description.

[0056] The computing system may be adapted such that the textual description may include a request for proposal (RFP) or other technical system description.

[0057] The computing system may be adapted to train an AI model using the annotated training batches to generate processing logic.

[0058] The computing system may be adapted such that the processing logic is operable to automatically process textual system descriptions relating to public service systems (such as power systems), industrial systems, or computing and / or communications systems.

[0059] The computing system may be adapted such that the set of tags may include an intent category.

[0060] The computing system may be adapted such that a set of intent categories may include intent categories related to different devices of the system.

[0061] The computing system may be adapted such that the set of tags may include attributes of at least some of the intent categories.

[0062] The computing system may be adapted such that attributes of an intent class may define entities related to a device to which the intent class relates.

[0063] The computing system may be adapted such that the entity may be annotated independently of semantic distance.

[0064] The computing system may be adapted such that annotating an entity may include operating a user interface to allow receipt of information about the entity via the user interface.

[0065] The computing system may be adapted to execute the AI ​​model after training to process the textual system description into an annotated textual system description.

[0066] The computing system can be adapted to iteratively repeat the steps of automatically labeling and training the AI ​​model until the performance criteria are met.

[0067] The computing system may be adapted to modify the set of labels or to modify labels assigned in automatic labeling.

[0068] The computing system can be adapted to retrain the AI ​​model after modifying the set of labels or modifying the labels assigned in automatic labeling.

[0069] The computing system may be adapted to modify the set of tags.

[0070] The computing system may be adapted such that modifying the set of tags may include one or more of the following: merging at least two tags in the set of tags into one tag; dividing a tag in the set of tags into at least two tags; retaining a tag in the set of tags.

[0071] The computing system may be adapted such that modifying the set of tags may be performed automatically.

[0072] Alternatively or additionally, the computing system may be adapted such that modifying the set of tags may be performed in response to user input received via a user interface.

[0073] The computing system may be adapted to iteratively repeat at least one (i.e., one or more) of the following steps until a performance criterion is met:

[0074] - Perform automatic tagging

[0075] -Training AI models

[0076] - Modify the set of labels or modify the labels assigned in the automatic labeling and retrain the AI ​​model.

[0077] The computing system may be adapted such that performing automatic labeling may include annotating each text item included in the training batch with a corresponding label in the set of labels if a semantic distance to at least one cluster in the clustering technique satisfies a semantic distance threshold criterion.

[0078] The computing system may be adapted such that performing automatic labeling may include leaving text items included in the training batch unannotated if the semantic distances to all clusters in the clustering technique do not satisfy a semantic distance threshold criterion.

[0079] The computing system may be adapted such that the semantic distance of a text item to a cluster may be an average semantic distance of the text item to all text items belonging to the respective cluster.

[0080] The computing system may be adapted such that the AI ​​model may have an input layer that receives text items included in the text system description and an output layer that outputs annotations to the text items included in the text system description.

[0081] The computing system may be adapted such that the AI ​​model may include a natural language processing (NLP) transformer.

[0082] The computing system may be adapted such that the NLP transformer may include an encoder-decoder model.

[0083] The computing system can be adapted such that the NLP transformer can include an encoder-decoder model with a self-attention layer.

[0084] The computing system may be adapted such that an encoder of the NLP converter may include an encoder layer stack.

[0085] The computing system can be adapted such that each encoder layer can include a self-attention layer and a feed-forward network.

[0086] The computing system may be adapted such that the feed-forward network may be a fully connected 2-layer network.

[0087] The computing system may be adapted such that the AI ​​model may include a conditional random field (CRF) layer that receives the output of the NLP converter.

[0088] The debugging, configuration and / or operating system may be operable to execute the generated processing logic to automatically process the textual system description related to the system and automatically perform at least one debugging, configuration and / or operating action based on an output of the processing logic.

[0089] The debugging, configuration and / or operating system may be adapted to execute the method of the embodiment to generate processing logic.

[0090] The system for automatically generating and / or evaluating a system configuration may be adapted to perform the method of an embodiment to generate processing logic operable to automatically process a textual system description related to the system.

[0091] The system for automatically generating and / or evaluating system configurations may be adapted to perform the following operations: receiving a textual system description related to a system; processing the textual system description using the AI ​​model to generate an annotated textual system description; and generating an output based on the annotated textual system description.

[0092] The system for automatically generating and / or evaluating system configuration can be adapted so that the generated output can include one or more of the following: automatically configuring at least one device of the system based on the annotated text system description; automatically generating configuration data for at least one device of the system based on the annotated text system description, and storing the configuration data in the at least one device for field operation of the system, optionally, wherein the configuration data may include at least one substation configuration description SCD file, further optionally, wherein the SCD file is an SCD file compliant with IEC 61850; automatically selecting devices for deployment in the system based on the annotated text system description; automatically determining the system topology of the system based on the annotated text system description.

[0093] Also disclosed is a public service system (e.g., an electric power public service system) and / or a communication system, the system comprising one or several devices (such as intelligent electronic devices (IEDs)) and a system capable of operating for debugging, configuring and / or controlling the operation of the public service system or the communication system according to an embodiment.

[0094] The method, apparatus and system according to the present invention may be used to generate and / or use processing logic capable of automatically processing a textual system description.

[0095] The methods, devices, and systems allow for the generation of labeled text descriptions that can be used for training AI models (but are not limited to such).

[0096] The labeled text descriptions may be generated using automatic tagging, which includes performing clustering techniques based on semantic distances.

[0097] The method, device and system according to the invention may be used in connection with a public service system and / or a communication network of a public service system or an IACS, for example a communication network of a power generation, transmission or distribution system.

[0098] Again, the text descriptions (eg, text descriptions included in a training batch and / or text descriptions included in an annotated training batch and / or text system descriptions being processed) may all relate to the same domain or be part of the same domain.

[0099] Compared with the prior art, in particular, compared with the system and method disclosed in document D1 (US2021 / 0264111 A1), the present invention provides several advantageous technical effects, including the following:

[0100] - The present invention does not necessarily require a weighted loss function that applies a first weight to clean data and at least one second weight to noisy data, thereby reducing model and process complexity, thereby saving processing resources.

[0101] - The present invention does not necessarily require a step of de-lexicalizing a set of unannotated sentences, and since this step is a pre-processing step, data pre-processing time / computation time can be saved again.

[0102] - The present invention does not necessarily involve separate source and target domains, and therefore does not have to invoke a (heuristic) mapping between the source and target domains, thereby improving the accuracy of the procedure.

[0103] - The present invention does not rely on the use of deep neural networks (DNNs) to generate any intent annotations, but instead uses semantic distances to previously annotated text clusters to assign labels (intent categories) to unannotated / unseen text. Clustering is a relatively less computationally intensive process.

[0104] - The purpose of D1 appears to be to generate annotations / labels. In other words, to generate training data without training an AI model based on such data. In contrast, the present invention annotates the data, trains an AI model based on such data, and can then refine the annotations again if necessary. Therefore, there is a significant difference compared to the scope of D1. Since the claimed invention actually trains an AI model based on the annotations, it provides the advantage of validating the accuracy of the model, which depends on the accuracy of the annotations.

[0105] - D1 uses a heuristic model to rename annotations for the target domain. In the case where the source domain has incorrect annotations, there is also the possibility of transferring the wrong annotations to the target domain. Moreover, even though a "heuristic mapping" of annotations is mentioned, the amount of detailed information D1 gives about this mapping is very limited. For example, it is not clear how potentially wrong annotations can be corrected. In contrast, the present invention provides the possibility to correct wrong annotations. This can provide the end user with fine control over the entire annotation process. In addition, merge and split operations can be called to allow a complete change of a set of labels, rather than just one label (which is usually the case for manual editing operations).

[0106] - D1 essentially transfers / maps annotations from a dataset belonging to a source domain to a different target domain. The mapping is expected to work when the two domains involve similar intents or operations (e.g., buying food vs. booking movie tickets). However, in cases where the domains and intents are different, e.g., buying tickets vs. getting the expected router specifications from a tender document, the proposed mapping may be potentially error-prone. In other words, D1 is somewhat limited in scope, while the present invention can be applied to any domain without making any assumptions.

[0107] - Due to its typically rather broad scope, the present invention has a more expressive ability to capture multiple intent annotations in the context of, for example, an RFP. For example, two text items "a router should have a speed of 10 Gbps" and "a switch should have a speed of 10 Gbps" might be assigned the same label by D1. However, these two texts indicate two different technical requirements belonging to two different technical devices, potentially related to two different organizations. The present invention allows to distinguish between them by enabling the creation of two different labels (e.g., "intent_router" and "intent_switch", respectively).

[0108] - The foregoing arguments are particularly important and advantageous where the present invention is applied to the automatic configuration / debugging of technical systems (e.g. public service systems, industrial systems, computing systems or networks and / or communication networks), in particular large-scale systems that require a lot of time and effort for manual configuration / debugging, thereby reducing configuration errors and system failures.

[0109] -D1 generates annotations for data from a target domain. In other words, D1 only generates training data for the target domain. D1 does not actually train an AI model for the target domain. Although training data is important, it does not produce an AI model by itself. In other words, the user still needs to use this data to train an AI model. In contrast, the present invention provides an end-to-end method for 1) automatically or semi-automatically annotating data from a given domain, and 2) using this data to train an AI model for a related domain.

[0110] - Continuing with the above argument, any computation time reported by D1 only involves the generation of training data. D1 does not appear to report the time to train the AI ​​model for the target domain. Since the present invention can use clustering to assign labels (intents), the normalized time spent generating labeled data can be much less than D1.

[0111] -Still continuing with the previous argument, in D1, the user is left to identify and correct any potential errors in the annotations by themselves. In contrast, the present invention can adopt an incremental batch-by-batch labeling and training method that allows the user to: 1) annotate some new text items; 2) train a new AI model; and 3) verify the accuracy (or any other performance metric) of the new model after adding new data points. In addition, if the performance of the new model deteriorates after adding new data, the user can revert to a previous better configuration and start labeling again (see e.g. Fig.19 as well as Fig.12 Operations 172 and 183 in, discussed further in this regard below).

[0112] -This approach is different from the typical process of training AI models, because the training data set in the typical process usually remains fixed. In contrast, the present invention can iteratively train AI models based on dynamically growing or shrinking data sets.

[0113] - For example, the present invention covers a label-train loop that can be run until a certain performance threshold is reached. Since D1 does not train an AI model for a target domain, there is no similar loop in D1. Moreover, adding such a procedure may require significant changes in the architecture proposed in D1.

[0114] In summary, the present invention provides significant advantages, especially in the following aspects:

[0115] - A more expressive capability to capture multiple kinds of annotations related to technical texts, e.g. those in the domain of bids, tenders and RFPs, and those in the domain of automatic configuration / debugging information for (large-scale) technical systems and their equipment.

[0116] - Lower "normalization" computational cost for adding annotations to unseen text.

[0117] -The ability to train AI models based on annotations and to modify such annotations in conjunction with the performance of the target AI model. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] The subject matter of the invention will be explained in more detail with reference to preferred exemplary embodiments illustrated in the accompanying drawings, in which:

[0119] Figure 1is a schematic representation of a system operable to generate processing logic.

[0120] Figure 2 is a flow chart of the method.

[0121] Figure 3 is a schematic representation illustrating clustering based on semantic distance.

[0122] Figure 4 is a flow chart of the method.

[0123] Figure 5 is a schematic representation of the system.

[0124] Figure 6 is a schematic representation of the system.

[0125] Figure 7 is a flow chart of the method.

[0126] Figure 8 is a schematic representation of an artificial intelligence (AI) model.

[0127] Fig. 9 is a schematic representation of an artificial intelligence (AI) model.

[0128] Fig.10 is a schematic representation of an artificial intelligence (AI) model.

[0129] Fig.11 is a block diagram representation of the method.

[0130] Fig.12 is a flow chart of the method.

[0131] Fig.13 is a flow chart of the method.

[0132] Fig.14 is a schematic representation of an exemplary tag.

[0133] Fig.15 is a schematic representation of exemplary additional tags.

[0134] Fig.16 is a flow chart of the method.

[0135] Fig.17 An annotated text item is illustrated having a label including an intent category and at least one attribute.

[0136] Fig.18 is a flow chart of the method.

[0137] Fig.19 is a schematic representation of data structures that may be used in a method and / or by a system according to an embodiment.

[0138] Fig. 20 is a flow chart of the method.

[0139] Fig.21 is a flow chart of the method.

[0140] Fig. 22 is a diagram illustrating the system. DETAILED DESCRIPTION

[0141] Exemplary embodiments of the present invention will be described with reference to the accompanying drawings, in which the same or similar reference numerals represent the same or similar elements. Although some embodiments will be described in the context of other components of a communication system or a public service system (such as a power generation, transmission and / or distribution system), the methods, devices and systems described in detail below can be used in a variety of systems (such as other public service systems or industrial systems).

[0142] Unless specifically stated otherwise, the features of the embodiments may be combined with each other.

[0143] According to an embodiment of the present invention, a clustering technique based on semantic distance is used to automatically label text descriptions related to a technical system (such as a public service system, a communication system, or a communication system of a public service system). The automatically labeled text descriptions can be used to train an artificial intelligence (AI) model, but are not limited thereto.

[0144] The AI ​​model may have an input layer for receiving text items, an output layer for outputting at least one label of the text labels (and optionally both intent categories and one or more attributes of at least some of the text items), and a hidden layer. The AI ​​model may include a natural language processing (NLP) transformer.

[0145] The trained AI model may be used to process the text system description. The results of the processing performed by the trained AI model acting as processing logic may be used to debug, configure and / or control the operation of one or several devices of a public service system (such as a power system), a communication system, or a communication system of a public service system, but is not limited thereto.

[0146] As used herein, "annotated text description" refers to a text description that includes labels for text items of the annotated text description. All, most, or at least most of the text items of the text description may, but are not required to, be provided with labels.

[0147] Any text item can be equipped with more than one tag. To illustrate, a text item can be tagged with an intent and one or several optional attributes for the intent.

[0148] As used herein, the term "text item" refers to a portion of a natural language text. A text item may include or may be a sentence, a phrase, a word, and / or another alphanumeric term (such as a numerical value and a unit).

[0149] Figure 1 is a representation of a system 10 including a data storage device 40 and a processing logic generation system 20 .

[0150] The data storage device 40 is operable to store a number of text descriptions of a technical system, such as a public service system, for example, an electric power system or a communication system. The number of text descriptions may include specifications of one or several technical systems, which may be of the same technical system type, such as a substation, or may be assigned to various different technical system types. A first subset of the text descriptions may be annotated with tags. A second subset of the text descriptions may not be annotated before being processed by the processing logic generation system 20. The data storage device 40 may be separate from the processing logic generation system 20, but may also be a component thereof.

[0151] The processing logic generation system 20 is generally operable to automatically assign tags to text items included in the text description that have not yet been annotated. The processing logic generation system 20 can be operated to generate a larger set of annotated text descriptions that can be used for AI model training, thereby generating processing logic. The processing logic generation system 20 can be operated to deploy the generated processing logic for execution by another computing device. Alternatively or in addition, the processing logic generation system 20 can be operated to execute the generated processing logic to perform debugging, configuration and / or operation tasks of a technical system (such as a public service system or a communication system).

[0152] The processing logic generation system 20 comprises a first interface 21 operable to retrieve a system description included in the first subset (ie, annotated system description) and / or a system description included in the second subset (ie, unannotated system description).

[0153] The processing logic generation system 20 may include a processing system 30. The processing system 30 may include one or more circuits, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), (multiple) controllers, (multiple) processors, (multiple) superconducting circuits, quantum bit (qubit) processing circuits, but is not limited thereto.

[0154] The processing system 30 may be operable to temporarily store the system description in the storage system 23. The processing system 30 may be operable to perform an automatic tagging process 31. The automatic tagging process may be performed to generate tags for text items included in the text descriptions of the second subset (ie, the initially unannotated text descriptions).

[0155] The labeling may be performed as part of a training set generation process 32 that automatically generates a training set for AI model training. The processing logic generation system 20 may be operable to automatically generate at least some annotated text descriptions included in a training batch for training an AI model.

[0156] The processing system 30 may be operable to obtain information about the topology of one or more candidate AI model architectures from the storage system 23. For illustration, the storage system 23 may include information about the AI ​​model architecture (specifying hyperparameters, such as the number of encoder or decoder layers, etc.). The processing system 30 may obtain the architecture for training. The processing system 30 may be operable to control a user interface so that a user can enter information specifying which AI model architecture will be used for AI model training.

[0157] The processing logic generation system 20 may have a second interface 22. The trained AI model may be output via the second interface 22 as processing logic for execution by a debugging, configuration, and / or system control device. Alternatively or additionally, the results obtained by executing the trained AI model as processing logic may be output via the second interface 22. This may be done to perform debugging, configuration, and / or system control operations, but is not limited thereto.

[0158] The processing logic generation system 20 may be operable to automatically annotate the text description using a clustering technique, such as a semantic distance-based clustering technique.

[0159] The processing logic generation system 20 may be operable to perform an iterative process in which a set of labels from which labels assigned by the processing logic generation system 20 may be selected and / or the assigned labels may be modified. This may include controlling a user interface to enable a user to modify the set of labels and / or the assigned labels. Alternatively or additionally, the processing logic generation system 20 may be operable to automatically modify the set of labels and / or the assigned labels.

[0160] Figure 2 is a flow chart of method 50. The method may be automatically performed by or using a computing system. The method may be automatically performed by processing logic generation system 20.

[0161] At step 51, automatic labeling is performed on the text descriptions included in the training batch. Automatic labeling can be performed using a clustering technique based on semantic distance. Automatic labeling can assign labels (which can include intent categories and optionally one or several attributes of at least some of the intent categories) to the text items. This can be performed for some, at least 30%, at least 40%, at least 50%, at least 60%, at least 70%, at least 80%, at least 90%, or all of the text items in the text description.

[0162] At step 52, the AI ​​model may be trained using the annotated text description. The AI ​​model may have an input layer operable to receive text items, a natural language processor (NLP), and an output layer operable to output labels for the text items. The training may use various implementations, such as supervised learning techniques, but is not limited thereto.

[0163] At step 53, text may be input to the trained AI model. Testing may include evaluating performance based on annotated text descriptions included in the data storage device 40 that were not used for training. Thresholding may be used to determine whether the trained AI model meets or fails to meet the performance criteria. Depending on the results of the performance evaluation, steps 51-53 may be repeated in an iterative manner until the performance criteria are met or another termination criterion is met (such as a comparison of the number of completed cycles to a termination threshold).

[0164] At step 54, the trained AI model determined in the last iteration may be deployed as processing logic. Alternatively or additionally, the processing logic may be executed by a debugging, configuration and / or system control tool to process a text description of a technical system to be debugged, configured and / or controlled and perform debugging, configuration and / or system control tasks based thereon.

[0165] The processing logic generation system 20 and / or the method 50 can use the clustering based on semantic distance to automatically assign the label to the text item. The text item can be assigned to the given label included in a set of labels based on the average semantic distance from at least one, at least two, most or all text items previously assigned to the given label. The text item can be assigned to the label with the smallest average semantic distance among the calculated average semantic distance. This can be conditionally performed according to the quality standard. For illustration, the text item can be automatically marked with the label with the smallest average semantic distance among the calculated average semantic distance, provided that the minimum value is less than or does not exceed the distance threshold. Otherwise, the text item can remain unmarked.

[0166] Figure 3 Clustering based on semantic distance is illustrated. Figure 3 shows the space spanned by the semantic features. Figure 3 A two-dimensional graph is shown in FIG. 1 for illustration, but it should be understood that more than two semantic features (e.g., three or more, four or more, five or more, ten or more, twenty or more, fifty or more semantic features) may be used in the clustering technique.

[0167] The text item 60 can be obtained from an unannotated text description. The text item 60 can be a sentence, a phrase, a word, or a combination of a numerical value and a unit.

[0168] In an annotated text description (e.g., a text description stored as a manually annotated document in the data storage device 40 and / or annotated by a human expert in response to a control operation of the processing logic generation system 20 that enables the human expert to enter annotations), there may be several clusters 61-63 of annotated text items. The clusters 61-63 may include a first cluster 61, a second cluster 62, and a third cluster 63. There may be, and typically are, more clusters.

[0169] All annotated text items 71, 72 in the first cluster 61 are labeled with the same or similar first label. For illustration, all text items 71, 72 in the first cluster 61 may have been previously labeled with the same first intent category. The text items 71, 72 in the first cluster 61 may, but need not, be labeled with the same attributes.

[0170] All annotated text items 73, 74 in the second cluster 62 are labeled with the same or similar second label. For illustration, all text items 73, 74 in the second cluster 62 may have been previously labeled with the same second intent category. The text items 73, 74 in the second cluster 62 may, but need not, be labeled with the same attributes.

[0171] All annotated text items 75, 76 in the third cluster 63 are labeled with the same or similar third label. For illustration, all text items 75, 76 in the third cluster 63 may have been previously labeled with the same third intent category. The text items 75, 76 in the third cluster 63 may, but need not, be labeled with the same attributes.

[0172] The semantic distance of the text item 60 to be tagged from the existing clusters 61-63 may be determined.The text item 60 may be tagged (ie, assigned a label, which may include an intent category and optional attributes) based on the semantic distance.

[0173] The semantic distance may be determined from a metric using any of a variety of conventional ways of determining semantic distance.

[0174] The semantic distance may be determined by calculating the semantic distances in the space spanned by two or more semantic features of the text item 60 from at least one text item assigned to each of the several clusters 61 - 63 , respectively.

[0175] Averaging techniques can be used. For illustration, the average semantic distance of the text item 60 from the first cluster 61 can be calculated by averaging the distances 81, 82 between the text item 60 to be marked and two, more than two, or all text items 71, 72 that have been previously determined to belong to the first cluster. The average semantic distance of the text item 60 from the second cluster 62 can be calculated by averaging the distances 83, 84 between the text item 60 to be marked and two, more than two, or all text items 73, 74 that have been previously determined to belong to the second cluster. The average semantic distance of the text item 60 from the third cluster 63 can be calculated by averaging the distances 85, 86 between the text item 60 to be marked and two, more than two, or all text items 75, 76 that have been previously determined to belong to the third cluster.

[0176] A label (eg, intent category and optional attribute(s)) for the text item 60 may then be determined based on the determined semantic distance.

[0177] As mentioned above and Figure 4 As further illustrated in , a text item to be marked can optionally remain unmarked.

[0178] Figure 4 is a flow chart of method 90 .

[0179] At step 91 , the average semantic distance of the text item to be marked from a number of clusters is determined.

[0180] At step 92, it is determined whether at least one of the average semantic distances meets a proximity criterion. The proximity criterion may include that at least one of the average semantic distances is less than or not greater than a distance threshold.

[0181] If it is determined that the proximity criteria are met, the text item is automatically labeled at step 93. The text item may be labeled with the label (eg, intent category and optional attribute(s)) with the smallest average semantic distance.

[0182] At step 94, if it is determined that the proximity criteria are not met, the text item may remain unmarked temporarily. The processing logic generation system may control the user interface to enable the user to mark the text item. Alternatively or additionally, the program 90 may be re-executed later. For illustration, if tags are added and / or modified, the proximity check performed at step 92 may produce different results in subsequent iterations.

[0183] Figure 5 is a schematic representation of system 120 .

[0184] System 120 includes processing logic generation system 20, which may be operated and / or designed as described herein.

[0185] System 120 includes processing logic execution system 110. Processing logic execution system 110 may be coupled to processing logic generation system 20 via a local area network (LAN) or a wide area network (WAN).

[0186] The processing logic execution system 110 may receive the processing logic generated by the processing logic generation system 20. The processing logic may be stored in the processing logic execution system 110 in a non-volatile manner. The generated processing logic may be an AI model trained by the processing logic generation system 20 and / or trained using the method disclosed herein.

[0187] The processing logic execution system 110 may have a first interface 111 operable to receive a textual system description 100. The textual system description may be unannotated. The processing logic execution system 110 may be operable to execute processing logic to generate an annotated textual system description 100, wherein some, most, or all text items of the textual system description 100 have been tagged with labels (e.g., with intent categories and optional attributes).

[0188] The processing system 110 may include one or more circuits 112 for executing processing logic, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), (multiple) controllers, (multiple) processors, (multiple) superconducting circuits, quantum bit (qubit) processing circuits, but are not limited thereto.

[0189] The processing logic execution system 110 may have a second interface 113. The results of the execution of the processing logic by the processing logic execution system 110 may be used to generate control signals. The control signals may be generated to at least one of: control the operation of the (multiple) devices 121 of the technical system to which the text description 100 relates; control a debugging and / or configuration apparatus or tool 122 to debug and / or configure at least some of the (multiple) devices 121 to operate in the technical system to which the text description 100 relates; and / or control a user interface, such as a graphical user interface (GUI) 123, to output information about the processing results and / or enable user input actions in response to the execution results of the processing logic applied to the text description 100.

[0190] The processing logic generation and execution system can be formed as a whole, such as Figure 6 As shown.

[0191] Figure 6 A processing logic generation and execution system is shown, which is capable of operating to perform automatic tagging, (optional) AI model training and / or other tasks, and also includes a processing logic execution routine 34 for executing the generated processing logic to process a text description 100 received by the combined processing logic generation and execution system 20.

[0192] The labels may be selected from a predefined set. The labels may include more complex structures. For illustration, the label may include an intent category selected from a set of intent categories. For at least some intent categories (and depending on the intent category), the label may include one or several attributes. The attribute(s) may vary depending on the intent category. Assigning labels may include assigning intent categories and assigning attribute(s) to those intent categories for which the attributes exist.

[0193] Various intent categories may relate to different technical devices of a technical system (such as communication switches, communication firewalls; and / or circuit breakers, power system switches, distributed energy resources, energy storage devices; and / or controllable valves, controllable pumps, etc.).

[0194] Figure 7 is a flow chart of method 130. The method may be automatically performed by the processing logic generation system 20.

[0195] At step 131, an intent category is automatically determined for a text item to be tagged. The intent category may be determined based on an average semantic distance of the text item to be tagged from text items belonging to clusters assigned to different intent categories.

[0196] At step 132, attribute(s) may be automatically determined for the text item to be tagged. Step 132 may be selectively performed depending on whether the determined intent category supports the attribute(s).

[0197] At step 133 , the text item may be tagged with the intent category, and if the intent category supports attribute(s), the text item may be tagged with the determined attribute(s).

[0198] Figures 8 to 10 An AI model architecture that can be used is shown.

[0199] Figure 8 is a schematic representation of an AI model 140 that may be used by the processing logic generation system 20 and / or used in the methods disclosed herein.

[0200] The AI ​​model may have an input layer 141 operable to receive text items included in a text system description. The AI ​​model may have an output layer 142 operable to output annotations (i.e., labels such as intent categories and optional attribute(s)) for text items included in the text system description.

[0201] The AI ​​model may have one or several hidden layers 143 .

[0202] During AI model training, parameters of the AI ​​model 140 may be set to improve the performance of the AI ​​model in its intended task (ie, automatic labeling).

[0203] Fig. 9 is a schematic representation of an AI model 150 that may be used by the processing logic generation system 20 and / or used in the methods disclosed herein.

[0204] The AI ​​model may have an input layer 141 operable to receive text items included in a text system description. The AI ​​model may have an output layer operable to output annotations (i.e., labels such as intent categories and optional attribute(s)) for text items included in the text system description.

[0205] The AI ​​model may include an NLP transformer 151. The NLP transformer 151 may include an encoder-decoder model with a self-attention layer.

[0206] The AI ​​model may include a conditional random field (CRF) layer 144 that receives the output of the NLP transformer 151 .

[0207] During AI model training, parameters of the AI ​​model 150 may be set to improve the performance of the AI ​​model in its intended task (i.e., automatic labeling).

[0208] Fig.10 is a schematic representation showing a more detailed implementation of an AI model 150 that may be used by the processing logic generation system 20 and / or used in the methods disclosed herein.

[0209] The NLP converter 151 may include an encoder-decoder model. The NLP converter 151 may include an encoder-decoder model with a self-attention layer.

[0210] The encoder of the NLP transformer may include a stack of encoder layers 152, 153, each encoder layer 152, 153 including a self-attention layer 152a, 153a and a feed-forward network 152b, 153b.

[0211] Each feed-forward network 152b, 153b may be a fully connected 2-layer network.

[0212] The techniques disclosed herein may be applied to a textual description that is or includes a request for proposal (RFP), but is not limited thereto.

[0213] The systems and methods disclosed herein may be operable to:

[0214] - Extracting text from many (eg, hundreds to thousands) of textual system descriptions (eg, RFPs, but not limited to) belonging to a domain (eg, belonging to a public service system, such as an electric power system).

[0215] - Enable domain experts to annotate a small portion of the textual system description by assigning appropriate labels to sentences or lines. This can constitute an initial annotated dataset. The labels can act as clusters.

[0216] -Process the remainder of the extracted text in multiple batches.

[0217] -For each individual text item in the batch, find the closest matching label by considering the semantic distance between the item and the already assigned data points in the cluster (label).

[0218] -At the end of the batch, the AI ​​model is trained based on the available dataset.

[0219] - Iterate the prediction errors and reassign labels (if necessary). Automatic recommendations on modifying (e.g., splitting / merging) labels may be provided when appropriate. Reassignment of labels may include controlling a user interface to enable a user to reassign or otherwise modify labels.

[0220] These aspects are discussed in more detail below.

[0221] The systems and methods disclosed herein allow for the definition and subsequent refinement of data models based on information extracted from textual system descriptions.

[0222] The disclosed systems and methods may include using supervised learning techniques to generate a labeled dataset (eg, a collection of annotated text descriptions).

[0223] The systems and methods disclosed herein enable the generation and selection of AI models trained with labeled data based on performance.

[0224] Fig.11 An automated text annotation process that can be used to create a labeled data set is illustrated. The process can be triggered after text from a textual system description (eg, a large text description such as an RFP) is extracted and stored in the data storage device 40.

[0225] At 161, a portion of the available text may be annotated by selecting appropriate tags. This may be done by controlling a user interface to request expert input for tagging. Such initial tagging may, but need not, form part of the method. That is, the disclosed technique may also start from an initial data set that already includes (some) tagged text descriptions in addition to (more) untagged text descriptions.

[0226] For example, a sentence in a textual system description may specify expected features of a communications switch (e.g., "it shall support Rapid Spanning Tree Protocol"). This sentence may be tagged "switch_specification". In general, technical specifications and optional other information from a textual system description may be tagged. For example, the sentence "the bid shall be valid for 180 days" may be tagged "bid_validity". New tags may be created manually if desired.

[0227] At 161, a startup training may be performed. Annotations may be added to the initial data set. This may continue until the desired volume and performance are achieved. While the desired volume and performance may depend on the context, an exemplary goal is to achieve at least 100 data points per label. With such volume, one may aim to achieve a precision and recall of 75-85% each. These thresholds will be determined appropriately.

[0228] After the initial data model is trained with the initial set of data points, at step 162 , the remaining text extracted from the textual system description may be automatically annotated in batches.

[0229] For this purpose, each label (e.g., intent) can be considered as a cluster. All texts annotated with that particular label can be considered as points or items belonging to that particular cluster.

[0230] For illustration, a single unannotated text item will be considered. The semantic distance of the text item from each point of this cluster can be calculated, and the average distance from this cluster can be obtained accordingly. The process can be repeated for each label or cluster. The cluster with the smallest average semantic distance can be determined. This may be the cluster to which the text item in question should belong. In other words, the text item is annotated with this particular cluster label.

[0231] This process can be repeated for all unannotated text items in the current batch.

[0232] At step 162, the initial dataset is augmented with new annotations obtained from the current batch.

[0233] At step 164, a new AI model is trained based on the new augmented data set. The model can be tested using test data. Evaluation can involve controlling a user interface to enable a user to input feedback. Alternatively or additionally, evaluation can be performed automatically. For illustration, the user interface can be controlled so that the user can be presented with actual labels (annotations) indicating text items and prediction errors of predicted labels. Based on domain knowledge and understanding, the user can (1) change the annotation to a predicted annotation, (2) keep the original annotation, or (3) delete the annotation and text item. The process can be repeated for some, most, or all prediction errors.

[0234] The process can be repeated iteratively. For illustration, a new model can then be trained. The labels can be further refined based on the prediction errors, or the current batch can be marked as completed.

[0235] Automatic tagging

[0236] When generating an initial set of annotated text descriptions (e.g. Fig.11 ) and when generating an updated annotated text description (e.g., Fig.12 In step 162 in, automatic labeling can be performed in subsequent iterations of the process.

[0237] Automatic labeling can be performed by performing an automatic annotation process based on supervised clustering.

[0238] Starting with an initial dataset of intent annotations, supervised clustering methods such as k-nearest neighbors are used to automatically generate a larger set of data points. Based on a pre-trained natural language understanding (NLU) model, intent labels can be assigned to a set of unseen plain text data extracted from textual system descriptions. An NLU model is an example of an AI model or at least a portion of an AI model used in this article.

[0239] This can be implemented as follows:

[0240] X i represents a non-empty set of labels (such as intentions) i, Vi.xi j Indicated by label X i annotated text item, y represents the unannotated text item. d(y, z) represents the semantic distance between two text items y and z.

[0241] In the case where two sentences are very similar, their semantic distance is close to 0. On the other hand, the semantic distance between two very different sentences is close to 1.

[0242] D(y,X i )=(1 / |X i |)∑ j d(y,x ij ) represents y and label X i The average semantic distance between

[0243] The label (or cluster) to which the text y belongs can be determined as

[0244] In this way, text y can be automatically annotated with label Y. However, if the average semantic distance is greater than a predefined threshold, annotation of the text item can be skipped.

[0245] The above process can be repeated for all unannotated text items in the current batch.

[0246] Fig.12 The above-described method 170 is illustrated.

[0247] At step 171 , the method starts.

[0248] At step 172 , a configuration may be loaded, which may include previously marked text items and their labels.

[0249] At step 173, a new text item may be selected to be marked.

[0250] At steps 174 - 177 , the average semantic distance of the text item to be labeled from a number of previously identified clusters (labels) is determined.

[0251] At step 178 , it is determined whether the text item satisfies a proximity criterion (eg, a minimum distance D(y, Y)) that is less than or not greater than a predefined threshold.

[0252] At step 179, the determined tags may be added.

[0253] At step 180, it is determined whether all text items in the batch have been considered. Steps 173-179 may be repeated until all text items have been considered.

[0254] At step 181, attributes may be added. To illustrate, entities may be annotated in sentences that have been annotated with intent. This may be done automatically. Annotation with entities may include controlling a user interface to allow a user to provide entity annotations.

[0255] At step 182, the previously existing set of annotated text descriptions is augmented by adding the annotated text descriptions to which tags have been added in steps 173-181.

[0256] At step 183, the configuration may be saved.

[0257] At step 184, the method ends.

[0258] Additionally, after all relevant text in the unseen dataset has been annotated with intents, entities (if any) can be annotated. Pre-trained NLU models and / or other advanced techniques (such as conditional random fields) can be used to identify entities within intent-annotated sentences. The phrases so identified are then annotated with entities.

[0259] The preceding supervised clustering-based intent annotation process ends with augmenting the initial training dataset with the newly generated dataset. The configuration (discussed later) is saved for future use.

[0260] Data Model

[0261] A data model specifying possible labels may be used. Generation of the data model may precede the techniques disclosed herein.

[0262] The processing logic generation system 20 can be operated to control a user interface to allow a user to define a data model. Before the text is annotated, the underlying data model can be defined. The data model can be refined at a later stage. The processing logic generation system 20 can be operated to control a user interface to allow a user to add a set of tags or categories to the system. These tags can relate to products and services that can be used in the technical system to which the text system description relates. For example, if the text description relates to a communication system that may include a network switch, a "switch_specification" tag can be created to capture technical specifications related to the network switch. The name indicated in the tag can be singular, but this is not mandatory. Consistency in naming is generally expected. Tags can also relate to other information of interest. For example, "bid_validity" is another tag indicating the validity period of a bid submitted in response to the text system description.

[0263] Fig.13 An iterative process 190 of data model generation is illustrated. The tags or categories may be hierarchical in nature. For example, the tags "core_switch_specification" and "edge_switch_specification" may be labeled as subcategories of the parent category "switch_specification". Additionally, for each tag (category), a set of attributes are defined. For example, a switch may have an attribute "protocol" that captures all protocols that need to be supported. In the case of a hierarchical model, the attributes of a parent category are inherited by all of its subcategories. The process may be repeated until all desired categories (intents) and attributes (entities) are added to the system.

[0264] At step 191, the process begins.

[0265] At step 192, a new category is added. This may include accessing a database (e.g., an inventory database) to automatically retrieve possible labels and / or controlling a user interface to allow a user to specify (e.g., enter or select) a label for the new category.

[0266] At step 193, new attributes may be added. This step may be optional depending on the categories (e.g., intent categories) defined at step 192. Attributes may be determined by accessing a database (e.g., an inventory database) to automatically retrieve possible labels and / or controlling a user interface to allow a user to specify (e.g., enter or select) a label for a new category.

[0267] At step 194, it may be determined whether the category of step 192 is a subcategory. If so, step 195 allows the category to be associated with its parent category.

[0268] At step 195, it is determined whether all categories have been added. If not, the method returns to step 192.

[0269] At step 197, if all categories (possible labels) have been added, a set of possible labels is generated. A label can be selected from this set in automatic labeling.

[0270] At step 197, two sets of labels may be generated. The first set of labels may contain intents that need to be captured from the text system description. The second set of labels may correspond to entities that need to be captured from the intents. The processing logic generation system 20 may be operable to control a user interface so that multiple sets of intent labels and entity labels are displayed to a user in a graphical user interface (GUI).

[0271] At step 198, the method ends.

[0272] Fig.14 An example of a data model 200 that can be used to capture technical specifications from a textual system description of a communicating (e.g., networking) system is shown. Rectangles indicate labels (or intents), while ovals represent attributes (entities). The specific intents present in the data model depend on the corresponding technical system and the type of product or service used therein.

[0273] Fig.15 Another data model 210 for capturing other data from a textual system description is shown. These intents and entities may (but not necessarily) aid in the specific design and configuration of a system. For example, it may be desirable to be able to parse eligibility criteria in order to determine whether a bid can be made. Fig.15 The other data models 210 may be suitable for broader text descriptions and may be combined with the more domain-specific data models 200 of the corresponding technical system domain.

[0274] In addition to (or in some cases also as an alternative to) data model 200 , other data models 210 may be used.

[0275] Initial data generation

[0276] An initial data model (which may be determined as described above) may be used to prepare an initial dataset of annotated text. Fig.16 Process 220 is illustrated. Initial data generation may be, but need not be, a method step of the methods disclosed herein. To illustrate, the processing logic generation system 20 and / or the methods disclosed herein may be operable to perform automatic tagging based on previously annotated text descriptions.

[0277] The text system description may contain unstructured text in heterogeneous formats. The processing logic generation system 20 may execute a module to extract the content of the text system description in plain text format. The processing logic generation system 20 may allow a user to annotate sentences. The processing logic generation system 20 may control a user interface to allow a user to select a sentence (or other text item) in a graphical user interface (GUI). For example, the processing logic generation system 20 may allow a user to select a sentence as a whole by dragging a mouse or any other pointing device. The processing logic generation system 20 may allow a user to select an appropriate intent (label). In other words, in the initial data generation, a label may be assigned to a sentence that completes its intent annotation.

[0278] One or more phrases in such annotated sentences may contain useful information. The processing logic generation system 20 may allow the user to select the text of such a phrase using the aforementioned process. Subsequently, the selection of the appropriate attribute (entity) may be achieved by controlling the user interface. This completes the entity annotation for this particular phrase. The process may be repeated to annotate all entities (if any) in the sentence that has been annotated with the intent.

[0279] Entity annotations are optional. For illustration, informative phrases may not be available in some sentences.

[0280] Fig.16 is a flow chart of a method 220 operating as described above.

[0281] The method starts at 221. At step 222, sentences are extracted from a technical system description (such as, but not limited to, an RFP).

[0282] At step 223 , the control user interface receives input to annotate the sentence with an intent.

[0283] At step 224, the user interface may be controlled to receive input to provide entity annotation. As described above, this step is optional, or may be selectively performed only for a portion of the text item.

[0284] At step 225, the annotation is saved.

[0285] At step 226, the NLU model may be trained.

[0286] The method ends at step 227 .

[0287] Fig.17 Illustrated is an example sentence 230 annotated with an intent 231 and a phrase annotated with an entity 232. The annotations 231, 232 may be obtained from a data model such as the data models 200, 210 explained above.

[0288] The foregoing intent and entity annotation process can be repeated for plain text extracted from multiple text system descriptions. All of these annotations can be saved. Finally, the processing logic generation system 20 can use the saved annotations to train a natural language understanding (NLU) model that allows the identification of intents and entities from a given text.

[0289] Modification of tags (e.g., intent)

[0290] In some cases, the data model (which may include labeled text items and associated labels) may not be sufficient to capture all intents accurately enough. Available labels and / or already assigned labels are modified. This can be done after the AI ​​model is trained, respectively in Fig.11 Therefore, as more training data becomes available and increases in diversity, the data model can be improved.

[0291] Let X and Y be two different intents (labels). Let a(X,Y) be the relative ambiguity between X and Y. After testing a previously trained NLU model with test data, the relative ambiguity can be obtained based on the predicted output. For example, the strength of a cell in a confusion matrix can be determined and the strength can provide an indication of the relative ambiguity. Such an indication of the relative ambiguity can be output.

[0292] If a(X,Y) is less than a predefined threshold, then there is no significant ambiguity between the two classifications. In other words, the two intent classes (labels) X and Y appear to be clearly distinct, and therefore they are retained. On the other hand, if the relative ambiguity between any two intents exceeds a predefined threshold, then the two intent classes may lack sufficient unique features. It can be automatically determined whether the two classifications are sufficiently distinct to allow reliable differentiation.

[0293] If the two classifications are not sufficiently distinguished, the user interface may be controlled to allow one of the following actions to be taken:

[0294] -Change the intent label of one or more sentences until the NLU model improves.

[0295] - Do nothing, i.e. keep your mind Figure X and Y both.

[0296] - Combine X and Y into a single intent category, say Z, and use either of the labels to identify the latter. Thus, all text tagged with X or Y is automatically re-annotated with label Z.

[0297] - Create a new set of data points Z and move the most similar data points from X and Y to Z; create two new intent categories XZ and YZ, which contain data points that exist in X but not in Z and those that exist in Y but not in Z, respectively.

[0298] The programming logic generation system 20 may be operable to enable a user to be requested to select one of these options.

[0299] In response to the user selecting the second option, programming logic generation system 20 makes no changes.

[0300] In response to the user selecting the third option, the existing text annotated as X may be annotated as Y; X may be deleted by the programming logic generation system 20. Alternatively, the existing text annotated as Y may then be annotated as X; Y may be deleted by the programming logic generation system 20. A new set of configurations may be stored in the database.

[0301] In response to the user selecting the fourth option, the programming logic generation system 20 may store a new set of configurations in the database.

[0302] Fig.18 An implementation of label modification is illustrated. The programming logic generation system 20 may allow modification of an existing intent (steps 243-245). The programming logic generation system 20 may allow modification of an intent label (steps 248-250).

[0303] The two branches (i) steps 243-245 (modify existing intent) and (ii) steps 248-250 (modify label) can be performed in a mutually exclusive manner in each iteration of the iterative process. After the AI ​​model training has been performed at step 241, a decision can be made at step 242 as to which branch to follow.

[0304] The decision as to which of the two branches (modifying existing intent or managing labels) to use in any iteration of the iterative process may be made automatically at step 242 (eg, depending on the ambiguity a(X,Y)) or via user input through a control user interface.

[0305] At step 248, two or more sub-intents may be merged with the parent intent. For example, consider that the intents edge_router_specification and core_router_specification are sub-intents of the intent router_specification. In the event that there is not enough data to distinguish between the two sub-intents, they may be merged together into a single router_specification intent. All existing annotations are automatically updated. This process may be automatically performed by the processing logic generation system 20.

[0306] At step 249, the programming logic generation system 20 can retrain the AI ​​model based on these configurations. The newly trained model can be tested. New test results (optionally including relative fuzziness metrics) can be output, for example, by controlling the user interface.

[0307] At step 250, it may be determined whether to repeat steps 248 and 249 for other tags.

[0308] At step 251, the branch for modifying the label ends.

[0309] Steps 248-250 may be used to modify the intent label. The programming logic generation system 20 may be operable to make it impossible to modify the entity label in the branch through steps 248-250. However, the refinement annotation branches 243-246 may allow modification of the entity label in addition to enabling modification of the intent category.

[0310] As described above, and as shown in steps 243-246, an existing intent may be modified. This applies to both intent categories and entity labels.

[0311] At step 243, the tags assigned to the text may be changed, deleted, or retained. This may be performed automatically. Alternatively or additionally, the programming logic generation system 20 may be operable to cause a user interface to be controlled to enable a user to specify which and / or whose tags are to be changed, deleted, or retained.

[0312] At step 244, the programming logic generation system 20 can retrain the AI ​​model based on these changed intentions. The newly trained model can be tested. New test results (optionally including relative fuzziness metrics) can be output, for example, by controlling the user interface.

[0313] At step 245, it may be determined whether to repeat steps 243 and 244 for other annotations.

[0314] At step 246, the branch for modifying the label ends.

[0315] AI model training and / or testing

[0316] The AI ​​model (which may be an NLU model as explained above) may be trained using known techniques such as machine learning, deep learning, and natural language processing. The training may be performed automatically by the processing logic generation system 20. One or more of these techniques may be used in combination. In addition, the process may involve other components such as digital representations of words, sentences, and documents.

[0317] To train an AI (e.g., NLU) model, the entire annotated dataset can be collated and split into a training dataset and a test dataset. The training dataset can be used to train the NLU model. In particular, the training dataset can be trained to recognize intent from text based on annotations. Entities can optionally be recognized by training the AI ​​model accordingly. Annotations of entities can, but need not, be implemented based on semantic distance.

[0318] Training of intent recognition and (if implemented) entity recognition can be performed jointly or independently.

[0319] The training process may involve one or more preprocessing steps, such as converting the text to lowercase letters and generating tokens based on the text.

[0320] A model (which may also be referred to as an AI (e.g., NLU) model) may be generated through a training process and stored in a database (e.g., data storage device 23). A test data set is used to evaluate the performance of the trained AI (e.g., NLU) model by considering different performance metrics (e.g., precision and recall). The corresponding configuration is saved in a database (e.g., data storage device 23). The saved AI (e.g., NLU) model may be used to make predictions for any input text.

[0321] The data generated in the AI ​​model training and / or testing can be stored for subsequent use. For illustration, the model accuracy and / or the number of training data points can be determined and stored for subsequent use in determining the most appropriate AI model.

[0322] Data generated during AI model training and / or testing can be stored as historical and / or hierarchical configuration data.

[0323] Fig.19 The diagram illustrates the history and hierarchical configurations stored in the database. The values ​​used are for illustrative purposes only. "Conf0" refers to the root configuration with two labels (L1 and L2), four text items as data points, and a model accuracy of 0.30. Additional information, if any, may also be stored with the configuration.

[0324] “Conf 1” represents the configuration obtained after adding new data points t5 and t6, which improves the accuracy to 0.33. In “Conf 2”, a new label with two additional data points is created. “Conf 3” and “Conf 4” are derivatives of this particular configuration—in the former case, labels L2 and L3 have been merged together, and in the latter case, a new label L4 is created. Finally, “Conf 5” illustrates an instance of changing an existing text annotation from label L2 to label L4. The user can select any of these configurations as a parent configuration to generate the next configuration. In addition, when a configuration achieves the best performance in terms of the desired metric (e.g., accuracy and number of training data points), the processing logic generation system 20 can notify the user, for example using visual highlighting.

[0325] The processing logic generation system 20 can automatically evaluate multiple configurations to identify and recommend a relatively better configuration. Fig.19 , the processing logic generation system 20 may evaluate both "Conf 3" and "Conf 4" and recommend the latter because the latter provides higher accuracy. Thus, the system can help identify the best NLU model for textual system descriptions.

[0326] Annotation and training performance

[0327] The annotation and training cycle can continue ( Fig.11 ) until a satisfactory performance level is reached. The satisfaction criterion may be determined based on one or more criteria, such as the number of data points used for training and testing, contemporary values ​​of model precision and recall, and validation test loss. The processing logic generation system 20 may allow the user to specify whether an appropriate satisfaction has been achieved based on one or more desired metrics via a user interface controlled by the processing logic generation system 20. Alternatively or in addition, the processing logic generation system 20 may highlight when such satisfaction has been achieved or approached based on a predefined threshold.

[0328] To illustrate, in some cases, at least 10,000 training data points and a target accuracy of 90% may be desired. An appropriate tolerance boundary, such as 5%, may also be defined. Assume that 9,500 data points have been generated based on annotations of several textual system descriptions. The processing logic generation system 20 may determine and optionally confirm that one of the desired performance criteria has been achieved.

[0329] To further illustrate, at a later time, 9,700 training data points and a model accuracy of 87% may be obtained. The processing logic generation system 20 may automatically stop the annotation and training cycle at this time. Alternatively, the processing logic generation system 20 may control the user interface to request the user to specify whether to stop the process.

[0330] The processing logic generation system 20 may use the satisfaction level to automatically switch to an automated annotation process. For illustration, the determination of the initial data set may still require expert knowledge to annotate the initial set of data points with intents and optionally entities until the desired volume and performance level is achieved. The processing logic generation system 20 may use a predefined threshold of satisfaction level of the initial data set generation to identify when to stop the phase and move to the next phase. Alternatively, the processing logic generation system 20 may be operable to control a user interface to allow a user to select to initiate an automated phase.

[0331] The metrics used to determine satisfaction may constitute a partial ranking. For example, in the case of a binary classification between two categories, precision and recall may be prioritized over accuracy. In this case, even if the desired model accuracy threshold is reached, the processing logic generation system 20 may infer that the desired satisfaction has not been reached.

[0332] Use trained AI models as processing logic

[0333] The trained AI (e.g., NLU) model generated using the aforementioned procedure can be used to extract information from new textual system descriptions. This information can be used for different purposes, such as identification of suitable equipment and layout of communication systems or public service systems.

[0334] Specifications extracted from new textual system descriptions can be automatically matched against available products to identify the parts best suited to be installed in the system. Matching can be performed on a variety of attributes such as supported protocols, throughput, and certifications. Prior knowledge (from domain experts or from a knowledge base) and information from related domains can be used to identify suitable devices, device configurations, etc.

[0335] Illustrative examples

[0336] Different network switches (such as network switches that may be from different manufacturers or may have different device configurations, but not limited to this) may have similar features and specifications and be used for similar purposes. However, the former are often used in power grids, while the latter are often used in other industries, such as transportation. Domain knowledge or patterns from a knowledge base may be used to determine the most appropriate device. Similarly, if teleprotection is mentioned in the text system description, it indicates that one of the network switches may be more suitable than another.

[0337] Once relevant equipment, such as network switches, is identified, prior knowledge from an existing knowledge base can be used to automatically determine the appropriate network topology and generate the associated generic configuration files. For example, if the textual system description pertains to an inter-substation network and the number of stations is known, a ring topology can be generated. Alternatively, any topology specified in the textual system description, such as in text or graphical form, can be used in this regard.

[0338] Based on the topology, the appropriate profile can be automatically determined. A knowledge base can also be used in this process. For example, the amount and / or type of traffic to be injected into the network can be identified. Additionally or alternatively, appropriate VLANs and / or firewall rules can be created. The performance of such a system can be measured using simulation, emulation, testing, or any combination thereof. The measured performance can be compared to an appropriate service level agreement (SLA) taken from a textual system description or present in a knowledge base.

[0339] Fig. 20 is a flow chart of method 270. Method 270 may use the generated processing logic to debug and / or configure a technical system (eg, a power system or a communication system).

[0340] At step 271, a semantic distance-based clustering technique is used to generate training data for AI model training. The generation of training data may include labeling text system descriptions.

[0341] At step 272, the AI ​​model is trained. Fig. 20 Not shown, but steps 271 and 272 may be repeated iteratively.

[0342] At step 273, processing logic obtained by the trained AI model may be executed. The processing logic may be used to process the textual system description. The processing logic may be used to process the textual system description into an annotated textual system description.

[0343] At step 274 , the marked results may be used to debug and / or configure equipment of a technical system (such as a public service system or a communication system) based on the annotated textual system description.

[0344] Fig.21 is a flow chart of method 280. Method 280 may further include steps for generating a trained AI model, as described in conjunction with steps 271 and 272.

[0345] At step 281, processing logic obtained by the trained AI model may be executed. The processing logic may be used to process the textual system description. The processing logic may be used to process the textual system description into an annotated textual system description.

[0346] At step 282 , the marked results may be used to control the operation of at least a portion of the deployment and / or operation of devices of a technical system (such as a public service system or a communication system) based on the annotated textual system description.

[0347] Fig. 22is a schematic diagram of a system 300 in which an input device 40 and a command generation system 20 may be used to generate and issue commands to a controlled system, in the present case an electrical power system.

[0348] System 300 includes a distributed energy resource (DER) or a microgrid. Several generators 301-303 provide power. The DER or microgrid may also include several loads 316, 317 connected to the (multiple) generators 301-303 via a power transmission system and / or a distribution system 315. (Multiple) circuit breakers ((multiple) CBs), (multiple) protection relays and / or other power switches 312, 313, 319 may allow the generators 301-303 and / or loads 316, 317 to be selectively coupled. (Multiple) IEDs 311 are deployed to control the main components. (Multiple) IEDs may be communicatively coupled via a communication network 320. The communication network 320 may include communication devices, such as one or more firewalls 321 and / or one or more communication switches 323.

[0349] The processing logic generation system 20 may be operable to automatically tag a textual system description to generate processing logic. The processing logic generation may be performed using any of the techniques disclosed herein.

[0350] The processing logic execution system 110 may be operable to execute the generated processing logic.The processing logic execution system 110 may process the textual system description of the system 300 for debugging and / or configuring the system 300. This may include generating configuration data, such as a configuration file.

[0351] Alternatively or additionally, during field use of the system 300, the processing logic execution system 110 may process a textual system description of the system 300 for use in controlling devices of the public service system 300 (such as auxiliary devices 311, 314 or main devices) and / or for controlling devices of the communication system 320 (such as firewalls and / or network switches) during field use.

[0352] The systems, devices and methods disclosed herein achieve various effects. The systems, devices and methods help to automatically generate annotated text system descriptions from unannotated text system descriptions. These annotated text descriptions can be used to generate improved processing logic for processing text descriptions of technical systems.

[0353] The systems, devices and methods facilitate the automatic generation of large labeled datasets based on small initial datasets for training AI models. The systems, devices and methods allow for the modification of labels (and thereby the AI ​​model) in an iterative process by providing techniques for merging or splitting labels. This allows for improved accuracy. The systems, devices and methods facilitate the generation of optimized AI models by automatically considering different configurations / combinations of data points and / or labels.

[0354] The textual descriptions involved in the system, device and method may all relate to the same domain or a part of the same domain, so that interpretation and labeling errors caused by information transfer from one (technical) domain to another domain can be avoided or at least reduced.

[0355] For example, the systems, apparatus, and methods may be used in association with textual descriptions of public service systems (eg, power systems) and / or communication systems, but are not limited thereto.

[0356] In the following, further examples of the present invention are described:

[0357] 1. A method of generating processing logic operable to automatically process a textual system description relating to a system, in particular relating to a public service system, an industrial system, or a computing and / or communications network, the method comprising the following steps performed by a computing system:

[0358] performing automatic labeling on text descriptions included in a training batch of text descriptions to generate an annotated training batch, wherein performing the automatic labeling includes performing a clustering technique to annotate text items included in the training batch with a label from a set of labels based on semantic distances of the text items included in the training batch from text items previously assigned to a plurality of labels from the set of labels; and

[0359] The AI ​​model is trained using the annotated training batch to generate the processing logic.

[0360] 2. The method of Example 1, wherein the set of tags comprises intent categories, and optionally wherein the set of intent categories comprises intent categories related to different devices of the system.

[0361] 3. A method as described in Example 2, wherein the set of tags includes attributes of at least some of the intent categories, and optionally, wherein the attributes of the intent categories define entities related to the devices involved in the intent categories.

[0362] 4. The method as described in any one of the aforementioned Examples 1 to 3 further includes executing the AI ​​model after the training to process the textual system description into an annotated textual system description.

[0363] 5. The method as described in any of the aforementioned Examples 1 to 4 further includes iteratively repeating the steps of automatically labeling and training the AI ​​model until the performance criteria are met.

[0364] 6. The method as described in any of the aforementioned Examples 1 to 5 further includes modifying the set of labels or modifying the labels assigned in the automatic labeling, and retraining the AI ​​model.

[0365] 7. The method of example 6, wherein the method comprises modifying the set of tags, including one or more of the following:

[0366] Merge at least two tags in the set of tags into one tag,

[0367] Splitting a tag in the set of tags into at least two tags,

[0368] retaining a label from the set of labels,

[0369] Optionally, wherein modifying the set of tags is performed automatically and / or in response to user input received via a user interface.

[0370] 8. The method of any one of the preceding examples 1 to 7, wherein performing the automatic labeling comprises

[0371] Each text item included in the training batch is annotated with a corresponding label from the set of labels if the semantic distance to at least one cluster in the clustering technique satisfies a semantic distance threshold criterion.

[0372] 9. The method of example 8, wherein performing the automatic labeling comprises

[0373] If the semantic distance to all clusters in the clustering technique does not satisfy the semantic distance threshold criterion, the text items included in the training batch are left unannotated.

[0374] 10. The method as described in Example 8 or Example 9, wherein the semantic distance from the text item to the cluster may be an average semantic distance from the text item to all text items belonging to the corresponding cluster.

[0375] 11. The method according to any one of the preceding examples 1 to 10,

[0376] The AI ​​model has an input layer and an output layer, the input layer receives text items included in the text system description, the output layer outputs annotations to the text items included in the text system description, and / or

[0377] Wherein, the AI ​​model comprises a natural language processing (NLP) converter, and optionally, wherein the NLP converter comprises an encoder-decoder model with a self-attention layer,

[0378] Optionally, wherein the encoder of the NLP converter comprises a stack of encoder layers, each encoder layer comprising a self-attention layer and a feed-forward network, further optionally, wherein the feed-forward network is a fully connected 2-layer network, and / or

[0379] Optionally, the AI ​​model includes a conditional random field (CRF) layer that receives the output of the NLP converter.

[0380] 12. A method for automatically generating and / or evaluating a system configuration, in particular a system configuration of a public service system, an industrial system, or a computing and / or communication network, the method comprising the following steps performed by a computing system:

[0381] Executing the method of any one of the preceding examples 1 to 11 to generate processing logic operable to automatically process a text system description related to the system;

[0382] receiving said textual system description associated with said system;

[0383] Processing the textual system description using the AI ​​model to generate an annotated textual system description; and

[0384] An output is generated based on the annotated textual system description.

[0385] 13. The method of example 12, wherein generating an output comprises one or more of the following:

[0386] automatically configuring at least one device of the system based on the annotated textual system description;

[0387] Automatically generate configuration data for at least one device of the system based on the annotated text system description, and store the configuration data in the at least one device for field operation of the system, optionally wherein the configuration data comprises at least one substation configuration description SCD file, further optionally wherein the SCD file is an SCD file compliant with IEC 61850;

[0388] automatically selecting devices for deployment in the system based on the annotated textual system description;

[0389] A system topology of the system is automatically determined based on the annotated textual system description.

[0390] 14. A computing system adapted to generate processing logic operable to automatically process a textual system description relating to a system, in particular relating to a public service system, an industrial system, or a computing and / or communication network, the computing system comprising at least one circuit adapted to perform a method comprising the steps of:

[0391] performing automatic labeling on text descriptions included in a training batch of text descriptions to generate an annotated training batch, wherein performing the automatic labeling includes performing a clustering technique to annotate text items included in the training batch with a label from a set of labels based on semantic distances of the text items included in the training batch from text items previously assigned to a plurality of labels from the set of labels; and

[0392] The AI ​​model is trained using the annotated training batch to generate the processing logic.

[0393] 15. The computing system of example 14, wherein the computing system is adapted to perform the method of any one of examples 1 to 13.

[0394] 16. A method as described in any of Examples 1 to 11 or 12 to 13, wherein the text descriptions all relate to the same domain or are part of the same domain.

[0395] 17. The computing system of any of examples 14-15, wherein the text descriptions all relate to the same domain or are part of the same domain.

Claims

1. A method of generating processing logic operable to automatically process a textual system description associated with a system, wherein: The system is a public service system, an industrial system, or a computing and / or communication network, and the method comprises the following steps performed by the computing system: performing automatic labeling on text descriptions included in a training batch of text descriptions to generate an annotated training batch, wherein performing the automatic labeling includes performing a clustering technique to annotate text items included in the training batch with a label from a set of labels based on semantic distances of the text items included in the training batch from text items previously assigned to a plurality of labels from the set of labels; Using the annotated training batch to train an AI model to generate the processing logic; modify the set of labels or modify the labels assigned in the automatic labeling, Wherein, modifying the set of tags includes one or more of the following: Merge at least two tags in the set of tags into one tag, Splitting a tag in the set of tags into at least two tags, retaining a label from the set of labels, Optionally, wherein modifying the set of tags is performed automatically and / or in response to user input received via a user interface; and Retrain the AI ​​model.

2. The method of claim 1, further comprising: Iteratively repeat one or more of the following steps until the performance criteria are met: performing said automatic labeling, training the AI ​​model, Modify the set of labels or modify the labels assigned in the automatic labeling and retrain the AI ​​model.

3. The method according to claim 1 or claim 2, wherein: The set of tags comprises intent categories, optionally wherein the set of intent categories comprises intent categories relating to different devices of the system.

4. The method of claim 3, wherein: The set of tags includes attributes of at least some of the intent categories, optionally wherein the attributes of the intent categories define entities related to the devices involved in the intent categories.

5. The method of any of the preceding claims, further comprising, after the training, executing the AI ​​model to process the textual system description into an annotated textual system description.

6. A method as claimed in any one of the preceding claims, wherein: Executing the automatic labeling includes Each text item included in the training batch is annotated with a corresponding label from the set of labels if the semantic distance to at least one cluster in the clustering technique satisfies a semantic distance threshold criterion.

7. The method of claim 6, wherein: Executing the automatic labeling includes If the semantic distance to all clusters in the clustering technique does not satisfy the semantic distance threshold criterion, the text items included in the training batch are left unannotated.

8. The method of claim 6 or claim 7, wherein: The semantic distance between the text item and the cluster may be an average semantic distance between the text item and all text items belonging to the corresponding cluster.

9. A method as claimed in any one of the preceding claims, in, The AI ​​model has an input layer and an output layer, the input layer receiving text items included in the text system description, the output layer outputting annotations to the text items included in the text system description, and / or Wherein, the AI ​​model includes a natural language processing (NLP) converter, Optionally, wherein the NLP converter comprises an encoder-decoder model with a self-attention layer, Optionally, wherein the encoder of the NLP converter comprises a stack of encoder layers, each encoder layer comprising a self-attention layer and a feed-forward network, further optionally, wherein the feed-forward network is a fully connected 2-layer network, and / or Optionally, the AI ​​model includes a conditional random field (CRF) layer that receives the output of the NLP converter.

10. The method according to any one of the preceding claims 1 to 9, wherein: The text descriptions all relate to the same domain or are part of the same domain.

11. A method of generating processing logic operable to automatically process a textual system description relating to a system, in particular relating to a public service system, an industrial system, or a computing and / or communications network, the method comprising the following steps performed by a computing system: Automatic labeling is performed on the text descriptions included in the training batch of text descriptions to generate an annotated training batch, wherein: Performing the automatic labeling includes performing a clustering technique to annotate the text items included in the training batch with a label from a set of labels based on a semantic distance of the text items included in the training batch from text items previously assigned to a plurality of labels from the set of labels; and The AI ​​model is trained using the annotated training batch to generate the processing logic.

12. The method of claim 11, wherein: The text descriptions all relate to the same domain or are part of the same domain.

13. A method for automatically generating and / or evaluating a system configuration, in particular a system configuration of a public service system, an industrial system, or a computing and / or communication network, the method comprising the following steps performed by a computing system: performing a method as claimed in any one of claims 1 to 10 or 11 to 12 to generate processing logic in the form of a trained AI model, the processing logic being operable to automatically process a textual system description relating to the system; receiving said textual system description associated with said system; processing the textual system description using the trained AI model to generate an annotated textual system description; and An output is generated based on the annotated textual system description.

14. The method of claim 13, wherein: Build output includes one or more of the following: automatically configuring at least one device of the system based on the annotated textual system description; Automatically generate configuration data for at least one device of the system based on the annotated text system description, and store the configuration data in the at least one device for field operation of the system, optionally wherein the configuration data comprises at least one substation configuration description SCD file, further optionally wherein the SCD file is an SCD file compliant with IEC 61850; automatically selecting devices for deployment in the system based on the annotated textual system description; A system topology of the system is automatically determined based on the annotated textual system description.

15. A computing system adapted to generate processing logic operable to automatically process a textual system description associated with a system, wherein: The system is a public service system, an industrial system, or a computing and / or communication network, the computing system comprising at least one circuit adapted to perform a method comprising the following steps: performing automatic labeling on text descriptions included in a training batch of text descriptions to generate an annotated training batch, wherein performing the automatic labeling includes performing a clustering technique to annotate text items included in the training batch with a label from a set of labels based on semantic distances of the text items included in the training batch from text items previously assigned to a plurality of labels from the set of labels; Using the annotated training batch to train an AI model to generate the processing logic; modify the set of labels or modify the labels assigned in the automatic labeling, Wherein, modifying the set of tags includes one or more of the following: Merge at least two tags in the set of tags into one tag, Splitting a tag in the set of tags into at least two tags, retaining a label from the set of labels, Optionally, wherein modifying the set of tags is performed automatically and / or in response to user input received via a user interface; and Retrain the AI ​​model.

16. The computing system of claim 15, wherein: The at least one circuit is further adapted to: Iteratively repeat one or more of the following steps until the performance criteria are met: performing said automatic labeling, training the AI ​​model, Modify the set of labels or modify the labels assigned in the automatic labeling and retrain the AI ​​model.

17. A computing system as claimed in claim 15 or claim 16, wherein: The computing system is adapted to perform the method of any one of claims 1 to 14 .

18. A computing system adapted to generate processing logic operable to automatically process a textual system description relating to a system, in particular relating to a public service system, an industrial system, or a computing and / or communication network, the computing system comprising at least one circuit adapted to perform a method comprising the steps of: Automatic labeling is performed on the text descriptions included in the training batch of text descriptions to generate an annotated training batch, wherein: Performing the automatic labeling includes performing a clustering technique to annotate the text items included in the training batch with a label from a set of labels based on a semantic distance of the text items included in the training batch from text items previously assigned to a plurality of labels from the set of labels; and The AI ​​model is trained using the annotated training batch to generate the processing logic.

Images