Method for dataset validation in communication network

Abstract

A method (100) for dataset validation The method comprises: obtaining (S110) first features corresponding to a first dataset; determining (S120) a first metric, wherein the determining comprises creating (S121) a data schema for the first features using a querying system and evaluating (S122) the first features based on the data schema; determining (S130) a second metric, wherein the determining comprises obtaining (S131) second features corresponding to a second dataset and evaluating (S132) the first features based on the second features; determining (S140) a third metric, wherein the determining comprises obtaining (S141), from a knowledge database, first information which is based on the first features and evaluating (S142) the first features based on the first information; and sending (S150), to a first communication device, a fourth metric of the first dataset, wherein the fourth metric is an aggregation of the first, second, and third metric.

Description

[0001] METHOD FOR DATASET VALIDATION IN COMMUNICATION NETWORK

[0002] TECHNICAL FIELD

[0003] The disclosure relates to a computer-implemented method for dataset validation, performed by a network node in a communication network. The disclosure also relates to a system of network nodes, a computer program, and a computer program product.

[0004] BACKGROUND

[0005] There is a demand for further improvements to a Radio Access Network (RAN) accomplished by features such as traffic prediction, secondary cell selection, load balancing, scheduling, service orchestration and proactive service assurance. These features are in turn facilitated by modeling and predictive tools, namely data-driven models, trained via machine learning methods, using training samples from datasets that have been obtained from various components of a RAN.

[0006] Data-driven models, however, can only be effective when the corresponding datasets are of high quality, i.e., complete, correct, consistent, and thus accurately representing the operation of a real RAN. This assumption rarely holds in practice, as there may be low quality training samples due to, inter alia, unaccounted disabled network cells, data corruption during transits, and data corruption while in storage.

[0007] A RAN dataset has several characteristics that complicate the assessment of the quality of the dataset. First, a RAN dataset contains nested structures that are connected geographically (e.g., cells and their neighbors) and hierarchically (e.g., a base station and its cells). Secondly, a RAN dataset comprises data with complex data types having both static data such as Configuration Management (CM) data, and dynamic, possibly time series data such as Performance Management (PM) data.

[0008] Prior art does leverage these characteristics when assessing the quality of RAN datasets. The document S. Schelter et. al., "Automating large-scale data quality verification," in International Conference on Very Large Data Bases, 2018, discloses a system for automating the verification of data quality at scale. The system provides a declarative Application Programming Interface (API) which combines common quality constraints with user-defined validation code, and thereby enables "unit tests" for data.

[0009] The document E. Breck et. al., "Data Validation for Machine Learning," in Proceedings of Machine Learning and Systems, 2019, discloses a data validation system that is designed to detect anomalies in data fed into machine learning pipelines.

[0010] The systems disclosed by the afore-mentioned documents require an a priori and manual definition of domain rules (e.g., rules pertaining to RAN operation and data) since neither system comprises telecommunication information. For example, neither system can detect that a received PM counter of active users in a cell cannot be negative, unless a user has provisioned the system with this information as a domain rule a priori. Such a provision is neither automatic, nor scalable, considering the large number (2000+) of CM and PM counters in a real, typical RAN, as well as the fact that CM and PM counter definitions may be different in various RAN implementations.

[0011] Moreover, the afore-mentioned systems require exact knowledge of which CM data and / or PM data are relevant for a given application. This leads to a degradation of dataset validation in applications where the PM data and / or CM data evolve over time, for example, when predicting hourly RAN traffic per cell.

[0012] SUMMARY

[0013] An object of the invention is to enable a more efficient operation and / or improved performance of a communication device connected to a communication network.

[0014] According to a first aspect of the invention, there is provided a computer-implemented method for dataset validation, performed by a network node in a communication network. The method comprises obtaining a first set of features corresponding to a first dataset. The method further comprises determining a first metric, wherein the determining of the first metric comprises creating a data schema for the first set of features using a querying system and evaluating the first set of features based on the data schema. The method further comprises determining a second metric, wherein the determining of the second metric comprises obtaining a second set of features corresponding to a second dataset and evaluating the first set of features based on the second set of features. The method further comprises determining a third metric, wherein the determining of the third metric comprises obtaining, from at least one knowledge database, first information which is based on the first set of features and evaluating the first set of features based on the first information. The method further comprises sending, to a first communication device, a fourth metric of the first dataset, wherein the fourth metric is an aggregation of the first metric, second metric, and third metric. Hereby is achieved the automated provision of a communication device with a metric indicative of the quality of a dataset, wherein the quality represents, for a datum in the dataset, the level of correctness of its data type, its similarity to a datum of a reference dataset, and its compatibility with information from a knowledge database. Furthermore, the metric is more indicative of the quality of a dataset over the first, second, or third metric taken either independently or as a combination.

[0015] According to an embodiment of the first aspect, obtaining the first set of features comprises the receiving a first dataset and at least one of: a second dataset and a third dataset. Obtaining the first set of features further comprises using the querying system, a candidate set of features based on the first dataset, second dataset, and / or third dataset. Obtaining the first set of features further comprises obtaining, from the knowledge database, second information which is based on the candidate set of features. Obtaining the first set of features further comprises evaluating the candidate set of features based on the second information. Obtaining the first set of features further comprises, based on the evaluation, accepting the candidate set of features, or repeating the generating and evaluating of a candidate set of features. Obtaining the first set of features further comprises, selecting the first set of features corresponding to the first dataset, based on the accepted set of features. Hereby is achieved the obtaining of a first set of features that is better suited for dataset validation, by referencing multiple datasets and iterating repeatedly over information in a knowledge base. According to an embodiment of the first aspect, the querying system comprises, or is communicatively coupled via the network node to, a generative model. The generative model is in one embodiment a Large Language Model, and may have been trained or tuned, based on the third dataset. Hereby is achieved a more scalable querying system that can be augmented with new information in a computationally tractable manner.

[0016] According to an embodiment of the first aspect, using the querying system comprises obtaining third information from the knowledge database, wherein the third information corresponds to an input. Using the querying system further comprises providing a prompt to the generative model, wherein the prompt is based on the input and the third information. Using the querying system further comprises obtaining an output from the generative model. In one embodiment, the querying system comprises, or is communicatively coupled to, a Retrieval Augmented Generation system. Hereby is achieved an adaptive querying system that can utilize additional information from a knowledge database as well as information provided dynamically from a user.

[0017] At least one or more of the embodiments, advantageously enable an improved dataset validation technique by means of an improved or augmented querying system, the querying system being better adapted to the type of dataset which is to be validated.

[0018] According to an embodiment of the first aspect, each dataset is a telecommunication dataset.

[0019] According to an embodiment of the first aspect, each set of features comprises Configuration Management, CM, data and / or Performance Management, PM, data.

[0020] At least one or more of the embodiments, advantageously enable the provision of a communication device with a metric indicative of the quality of a RAN dataset.

[0021] According to an embodiment of the first aspect, the method further comprises sending a notification to the first communication device or a second communication device if no information is obtained from the knowledge database. The method further comprises receiving data from the first communication device or second communication device.

[0022] The method further comprises updating the knowledge database based on the received data. Hereby is achieved the automatic and iterative updating of a knowledge database with information relevant for dataset validation.

[0023] According to a second aspect of the invention, there is provided a system of network nodes adapted for dataset validation comprising processing circuitry configured to perform the method according to the first aspect.

[0024] According to a third aspect of the invention, there is provided a computer program comprising machine readable instructions which, on execution by a processing circuitry of a system, cause the system to perform the method according to the first aspect.

[0025] According to a fourth aspect of the invention, there is provided a computer program product comprising a non-transient storage medium, on which is stored a computer program in accordance with the third aspect.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above, as well as additional objects, features, and advantages of the invention, will be better understood through the following illustrative and non-limiting detailed description of embodiments of the invention, with reference to the appended drawings, in which:

[0028] Figure 1 is a flow chart illustrating embodiments of a method for dataset validation;

[0029] Figure 2 is a flow chart illustrating embodiments of a method for dataset validation;

[0030] Figure 3 is a flow chart illustrating embodiments of a method for dataset validation;

[0031] Figure 4 is a flow chart illustrating embodiments of method steps for dataset validation; Figure 5 is a block diagram illustrating a network node in a communication network;

[0032] Figure 6 is a block diagram illustrating a system of network nodes in a communication network;

[0033] Figure 7 is a block diagram illustrating a computer program product.

[0034] All the figures are schematic, and generally only show parts which are necessary to elucidate the invention, whereas other parts may be omitted or merely suggested.

[0035] DETAILED DESCRIPTION

[0036] In Figure 1 is provided a computer-implemented method 100 for dataset validation performed by a network node 500 (see Fig. 5) in a communication network 510 (see Fig 5). Dataset validation is a process that determines the quality of a dataset, defined herein as the extent to which a dataset comprises data of the correct type, having no missing or inaccurate values, and satisfying one or more quantifiable criteria.

[0037] The method 100 comprises, in a first step S110, obtaining a first set of features corresponding to a first dataset. A dataset comprises data, data points, or data elements.

[0038] In a second step S120, the method 100 comprises determining a first metric, wherein the determining of the first metric comprises a step S121 of obtaining a data schema for the first set of features using a querying system 520 (see Fig. 5). A data schema is a dataset containing information regarding the data type of each feature, wherein the data type is admissible or expected by the network node 500. The determining of the first metric further comprises a step S122 of evaluating the first set of features based on the data schema.

[0039] In a third step S130, the method 100 comprises determining a second metric, wherein the determining of the second metric comprises a step S131 of obtaining a second set of features corresponding to a second dataset. The determining of the second metric further comprises a step S132 of evaluating the first set of features based on the second set of features.

[0040] In a fourth step S140, the method 100 comprises determining a third metric, wherein the determining of the third metric comprises a step S141 of obtaining, from a knowledge database 530 (see Fig. 5), first information which is based on the first set of features. A knowledge database is a repository which may receive, store, modify, and provide information associated with knowledge representation(s) of datasets and / or features of datasets. The determining of the third metric further comprises a step S142 of evaluating the first set of features based on the first information.

[0041] In a fifth step S150, the method 100 comprises sending, to a first communication device 540 (see Fig. 5), a fourth metric of the first dataset, wherein the fourth metric is an aggregation of the first metric, second metric, and third metric.

[0042] Therefore, the method 100 enables the automated provision of a communication device with a metric indicative of the quality of a dataset, wherein the quality represents, for a datum in the dataset, the level of correctness of its data type, its similarity to a datum of a reference dataset, and its compatibility with information from a knowledge database. Furthermore, the metric is more indicative of the quality of a dataset over the first, second, or third metric taken either independently or as a combination.

[0043] Using Figure 1 also for a description of a first embodiment, the method is performed by the network node 500 in the communication network 510, which in this embodiment, is a telecommunication network. The network node receives a RAN dataset comprising Configuration Management, CM, data and / or Performance Management, PM, data associated with a set of RAN network elements, e.g., cells and base stations in the telecommunication network. The PM data and CM data optionally relate to a specific time, e.g., a specific hour, day, week, or month during which the RAN network elements were operable. In the first step S110, the network node obtains, by deriving, a first set of features from the RAN dataset, and the first set of features is a subset of the PM data and / or CM data. The first set of features is pre-processed into a three-dimensional tensor of real values ranging from 0 to 1, wherein the first axis identifies a PM datum and / or a CM datum, the second axis identifies a RAN network element associated with the datum, and the third axis identifies a time during which the datum has been sampled. Each element of the tensor broadly indicates the quality of its corresponding datum, with a value of 0 indicating the lowest quality and a value of 1 indicating the highest quality.

[0044] In the second step S120, the network node determines a first metric, wherein the first metric is a three-dimensional tensor as described above, each element of the tensor indicating whether the datum corresponding to the element, has a correct data type. In the step S121, the network node creates a data schema by using the querying system 520, wherein the network node provides the first set of features and a prompt to the querying system and wherein the querying system is a generative model. A generative model is a machine learning model that may generate data having the same probability distribution as the training data which has been used to train or fine-tune the model. The generative model is fine-tuned on a dataset comprising information on PM data and / or CM data, such as product documentation, instruction manuals, expert reports, etc. The prompt is a natural language text indicating a request for a data schema for the first set of features. The generative model generates a data schema, wherein the data schema is a list comprising a range of possible PM data and / or CM data and a corresponding set of admissible / expected data types for each possible datum.

[0045] In the step S122, the network node receives the data schema from the generative model and evaluates the first set of features based on the data schema. The evaluating comprises a comparison between the data type of each PM datum and / or CM datum of the first set of features, and the corresponding entry for that datum on the data schema. When the data type of the PM and / or CM datum of the first set of features is listed on the corresponding entry of the data schema, the network node assigns the value of 1 to the element of the first metric that corresponds to the PM and / or CM datum and assigns the value of 0 otherwise. In the third step S130, the network node determines a second metric, wherein the second metric is a three-dimensional tensor as described above, each element of the tensor indicating the similarity of the datum corresponding to the element, to a feature of a second dataset. The second dataset is a RAN dataset which serves as a reference for the first dataset and is assumed to be of high quality. The quality of the second dataset may have been verified a priori by a domain, or subject-matter, expert.

[0046] In the step S131, the network node obtains, by deriving, the second set of features from the second dataset, wherein the deriving comprises comparing the first set of features with a candidate set of features of the second dataset and selecting a candidate set of features as the second set of features. The candidate set of features is a subset of the second dataset, comprising PM data and / or CM data. The comparing comprises determining a characteristic difference between a first RAN network element associated with a feature from the first set of features and a second RAN network element associated with a feature from the second set of features. A characteristic similarity may be a geographic distance, a difference in network size, a difference in RAN technology type, e.g.. Universal Terrestrial Radio Access Network, Long-Term Evolution, New Radio, and any future 3rdGeneration Partnership Project RAN technology such as the future 6G RAN. A characteristic similarity may also be a difference in frequency spectra, a difference in software versions, etc. The selecting comprises determining that the candidate set of features and the first set of features, have a minimal corresponding characteristic difference.

[0047] In the step S132, the network node evaluates the first set of features based on the second set of features. The evaluating comprises a comparison between the first set of features and the second set of features. The comparison comprises determining a statistical similarity measure between a first distribution of PM and / or CM data over a time interval, derived from the first set of features, and a second distribution of PM data and / or CM data over a time interval, derived from the second set of features. The statistical similarity measure may be a Kullback-Leibler, a Bregman divergence, or a difference between the means, standard deviations, or correlations of the first distribution and the second distribution. The network node may normalize the similarity measure to have a value ranging from 0 to 1. The network node assigns the value of the similarity measure to the value of every element of the second metric that corresponds to a PM and / or CM datum associated with the distribution derived from the first set of features.

[0048] In the fourth step S140, the network node determines a third metric, wherein the third metric is a three-dimensional tensor as described above, each element of the tensor indicating the compatibility of the datum corresponding to the element, to information from the knowledge database 530. In step S141, the network node provides the first set of features to the knowledge database and in turn obtained, from the knowledge database, first information comprising a rule associated with a feature from the first set of features. A rule may be, for example, that a PM and / or CM datum of a feature must be an integer with a value ranging from 0 to 1000, that the datum is always inadmissible, or that the datum must be an integer with a value ranging from 0 to 100 during a pre-determined time instance, and 0 to 80 otherwise.

[0049] In the step S142, the network node evaluates the first set of features based on the first information. The evaluating comprises verifying that each feature is in accordance with the obtained rule corresponding to the feature. When the feature is in accordance with the obtained rule, the network node assigns the value of 1 to the value of every element of the third metric that corresponds to a PM and / or CM datum associated with the feature and assigns the value of 0 otherwise.

[0050] In the step S150, the network node sends a fourth metric to a first communication device, wherein the first communication device is a RAN network element. The fourth metric is a three-dimensional tensor as described above and is derived by the network node as the weighted average of the first, second, and third metric, wherein the average is evaluated element-wise. The fourth metric is comprised in, for example, a Hypertext Transfer Protocol (HTTP) message, a HTTP Secure (HTTPS) message, a Message Queuing Telemetry T ransport (MQTT) message, a WebSocket message, and a Bluetooth message. In a second embodiment, which is an alternate embodiment of the first embodiment, but for which Fig. 1 also is used to illustrate the embodiment, the first dataset is a comma-separated values (CSV) file comprising numerical data. In step S110, the network node 500 obtains, by receiving, the first set of features from another network node in the communication network 510, wherein the first set of features are a subset of the first dataset and may be based on a feature extraction technique performed on the data of the CSV file, such as Principal Component Analysis.

[0051] In the step S121, the network node provides the first set of features to a querying system 520 and an indication of a data schema request. The querying system 520 comprises algorithms capable of performing searching, retrieving, counting, and aggregating operations on entries of a dataset. The querying system 520, upon receiving the indication, performs a search operation on a database, which may be a knowledge database 530, comprising information on an admissible data type for one or more entries of the first dataset (see also Fig. 5). The querying system retrieves and aggregates the information before providing it to the querying system, which in turn generates the data schema, wherein the data schema is a CSV file comprising all admissible / expected data types for each feature of the first set of features. The network node then obtains the data schema from the querying system.

[0052] In the step S122, the network node evaluates the first set of features based on the data schema, the evaluating comprises a comparison between the data type of every feature in the first set of features and all corresponding entries in the data schema. The network node determines the first metric as the ratio of the number of features in the first set of features having a data type that is in accordance with the data schema, and the total number of features in the first set of features.

[0053] In the step S131, the network node receives the second set of features from another network node in the communication network, wherein the second set of features are a subset of the second subset and may have also been derived based on a feature extraction technique, and wherein there is a correspondence, i.e., a bijective mapping. between the first set of features and the second set of features, such that they have the same data type.

[0054] In the step S132, the network node evaluates the first set of features based on the second set of features, the evaluating comprising the computation of either the Euclidian or Levenshtein distance between a feature from the first set of features and its corresponding feature from the second set of features, in accordance with the data type of the features. The network node then determines the second metric as the average over all computed distances, normalized down to a value between 0 and 1.

[0055] In the step S141, the network node provides the first set of features to the knowledge database and obtains, from the knowledge database, first information which is based on the first set of features. Here, the obtaining comprises providing to the querying system, the first set of features and an indication for an expert validation request. The querying system then performs a search operation on the knowledge database, retrieves and aggregates all entries related to the first set of features into the first information, and provides the first information to the network node.

[0056] In the step S142, the network node evaluates the first set of features based on the first information as in the first embodiment. The network node determines the third metric as the ratio of the number of features that are in accordance with the first information and the total number of features in the first set of features.

[0057] In the step S150, the network node sends a fourth metric to a first communication device 540, wherein the first communication device is another network node in the communication network. The fourth metric is the minimum of the first, second, and third metric.

[0058] In other embodiments, a dataset may comprise data which may be Boolean, numerical, or string values, binary files, structured data such as lists, arrays, and tables, e.g., relational databases, spreadsheets, JavaScript Object Notation (JSON) objects, extensible Markup language (XML) files or CSV files. Data may also be unstructured data such as natural-language text, documents with mixed content types, graph data comprising network nodes and edges for representing relationships, geospatial data such as coordinates, map, and spatial relationships, and multimedia data such as images, videos, audio clips, and animations, etc.

[0059] A dataset may be a telecommunication dataset, such as a RAN dataset. The RAN dataset may relate to Minimization of Drive Test, MDT, comprising data such as Uplink, UL, volume, UL Last T ransmission Time Interval, LastTTI, volume. Downlink, DL, volume, DL LastTTI volume, etc. In an example, RAN traffic data are vendor-specific cell traces, e.g., a Call Trace Record, CTR, a Cell Trace, Per-Call Measurement Data, PCMD, or Performance Call History Record, PCHR. The RAN dataset may also comprise Fault Management, FM, data.

[0060] The telecommunication dataset may alternatively be a core network dataset, i.e., it may comprise data related to network functions such as, in the case of a 5G core network, an Access and Mobility Management Function (AMF), a Session Management Function (SMF), a User Plane Function (UPF), a Policy Control Function (PCF) or interfaces and protocols such as an N1 or N2 interface, a Network Exposure Function (NEF) or a Session Initiation Protocol (SIP). A core network dataset may further, or alternatively, comprise subscriber data. Quality of Service (QoS) parameters, mobility management information, or session management information data. The telecommunication dataset may also be or include Non-access Stratum (NAS) data, and / or Access Stratum (AS) data in any combination with the above mentioned examples of core network dataset and RAN dataset.

[0061] The evaluating of the first set of features based on the data schema may be a comparison between the data type of one or more features from the first set of features and the corresponding entries of the data schema, wherein the comparison is based on every feature, or on a subset of features chosen randomly or deterministically. The comparison yields the first metric, which may be a Boolean, a numeric value, or a string, and which may be a measure of the quality of the first dataset with respect to data type. The first metric may be indicative of the degree to which the features of the first dataset comply with the data schema.

[0062] The second dataset is treated as a reference for the first dataset and is taken to have a quality exceeding a pre-defined threshold. The second set of features may be obtained in one of the ways herein described for the obtaining of the first set of features from the first dataset.

[0063] The evaluating of the first set of features based on the second set of features may be the determining that a similarity measure between a feature from a first set of features and a feature from the second set of features, does not exceed a pre-determined threshold. The similarity measure is a numeric value corresponding to the data type of the features being compared and may be a distance metric such as the Euclidean or Chebyshev distance, a string similarity such as the Levenshtein or Hamming distance, a statistical similarity such as the Mahalanobis or Hellinger distance, or the Kullback- Leibler or Bregman divergence, or a statistic such as the Jaccard index or Dice-Sorensen coefficient, etc. The second metric may be the similarity measure, a Boolean whose values correspond to whether the similarity measure exceeds a threshold, a numeric value derived from the similarity measure, or a string indicative of the similarity measure.

[0064] The knowledge representations may be logical statements or inference rules, ontologies, and knowledge graphs. A knowledge database may share an interface with a querying system or with a user. The first information relates to the first set of features and may be provided to the knowledge database by the querying system or the user, in response to the first set of features, either in the form of data that is then converted to a knowledge representation, or as a knowledge representation directly. Alternatively, the first information may have been present in the knowledge database a priori and, upon receiving the first set of features from the querying system or the user, the knowledge base associates the first information with the first set of features. The evaluating of the first set of features based on the first information yields the third metric which may be a Boolean, a numeric value, or a string, and is indicative of the extent to which a feature from the first set of features is in accordance with the first information. For example, when the feature comprises data, the first information may indicate a property that the data should possess, such as being of a certain data type, having a value within a specific range of values, having prescribed statistical properties, e.g., a mean or standard deviation within a prescribed range, etc.

[0065] The aggregation of the first metric, second metric, and third metric may be a weighted average, a maximum or minimum value, or the product of the first metric, second metric, and third metric.

[0066] In other embodiments, the querying system comprises, or is communicatively coupled via the network node to, a generative model. A generative model may be a Gaussian Mixture Model (GMM), a Hidden Markov Model (HMM), a Bayesian Network, a Generative Adversarial Network (GAN), a Variational Autoencoder (VAE), or a Large Language Model (LLM). Generative models may be used for tasks such as data augmentation, image, and text generation, as well as speech synthesis.

[0067] In some embodiments, the generative model has been trained or tuned based on a third dataset. The training or tuning may comprise training or tuning the model on a first subset of the third dataset and evaluating the performance of the model on a second subset of the third dataset.

[0068] In Figure 2 are provided optional steps S210-S260 for the method 100 according to a further embodiment of any of the above embodiments.

[0069] In the step S210, the network node 500 receives a first dataset and at least one of: a second dataset and a third dataset. The second and third datasets relate to the first dataset and may comprise data of the same type, data sharing a data source, or data indicative of the operation of a common network element. In the step S220, the network node generates, using the querying system 520, a candidate set of features based on the first dataset, second dataset, and / or third dataset. The candidate set of features may be generated, for example, via randomized sampling on the first dataset, second dataset, and / or third dataset, or generated from the first dataset, second dataset, and / or third dataset in accordance with rules that are pre-programmed into the querying system or obtained by the querying system from the knowledge base. The candidate set of features may alternatively be generated or derived via any other feature extraction technique known to those skilled in the art.

[0070] In the step S230, the network node obtains, from the knowledge database 530, second information which is based on the candidate set of features. The second information may be of the same type and obtained analogously to the first information of the step S141.

[0071] In the step S240, the network node evaluates the candidate set of features based on the second information. The evaluating may be carried out similarly to the evaluating of the first set of features based on the first information of step S142, wherein is obtained a metric indicative of the accordance of the candidate set of features with the second information.

[0072] In the step S250, based on the evaluation, the network node accepts the candidate set of features, or repeats the generating and evaluating of a candidate set of features. The network node is understood to accept the candidate set of features by, for example, storing them in a long-term memory, setting a flag indicative of the features being accepted, or setting a flag indicative that a candidate set features is not to be generated. In an embodiment, the network node accepts the candidate set of features when the metric indicative of the accordance of the candidate set of features with the second information, exceeds a pre-determined threshold.

[0073] In the step S260, the network node selects the first set of features corresponding to the first dataset, based on the accepted set of features. In an embodiment, all accepted features are the selected set of features. In another embodiment, the selected set of features are a randomized subset of the accepted set of features.

[0074] In Figure 3 are provided optional steps S310-S330 for the method 100 according to a further embodiment of any of the above embodiments.

[0075] In the step S310, the network node 500 obtains third information from the knowledge database 530, wherein the third information corresponds to an input. The third information may be of the same type and obtained analogously to the first information of the step S141, or the second information of the step S230. The input may be any of the inputs to the querying system 520 herein described. The input may, for example, be one or more of the first, second, or third dataset, one or more of a set of features from the first, second, or third dataset, or any combination thereof.

[0076] In the step S320, the network node provides a prompt to the generative model, wherein the prompt is based on the input and the third information. The prompt may be the input and / or the third information or may be natural language text indicative of the input and the third information.

[0077] In the step S330, the network node obtains an output from the generative model. The output may be any of the outputs of the querying system described herein. The output may, for example, be, a dataset, a set of features from a dataset, a data schema, a metric, or a natural language text.

[0078] In a further embodiment, the querying system 520 comprises, or is communicatively coupled via the network node 500 to, a Retrieval Augmented Generation, RAG, system.

[0079] In Figure 4 are provided optional steps S410-S430 of the method 100 according to a further embodiment of any of the above embodiments.

[0080] In the step S410, the network node 500 sends a notification to a communication device if no information is obtained from the knowledge database 530. The communication device may be the first communication device 540, or a second communication device 550 (see Fig. 5), wherein the second communication device is another network node in the communication network or is a communication device connected to the network. The notification may be an error message or a natural text indicating that no information could be obtained. The notification may further comprise a request for further information.

[0081] In the step S420, the network node receives data from the first communication device or the second communication device, wherein the data comprises information that serves as a substitute for the information that could not be obtained from the knowledge database.

[0082] In the step S430, the network node updates the knowledge database based on the received data. The updating may comprise providing the received data or a subset thereof to the knowledge database or may comprise first providing the data as an input to the generative model and then providing the output of the generative model to the knowledge database.

[0083] In Figure 5 is provided the network node 500 in the communication network 510 according to any of the above embodiments. The network node 500 is a physical device that is part of the communication network 510 and is communicatively coupled to the querying system 520 and the knowledge database 530. One or both of the querying system and the knowledge database may be comprised in the network node 500 or any other network node in the communication network, or a communication device connected to the communication network. The network node sends the fourth metric and optionally a notification to the first communication device 540, and optionally sends a notification to the second communication device 550. A communication device is a physical device separate from the communication network and operable to connect to the communication network and can, for example, be a terminal or a user equipment (UE) device. Optionally, the network node further receives data from the first communication device and / or the second communication device. Those skilled in the art can easily conceive of embodiments wherein the second communication device is instead a network node in the communication network.

[0084] The method 100 may be performed by a system of network nodes in a communication network 500, and the present disclosure provides a system of network nodes that is adapted to perform any, or all, of the steps of the above discussed method. The network node may be a computing device, a server etc.

[0085] In some embodiments, the steps S120, S130, S140 may be performed by virtual node hosted by the network node 500. A virtual node may comprise any logical entity, such as a Virtualized Network Function (VNF) which may itself be running in a cloud, edge cloud or fog deployment. The virtual node may be operable to be instantiated in a cloud-based deployment, as an rApp, in a physical or virtual radio access node, control network node, etc.

[0086] In Figure 6 is illustrated a system 600 of network nodes, wherein the network node 500 may implement the method 100, as illustrated in Figures 1 to 4, according to examples of the disclosure, for example on receipt of suitable instructions from a computer program 700 (see Fig 7.).

[0087] The system 600 of network nodes may comprise at least one network node 500. The network node 500 comprises processing circuitry 601. The network node 500 may further comprise a memory 602. The processing circuitry 601 is operable to perform some, or all, of the steps of the method 100 as discussed above with reference to Figures 1 to 4. The memory 602 contains instructions executable by the processing circuitry 601 such that the network node 500 is operable to perform some, or all, of the steps of the method 100, as illustrated in Figures 1 to 4. The instructions may also include instructions for executing one or more telecommunication protocols and / or data communication protocols. The instructions may be stored in the form of the computer program 700. In some examples, the processor or processing circuitry 601 may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, etc. The processor or processing circuitry 601 may be implemented by any type of integrated circuit, such as an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) etc. The memory X may include one or several types of memory suitable for the processor, such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, solid state disk, hard disk drive, etc.

[0088] In Figure 7 is illustrated a computer program product 710 comprising a computer readable storage medium 720. The computer readable storage medium 720 comprises instructions (e.g., computer program 700) which are executable by processing circuitry (such as the processing circuitry 601 of the network node 500 described herein) to cause the system 600 or the network node 500 to perform the method described herein in respect of the system 600. In one embodiment, the computer program product 710 is the same as the memory 602.

[0089] There is also provided a computer program 700 comprising instructions which, when executed by processing circuitry (such as the processing circuitry 601 of the network node 500 described herein), cause the processing circuitry to perform at least part of the method described herein.

[0090] The invention is herein described more fully with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. For example, even if the system 600 and the network node 500 have been disclosed as comprising the computer program 700, another embodiment of the system and the network node may be completely hardwarebased without involvement of a specific computer program.

[0091] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus. component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step.

[0092] Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features, and advantages of the enclosed embodiments will be apparent from the description.

Claims

CLAIMS1. A computer-implemented method (100) for dataset validation, performed by a network node (500) in a communication network (510), the method comprising: obtaining (S110) a first set of features corresponding to a first dataset; determining (S120) a first metric, wherein the determining of the first metric comprises creating (S121) a data schema for the first set of features using a querying system (520) and evaluating (S122) the first set of features based on the data schema; determining (S130) a second metric, wherein the determining of the second metric comprises obtaining (S131) a second set of features corresponding to a second dataset and evaluating (S132) the first set of features based on the second set of features; determining (S140) a third metric, wherein the determining of the third metric comprises obtaining (S141), from a knowledge database (530), first information which is based on the first set of features and evaluating (S142) the first set of features based on the first information; and sending (S150), to a first communication device (540), a fourth metric of the first dataset, wherein the fourth metric is an aggregation of the first metric, second metric, and third metric.

2. The computer-implemented method (100) according to claim 1, wherein obtaining the first set of features comprises: receiving (S210) a first dataset and at least one of: a second dataset, a third dataset; generating (S220), using the querying system (520), a candidate set of features based on the first dataset second dataset, and / or third dataset; obtaining (S230), from the knowledge database (530), second information which is based on the candidate set of features; evaluating (S240) the candidate set of features based on the second information; based on the evaluation, accepting (S250) the candidate set of features, or repeating the generating (S220) and evaluating (S240) of a candidate set of features; andselecting (S260) the first set of features corresponding to the first dataset, based on the accepted set of features.

3. The computer-implemented method (100) according to any one of the preceding claims, wherein the querying system (520) comprises, or is communicatively coupled via the network node (500) to, a generative model.

4. The computer-implemented method (100) according to claim 3, wherein the generative model is a Large Language Model.

5. The computer-implemented method (100) according to any one of claims 3 to 4, wherein the generative model has been trained or tuned, based on the third dataset.

6. The computer-implemented method (100) according to any one of claims 3 to 5, wherein using the querying system (520) comprises: obtaining (S310) third information from the knowledge database (530), wherein the third information corresponds to an inputproviding (S320) a prompt to the generative model, wherein the prompt is based on the input and the third information; and obtaining (S330) an output from the generative model.

7. The computer-implemented method (100) according to any one of claims 3 to 5, wherein the querying system (520) comprises, or is communicatively coupled via the network node to, a Retrieval Augmented Generation system.

8. The computer-implemented method (100) according to any one of the preceding claims, wherein each dataset is a telecommunication dataset.

9. The computer-implemented method (100) according to claim 8, wherein each set of features, comprisesConfiguration Management data and / or Performance Management data.

10. The computer-implemented method (100) according to any one of the preceding claims, comprising: sending (S410) a notification to the first communication device (540) or a second communication device (550) if no information is obtained from the knowledge database (530); receiving (S420) data from the first communication device (540) or a second communication device (550); and updating (S430) the knowledge database (530) based on the received data.

11. A system of network nodes (600) adapted for dataset validation comprising processing circuitry (601) and at least one memory (602), the memory containing instructions (603) executable by the processing circuitry, the system (600) configured to: obtain (S110) a first set of features corresponding to a first dataset; determine (S120) a first metric, wherein the determining of the first metric comprises creating (S121) a data schema for the first set of features using a querying system (520), and evaluating (S122) the first set of features based on the data schema; determine (S130) a second metric, wherein the determining of the second metric comprises obtaining (S131) a second set of features corresponding to a second dataset, and evaluating (S132) the first set of features based on the second set of features; determine (S140) a third metric, wherein the determining of the third metric comprises obtaining, from a knowledge database (530), first information (S141) which is based on the first set of features and evaluating (S142) the first set of features based on the first information; send (S150), to a first communication device (540), a fourth metric of the first dataset, wherein the fourth metric is an aggregation of the first metric, second metric, and third metric.

12. The system (600) according to claim 11, configured to obtain the first set of features, wherein the system(600) : receives (S210) a first dataset and at least one of: a second dataset and a third dataset;generates (S220), using the querying system (520), a candidate set of features based on the first, second, and / or third dataset; obtains (S230), from the knowledge database (530), second information which is based on the candidate set of features; evaluates (S240) the candidate set of features based on the second information; based on the evaluation, accepts (S250) the candidate set of features, or repeats the generating (S220) and evaluating (S240) of a candidate set of features; and selects (S260) the first set of features corresponding to the first dataset, based on the accepted set of features.

13. The system (600) according to any one of claims 11 to 12, wherein the querying system (520) comprises a generative model.

14. The system (600) according to claim 12, wherein the generative model is a Large Language Model, LLM.

15. The system (600) according to any one of claims 13 to 14, wherein the generative model is trained or tuned, based on the third dataset.

16. The system (600) according to any one of claims 13 to 15, configured to use the querying system (520), wherein the system (600): obtains (S310) third information from the knowledge database (530), wherein the third information corresponds to an input; provides (S320) a prompt to the generative model, wherein the prompt is based on the input and the third information; and obtains (S330) an output from the generative model;17. The system (600) according to any one of claims 13 to 15, wherein the querying system (520) comprises a Retrieval Augmented Generation, RAG, system.

18. The system (600) according to any one of claims 11 to 17, wherein each dataset is a telecommunication dataset.

19. The system (600) according to claim 18, wherein each set of features, comprises Configuration Management, CM, data and / or Performance Management, PM, data.

20. The system (600) according to any one of claims 11 to 19, wherein the system (600) is configured to: send (S410) a notification to the first communication device (540) or a second communication device (550), if no information is obtained from the knowledge database (530); receive (S420) data from the first communication device (540) or a second communication device (550); and update (S430) the knowledge database (530) based on the received data.

21. The system (600) of any one of claims 11 to 20, wherein the system (600) comprises only one network node (500).

22. A computer program (700) comprising instructions which, on execution by a processing circuitry (601) of a network node (500) of a system (600), or the system (600), cause the network node (500) or the system (600) to perform a method (100) according to any one of claims 1 to 10.

23. A computer program product (710) comprising a computer readable storage medium (720), on which a computer program (700) according to claim 22 is stored.

Images