Method, apparatus and system for configurable data acquisition for networked data analysis and management

Abstract

A networked data analysis management (DAM) system is provided that includes a data representation manager (DRM), a data acquisition and preprocessor (DCP), a relevance manager (CM), and a data source discoverer (DSD). The DRM obtains a representation of the data source for representing characteristics of the data source. And the DCP interacts with the data source so as to acquire data through a data acquisition action. The CM determines a correlation between different available data based on the representation. The DSD interacts with a data consumer and the CM to determine a set of data acquisition actions to perform and configure the DCP.

Description

Technical Field

[0001] The present invention generally relates to collecting and analyzing data using established data networks, and more particularly to supporting configurable data collection and analysis operations. Background Art

[0002] Networks using data analytics, such as those provided by the Third Generation Partnership Project (3 rd generation partnership project, 3GPP) TM The defined fifth generation (5G) network may include infrastructure equipment (e.g., access points or base stations, core network equipment, gateways, etc.) and client equipment (e.g., mobile devices, user equipment (UE), Internet of things (IoT) terminal devices, etc.).

[0003] The network can use data analysis functions to provide data services. Such data services may include data collection services, data privacy protection services, data analysis services, and data transmission services. The network can collect data from data sources through data analysis functions and transmit the collected data to data consumers. Data consumers can use data to perform tasks such as data analysis, artificial intelligence (AI) training, and AI reasoning.

[0004] It is expected that, especially in future networks (such as 6G networks), a large number of data sources and data items may be scattered throughout the network. For example, in addition to network functions (NFs), user equipment (UEs), radio access networks (RANs), mobile edge computing (MEC) nodes, routers, and edge nodes, there may be other data sources. These other data sources may involve vertical service providers, such as IoT service providers, autonomous driving service providers, etc. These vertical service providers can be part of the network function or a third party. Each data source can have a data set collected from the network devices to which it is connected (e.g., identified by device IDs, UE IDs, etc.). For example, each vertical service provider can have data collected from its users (e.g., sensor data). Data consumers may need data from these data sources to perform tasks such as data analysis, AI training, and AI reasoning.

[0005] In future sixth generation (6G) networks, data analysis functions may be intermediate network functions for connecting data sources and data consumers. Accordingly, data consumers and data sources may be transparent to each other and may not necessarily be aware of each other's existence. However, existing proposals for implementing data analysis functions (e.g., in future 6G networks) need to be improved in various aspects. For example, in large networks, existing implementations or strategies (e.g., deployed in 5G networks) may be inefficient or even ineffective in the ability to collect data to support specific tasks in future 6G networks, especially when data consumers have limited information.

[0006] Therefore, there is a need for a method, apparatus, and system for configurable data collection in the context of networked data analysis and management that obviates or reduces one or more deficiencies of the prior art.

[0007] The purpose of providing this background information is to disclose information that the applicant believes may be relevant to the present invention. It is not necessarily intended, nor should it be interpreted, that any of the above information constitutes prior art against the present invention. Summary of the invention

[0008] The purpose of embodiments of the present invention is to provide methods, devices and systems for configurable data collection in the context of network data analysis and management (DAM). Embodiments can facilitate some or all of the following operations: automatic registration and organization of data sources, configurable task-specific data collection, data preprocessing or both, automatic detection of correlations between different available data, and interactive support for discovering appropriate data collection actions.

[0009] According to an embodiment of the present invention, a networked computerized system is provided, comprising a data representation manager (DRM) module, a data collection and preprocessing (DCP) module, a correlation manager (CM) module and a data source discovery (DSD) module. It should be noted that in other embodiments, one or more of the DRM module, the DCP module, the CM module and the DSD module may be omitted. The DRM module is used to obtain at least one representation of one or more data sources, each representation corresponding to one of the one or more data sources, and the representation is used to represent the characteristics of the data source. The DCP module is used to interact with members of the one or more data sources to obtain data from the one or more data sources according to a set of data collection actions. The CM module is used to determine at least one correlation between different data that can be obtained from the one or more data sources, and the at least one correlation is at least partially based on the at least one representation. The DSD module is used to interact with a data consumer component and the CM module to determine the set of data collection actions to be performed to support the request from the data consumer; and configure the DCP module to perform the set of data collection actions.

[0010] According to an embodiment of the present invention, the set of data collection actions may include: collecting raw data from the one or more data sources; or collecting raw data from the one or more data sources and preprocessing the raw data collected from the one or more data sources.

[0011] According to an embodiment of the present invention, preprocessing the raw data may include one or more of the following operations: merging the raw data; filtering the raw data; cleaning the raw data; and normalizing the raw data.

[0012] According to an embodiment of the present invention, the at least one representation may include one or more of the following: information indicating the data source; information indicating the data that the data source can provide. The information indicating the data source may include one or more of the following: data source type; application category for which data from the data source can be used; location of the data source; services that the data source can provide; capabilities of the data source; context information of the data source. The information indicating the data that the data source can provide may include one or more of the following: metadata; data name; data title; data tag; keyword; data semantics; data attribute; data feature; data mode; data format; and application category for which the data can be used.

[0013] According to an embodiment of the present invention, the DRM module may be used to receive the at least one representation. The DRM module may be used to generate the at least one representation at least in part by: requesting a report from the data source; receiving and processing the report to obtain the at least one representation. Requesting a report from the data source may include: sending representation configuration information to the data source, the representation configuration information indicating the representation of the data to be provided by the data source to the DRM module.

[0014] According to an embodiment of the present invention, the DRM module may be used to receive data from the data source and process the received data into the at least one representation. The DRM module may also be used to send the at least one representation and the received data from the one or more data sources to the CM module. The CM module may evaluate at least one correlation between parts of the received data by parsing the received data and the at least one representation.

[0015] According to an embodiment of the present invention, the at least one correlation between different data obtainable from the one or more data sources may be an indication of coherence between two or more members of the different data. The coherence may reflect one or more of the following: a degree of equality or inequality; a degree of similarity or dissimilarity; a degree of including or excluding dependencies; and a degree of transitive correlation.

[0016] According to one embodiment of the present invention, the DRM module may be used to interact with the CM module to start the CM module to perform the determination of at least one correlation between different data that can be obtained from the one or more data sources. The interaction with the CM module may include: the DRM module sends an evaluation request to the CM module. The evaluation request may include one or more of the following: one or more representations of the at least one representation; a correlation type to be used for the determination of at least one correlation; an indication that the CM module is to evaluate the correlation information of the one or more representations of the at least one representation; an indication that the CM module is to perform the determination of at least one correlation based on the correlation type; an application category identifier indicating the application category to which the at least one representation belongs, or indicating that the CM module is to determine at least one correlation between the at least one representation and the application category, or a combination of these two indications; a data source identifier identifying one or more members of the one or more data sources, the one or more data sources providing the associated one or more representations in the at least one representation; one or more computer memory addresses storing the original data of one or more representations of the at least one representation. The interaction with the data consumer component may include: generating a query plan, the query plan indicating the data to be collected and the members of the one or more data sources from which the data is to be collected.

[0017] According to an embodiment of the present invention, configuring the DCP module to perform the set of data collection actions may include: selecting one or more DCP module instances from a plurality of DCP module instances based on the set of data collection actions, and configuring the selected one or more DCP module instances from the plurality of DCP module instances. The DCP module may be used to provide the result of the set of data collection actions to the data consumer.

[0018] According to an embodiment of the present invention, the interaction between the DSD module and the data consumer component may include: receiving and processing the content of the request from the data consumer component. The content of the request from the data consumer component may include parameters indicating one or more of the following: an application category identifier indicating the category for which the result of the data collection action is to be used; an indication of one or more characteristics of the data required by the data consumer; the number of related data sets required by the data consumer; an indication of the dependencies of the data sets required by multiple parties, and indicating the required correlation between the data sets required by multiple parties; an address of the device of the data consumer, the device being designated to receive the results of the set of data collection actions; one or more final target data requirements of the results of the set of data collection actions to be sent to the data consumer. The networked computerized system may also be used to perform the configuration of the DCP module to perform the set of data collection actions based at least in part on the parameters, wherein the parameters may be included in the request from the data consumer component.

[0019] According to an embodiment of the present invention, the DCP module may include a DCP controller and a plurality of DCP point devices. The DCP point device may respond to a configuration instruction of the DCP controller. The configuration instruction may enable the DCP point devices to jointly perform the set of data collection actions. The DCP controller may be deployed in the control plane of a network, and the DCP point devices may be deployed in the user plane or the data plane of the network. Due to the configuration of the DCP module, one of the DCP point devices may be used as an anchor device to provide the results of the set of data collection actions to one or more devices of the data consumer. The DCP controller may be used to perform one or more of the following: determining or optimizing one or more rules for preprocessing the acquired data; selecting one of the DCP point devices to perform the set of data collection actions; configuring and activating one of the DCP point devices; optimizing resource scheduling to support the execution of the set of data collection actions. The DCP controller may be configured to send a data collection and preprocessing requirement message to at least one of the DCP point devices, the data collection and preprocessing requirement message causing the at least one of the DCP point devices to perform one or more data collection tasks, data preprocessing tasks, or both, the tasks being configured based on the content of the data collection and preprocessing requirement message. The parameters of the data collection and preprocessing requirement message may include one or more of the following: an identifier of one of the DCP point devices to be configured and activated; an indication of whether data preprocessing is required; a data query statement indicating the type of raw data to be collected from a specified one of the set of networked data sources; an address of a device to which the at least one of the DCP point devices is to forward output; a requirement for final target data to be sent to the data consumer; and a data preprocessing rule indicating how to preprocess the collected raw data.

[0020] According to an embodiment of the present invention, the DCP module may include one or more devices, each device being configured to provide a capability indication of each device to the DSD module, and the DSD module performs the configuration of the DCP module based in part on the capability indication.

[0021] According to an embodiment of the present invention, the interaction between the DSD module and the CM module may include: sending a correlation information request to the CM module, the correlation information request specifies one or more types of correlations, and the correlation information request is a request to the CM module to identify the specified one or more types of correlations between members of the different data that can be obtained from the set of networked data sources. The correlation information request may include one or more parameters for specifying the required correlation information, and the one or more parameters include one or more of the following: an application category identifier indicating an application to be associated with the required correlation information; one of the representations of the data source associated with the required correlation information; and a correlation type associated with the required correlation information.

[0022] According to an embodiment of the present invention, configuring the DCP module to perform the set of data collection actions may include: the DSD module provides one or more configuration parameters to the DCP module, and the one or more configuration parameters include one or more of the following: an identifier of the DCP point device to be configured and activated; an indication of whether the acquired data is to be preprocessed; an indication of the original data to be collected from the specified member of the set of networked data sources; the address of the device to which the DCP module is to forward the output; an indication of the requirement for the final target data to be sent to the data consumer; an indication of one or more rules to be applied by preprocessing the acquired data; an indication of one or more data correlations between different representations involved in the representation of the data source, the indication being used for preprocessing the acquired data. The indication of one or more rules to be applied by preprocessing the acquired data can be used to indicate one or more of the following: one or more rules to be used to merge the acquired data; one or more rules to be used to filter the acquired data; one or more rules to be used to clean the acquired data, normalize the acquired data, or both; one or more indications of portions of the acquired data to which association rules in the one or more rules are to be applied; one or more conditions that trigger implementation of the association rules in the one or more rules.

[0023] According to an embodiment of the present invention, a data representation manager (DRM) networked computerized device is provided, which is used to: obtain at least one representation of one or more data sources, each representation corresponding to a corresponding data source in the one or more data sources, and the representation is used to represent the characteristics of the data source; interact with one or more other devices to support determining the correlation between different data that can be obtained from the one or more data sources, and the correlation is based at least in part on the at least one representation of the data source. In such an embodiment, the DRM device can be configured in one or more ways described above for the DRM module of the networked computerized system.

[0024] According to an embodiment of the present invention, there is provided a data collection and preprocessing (DCP) networked computerized device for: interacting with one or more other devices to configure the DCP device to perform a set of data collection actions; interacting with members of a set of one or more data sources to obtain data from the set of one or more data sources according to the set of data collection actions. The DCP module may also be used to preprocess the obtained data according to the set of data collection actions. In such an embodiment, the DCP device may be configured in one or more of the ways described above for the DCP module of the networked computerized system.

[0025] According to an embodiment of the present invention, there is provided a correlation manager (CM) networked computerized device for: determining at least one correlation between different data available from a set of networked data sources, the at least one correlation being based at least in part on at least one acquired representation of a data source belonging to the set of networked data sources; interacting with one or more other devices to determine a set of data collection actions to be performed to support a request from a data consumer. In such an embodiment, the CM device may be configured in one or more of the ways already described above for the CM module of the networked computerized system.

[0026] According to an embodiment of the present invention, there is provided a data source discovery (DSD) networked computerized device for: interacting with a data consumer component and one or more other devices to determine a set of data collection actions to be performed to support a request from a data consumer; and configuring another device to perform the set of data collection actions. In such an embodiment, the DSD device may be configured in one or more of the ways described above for the DSD module of the networked computerized system.

[0027] According to an embodiment of the present invention, a data management method is provided. The method may include: a data management module obtains at least one representation of one or more data sources, each representation corresponds to one of the one or more data sources, and the representation is used to represent the characteristics of the data source; the data management module receives a request from a data consumer, the request is used to request data; the data management module determines the data requested by the data consumer according to the request information and the at least one representation. Determining the data requested by the data consumer according to the request information and the at least one representation may include: collecting raw data from the one or more data sources according to the request information and the at least one representation; or collecting raw data from the one or more data sources according to the request information and the at least one representation, and preprocessing the raw data collected from the one or more data sources. The method may include: determining one or more rules for preprocessing the raw data; or determining resource scheduling to support the execution of the raw data collection or the raw data preprocessing. The one or more rules may include at least one of the following: one or more rules to be used for merging the raw data; one or more rules to be used for filtering the raw data; one or more rules to be used for cleaning the raw data; one or more rules to be used for normalizing the raw data; one or more indications of a portion of the raw data to which an association rule in the one or more rules will be applied; one or more conditions that trigger the implementation of an association rule in the one or more rules.

[0028] In various embodiments, the method may omit one or more actions. For example, a method may be provided that performs an action associated with only one of the following in this document: a DRM module or device, a DCP module or device, a CM module or device, and a DSD module or device. For another example, a method may be provided that performs an action associated with two or more such modules or devices.

[0029] According to an embodiment of the present invention, the data management module obtaining at least one representation of one or more data sources may include: requesting a report from the data source; receiving and processing the report to obtain the at least one representation. Requesting the report may include: sending representation configuration information to the data source, the representation configuration information indicating the representation of data to be provided by the data source to the DRM module.

[0030] According to an embodiment of the present invention, the data management module acquiring at least one representation of one or more data sources may include: receiving data from the data source, and processing the received data into the at least one representation. The data management module may determine at least one correlation between parts of the received data by parsing the received data from the one or more data sources and the at least one representation.

[0031] According to an embodiment of the present invention, the data management module obtaining at least one representation of one or more data sources may include: the data management module receiving at least one representation of the one or more data sources.

[0032] According to an embodiment of the present invention, the data management method may include: the data management module determines at least one correlation between different data that can be obtained from the one or more data sources, and the at least one correlation is at least partially based on the at least one representation of the data source. The data management module determines the data requested by the data consumer according to the request and the at least one representation, which may include: the data management module determines the data requested by the data consumer according to the request, the at least one representation and the at least one correlation.

[0033] According to an embodiment of the present invention, the data management module determining the data requested by the data consumer based on the request and the at least one representation may include: the data management module generating a query plan, the query plan indicating the data to be collected and members of the one or more data sources from which the data is to be collected; the data management module determining the data requested by the data consumer based on the query plan.

[0034] According to an embodiment of the present invention, the data management method may include: the data management module providing the data requested by the data consumer to the data consumer.

[0035] According to an embodiment of the present invention, the request from the data consumer may include parameters indicating one or more of the following: an application category identifier, indicating the category for which the result of the data collection action is to be used; an indication of one or more characteristics of the data required by the data consumer; the number of related data sets required by the data consumer; an indication of dependencies of data sets required by multiple parties, and indicating the required correlations between the data sets required by multiple parties; the address of the device of the data consumer, which is designated to receive the results of the set of data collection actions; and one or more final target data requirements for the results of the set of data collection actions to be sent to the data consumer.

[0036] Embodiments are described above in conjunction with various aspects of the present invention, and these embodiments can be implemented based on these aspects. It will be appreciated by those skilled in the art that embodiments can be implemented in conjunction with the aspects describing these embodiments, but can also be implemented together with other embodiments of this aspect. When the embodiments are mutually exclusive or incompatible with each other, this will be apparent to those skilled in the art. Some embodiments can be described in conjunction with an aspect, but can also be applied to other aspects, which will be apparent to those skilled in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0038] Figure 1A The operation of the DAM provided by the embodiment of the present invention is shown.

[0039] Figure 1B Details of a networked computerized DAM system provided by an embodiment of the present invention are shown.

[0040] Figure 2A The data source set provided by the embodiment of the present invention and different original data that can be obtained from each of the data sources are shown.

[0041] Figure 2B Two modes provided by an embodiment of the present invention are shown, wherein representations of data sources can be obtained through these two modes.

[0042] Figure 3A The embodiment of the present invention provides Figure 2A The correlation between data sources.

[0043] Figure 3B The exemplary content of the correlation library provided by the embodiment of the present invention is shown.

[0044] Figure 3C The embodiment of the present invention provides Figure 3A A more detailed representation of (correlation plot).

[0045] Figure 3D The embodiment of the present invention provides Figure 3C Simplified correlation graph derived from the graph of .

[0046] Figure 3E The embodiment of the present invention provides Figure 3C Another simplified correlation graph derived from the graph of .

[0047] Figure 4 , Figure 5A and Figure 5B The operation of the DAM provided by the embodiment of the present invention is shown.

[0048] Fig. 6A The figure shows the overlap of data sets provided by two different data sources according to an embodiment of the present invention.

[0049] Figure 6B A correlation graph provided by an embodiment of the present invention is shown, including information about several data sources and their representations.

[0050] Fig. 7A A horizontal federated learning scenario provided by an embodiment of the present invention is shown.

[0051] Figure 7B A vertical federated learning scenario provided by an embodiment of the present invention is shown.

[0052] Figure 8 The DAM and associated federated learning operations provided by an embodiment of the present invention interacting with a controller are shown.

[0053] Fig. 9 An exemplary implementation of the DAM provided by an embodiment of the present invention is shown, wherein optimal target data sets are obtained for multiple parties of the data consumer components, and the required target data sets of the data consumer components are related.

[0054] Fig.10 The data sources and data representations provided by other embodiments of the present invention are shown.

[0055] Fig.11 An exemplary implementation of the DAM provided by an embodiment of the present invention is shown, wherein data collection and data synthesis are performed.

[0056] Fig.12 is a schematic diagram of an electronic device provided by an embodiment of the present invention.

[0057] It should be noted that throughout the drawings, like features are identified by like reference numerals. DETAILED DESCRIPTION

[0058] Embodiments of the invention provide one, some, or all of the following: (e.g., networked) representation of data sources; configurable data collection and data preprocessing; resources for detecting and recording or tracking correlations between different available data; resources for interactively determining and implementing data collection actions to support data consumer requirements. Embodiments may be integrated as a networked computerized system into a network such as a 5G or 6G data and / or communication network, such that operations are performed by components of the network itself.

[0059] Embodiments of the present invention can be used to reduce potential problems in implementing data analysis. For example, in certain implementations contrary to the embodiments of the present invention, it can be assumed that the data consumer knows exactly what data is needed for a particular task. Then, the data consumer is expected to send a data request (e.g., through a subscription) with accurate indications (e.g., device ID, SUPI ID, cell ID, PDU session ID, network slice ID) to the network to guide the collection of data from one or more data sources. However, it should be recognized in this article that the data consumer may not know exactly what data is needed for a particular task (e.g., AI training) and the identity of the relevant data source. When such a data consumer requests the data analysis operating system to collect data from a data source, only limited indication information (e.g., fuzzy data features) indicating the required data may be available. It is challenging, complex, and resource-intensive to fully express and meet the requirements of data consumers based on such limited information.

[0060] Another potential problem with traditional data analysis implementations that may be addressed by embodiments of the present invention is that there may be a large number of data sources available (either to provide data or as participants) for the task of serving a data consumer. If all of this large amount of available data is exposed to the data consumer, data collection and transmission overhead may be increased. In addition, even if such a large amount of data is provided to the data consumer, the processing overhead of the data consumer may also be increased. Therefore, in some (but not necessarily all) embodiments of the present invention, data collection may be limited to collecting a limited amount of (e.g., the most suitable) data, data processing may be performed before transmitting the data to the data consumer, or both data collection and data processing. Accordingly, a set of data collection actions may be configured to retrieve only specific data that supports the data consumer's request.

[0061] Figure 1A The operation of the data analysis and management (DAM) 100 provided by an embodiment of the present invention is shown. The DAM 100 performs a data collection action (e.g., collects selected data) on a data source (set) 010, and provides a result 115 of the data collection action (e.g., provides the collected data or the result of processing the collected data) to a data consumer 030. The DAM 100 can collect data and provide results in response to queries conducted by the data consumer 030 (or another associated party) or generated by the DAM with the participation of the data consumer. Throughout the present invention, the DAM may also be referred to as a data management module. The various actions of the components (e.g., modules) of the DAM described herein may be viewed as actions of the DAM itself.

[0062] In various embodiments, the DAM supports data discovery and configurable data acquisition operations. Such support may involve intelligently and interactively assisting data consumers in selecting appropriate data sources and data items by interacting with the data consumers. Such assistance may be provided even when the data consumers provide only limited indicative information (e.g., ambiguous data features) to the DAM. For example, the data consumers may be human users or automated computing devices.

[0063] In various embodiments, the DAM supports data preprocessing. The acquired data can be preprocessed when necessary, but not all scenarios require preprocessing. For example, when the raw data acquired by the DAM cannot be used directly by the data consumer (component), or when the raw data includes content that is irrelevant to the requirements of the data consumer (also referred to herein as "useless" or "irrelevant" data), or when the raw data includes redundancy that can be deleted, preprocessing can be performed. For example, a data consumer may require a composite data set (that cannot be obtained from any single data source). In this case, the DAM can be used to create a composite data set based on (e.g., by merging or splicing) data from different data sources.

[0064] It can also be considered that the raw data obtained from the data source may include useless, irrelevant data, redundant data, or non-directly usable data. In addition, the data from two data sources may be partially redundant. Accordingly, in various embodiments, the DAM can be used to preprocess the raw data and then provide the results of the preprocessing (or more generally, the results of the data acquisition action) to the data consumer. For example, different data set structures may be required to support different tasks of the data consumer. Accordingly, the raw data can be preprocessed into the required data structure.

[0065] It can be considered that different data consumers (or multiple parties of a single data consumer) are not necessarily independent. The data sets required by different consumers may be related (for example, in terms of ID and feature alignment) and may be interdependent. Accordingly, in various embodiments, the DAM can be used to select and prepare data for such multiple data sets together, rather than selecting and preparing data for each data set separately. This can help meet the requirements for data correlation between multiple consumers (or multiple parties of consumers). Therefore, the required correlation between different data sets can be provided.

[0066] It can also be considered that given that the data required by different data consumers (or multiple parties of consumers) may be related, DAM can be used to maintain (e.g., determine, record, or track) the correlation between different data. DAM can be used to use this correlation information to help select appropriate data to be provided to each of multiple data consumers (or multiple parties of consumers). In addition, some data may be related to specific services, applications, or uses. In such a scenario, DAM can be used to maintain (e.g., determine, record, or track) the correlation between data and services, applications, uses, or a combination thereof. DAM can then use this information to support the selection of appropriate data to provide to data consumers.

[0067] In various embodiments, in order to correlate data from data sources, DAM can be used to provide the function of obtaining (e.g., capturing and maintaining) information about one or more data sources in the data source. For example, the information may include information about what data the data source can provide (i.e., the data that the data source can provide). There can be multiple data sources (e.g., NF, RAN, UE, network sensors, vertical service providers). In addition, these data sources can dynamically register to and leave the network. Therefore, in an embodiment of the present invention, DAM is used to capture, track and maintain information about data sources so that DAM can complete data discovery, collection and selection in an orderly manner based on the data source information.

[0068] According to an embodiment of the present invention, DAM is used to provide one or more modules to implement the above functions. One or more of the above functions can be implemented by one or more modules. The present application does not limit the correspondence between functions and modules. For example, a data representation manager (DRM) module for managing data source information (especially what data the data source can provide), and a correlation manager (CM) module for determining (e.g., capturing, detecting and maintaining) data correlation information. Data correlation information may include information about correlations between data, correlations between data and specific services / applications / uses, etc. Data correlation information may include information about available data (e.g., data that has not yet been acquired), wherein the data correlation information is based on data source information, for example. Based on the data source information and correlation information, DAM can be used to discover and select suitable (e.g., most suitable) data to provide to data consumers. Then DAM can collect useful data from the data source, optionally pre-process the data (e.g., by data filtering, data splicing, data cleaning, etc.), and transmit the pre-processed data to the data consumer.

[0069] As used herein, the term "module" or "device" may refer to a computerized device or functional aspects thereof that may be provided using a networked computing device (whether dedicated or virtualized). For example, a module may be a network function. A module may work with other modules to provide an overall system of networked computerized (e.g., electronic) devices.

[0070] Accordingly, embodiments of the present invention relate to methods, systems, and apparatus that facilitate data discovery and preprocessing, so as to intelligently assist data consumers in selecting and preparing appropriate data for use.

[0071] The current 5G network dedicated to providing communication services includes a network repository function (NRF) for managing what types of services the network function (NF) can provide. For example, the NF registers with the NRF, and the NRF maintains information about the types of services that the NF can provide. According to an embodiment of the present invention, a network that provides data (e.g., for AI tasks) may include a DRM module, which is used to manage what types of data a data source (e.g., NF, UE, RAN, MEC, database, vertical service provider) can provide. For example, a data source can register with the DRM, and the DRM can maintain information about what types of data the data source can provide. The DRM can be used to obtain information indicating the data that can be provided by each of one or more data sources (e.g., data name, data title, metadata, keywords, data attributes, data features, application categories).

[0072] According to an embodiment of the present invention, for example, in order to support the provision of data for AI tasks, factors such as data attributes and data characteristics can be used as the basis for correlating data from multiple data sources. According to an embodiment, the CM module is used to manage such data correlation. In addition to or instead of an identifier associated with the data, the CM may also consider various factors (e.g., data attributes) to evaluate correlation. Such identifiers (IDs) may include SUPI, UE IP, PDU session IDs managed by NF UDM and AUSF, and the like.

[0073] In various embodiments, it can be considered that the data consumer does not necessarily know or is actually prohibited from knowing the exact identity of the data source that can provide appropriate data. Accordingly, embodiments of the present invention are used to assist data consumers in discovering and selecting appropriate data (for a given task) from a potentially large number of available data items and a potentially large number of data sources. Accordingly, a data source discovery (DSD) module is provided to interactively facilitate the discovery of suitable data (and corresponding data collection actions). This discovery can be performed while taking into account the requirements of the data consumer, data relevance information, data access overhead, etc. The DSD module can be used to generate a data query statement that indicates the data to be collected and the data source from which the data is to be collected. That is, the DSD module can configure other devices, such as a data collection and preprocessing (DCP) module, to perform a set of data collection actions that have been determined by the DSD module to support requests from data consumers.

[0074] It can be considered that the raw data from the data source may not be directly or immediately available and may include irrelevant (with respect to a given request) or redundant data. In a 5G network dedicated to providing communication services, a session management function (SMF) is used to select a user plane function (UPF) instance and configure communication rules to the UPF for communication data transmission. The UPF performs the transmission of communication data based on the communication rules under the control of the SMF. According to an embodiment of the present invention, for example, in order to support the provision of data for an AI task, a module or function is provided to select a data preprocessing instance, construct data preprocessing rules for one or more of the instances, and configure the data preprocessing rules. The data preprocessing instance can perform data preprocessing tasks under the control of the DSD module. The DSD module can construct data preprocessing rules (e.g., rules for modifying and cleaning data that is not directly available, filtering and deleting irrelevant or redundant data) to indicate how to preprocess the collected raw data into target final data required by, for example, a data consumer.

[0075] The DCP module can be used to perform data preprocessing (e.g., merging, concatenating, filtering or cleaning data, or a combination thereof) under the control or instruction of the DSD module. In addition, the DSD module can select one or more of several instances of the DCP module and configure these instances of the DCP module to operate according to specified data preprocessing rules.

[0076] Accordingly, in various embodiments of the present invention, the DSD module can assist data consumers in discovering and selecting suitable data (e.g., related data sets), while comprehensively considering the requirements of data consumers and data relevance information. The CM function can evaluate and maintain data relevance information (e.g., maintain a data relevance library), and transmit the data relevance information to the DSD for use. Information about the data can be captured by the DRM module. The DRM module can transmit the captured data information to the CM module for relevance evaluation. In addition to the data discovery capability, the DSD function can also have the ability to construct data preprocessing rules, which indicate how to preprocess the collected raw data (e.g., non-directly available, irrelevant or redundant data) into the final target data required by the data consumer. The DSD module can configure the data preprocessing rules to the DCP that performs data preprocessing.

[0077] Figure 1B The networked computerized DAM system 100 provided by an embodiment of the present invention is shown. The system shown in the figure can at least be used to support data discovery, data collection and data preprocessing. The system may include multiple modules, which may be provided using one or more real or virtual networked computing devices. The system shown in the figure includes a DRM module 200, a DCP module 300, a CM module 400 and a DSD module 500. It should be noted that Figure 1B The modules included in the DAM system in the embodiment are only examples, and the functions of each module described below can also be implemented by one or more other modules. For example, the functions of the DRM module 200 and the DCP module 300 can be replaced by another module, or can be replaced by more modules.

[0078] The DRM module 200 is used to obtain representations of data sources (e.g., 010a, 010b, 010c) belonging to a (e.g., networked) data source set 010. At least one representation is obtained, each representation corresponding to a specific data source. The representation of the data source can be used to represent one or more characteristics of the data source. The characteristics can correspond to information indicating data that can be provided by the data source, or other information about the data or the data source itself. For example, the information may include an indication of data acquisition capabilities.

[0079] In various embodiments, the representation of a given data source may include characteristics such as information indicating the given data source itself (data source parameters), information indicating data that the given data source can provide (data parameters), or both. Such information indicating a given data source may include one or more data source parameters, non-limiting examples of which are described below.

[0080] The data source parameter may include a data source type. The data source type may include one or more of the following: UE, NF, RAN node, vertical service provider, third party, mobile edge computing (MEC) node, router, and edge node.

[0081] The data source parameters may include an application category (e.g., an application category ID) for which data from a given data source may be used. The application category may indicate a purpose for which data from a given data source may be used, which may be one or more of: federated learning, AI training, federated AI reasoning, network resource management, UE mobility management, network operation management, network policy management, network access management, slice management, session management, enhanced mobile broadband (eMBB) service, ultra-reliable low latency communication (URLLC) service, massive machine-type communication (mMTC) service, vertical services, vehicle services, and sensing services.

[0082] The data source parameters may include the location of a given data source, which may include one or more of: a cell or tracking area where the data source is located, a network to which the data source belongs, and a geographic location of the data source.

[0083] Data source parameters may include services that a given data source can provide, which may include one or more of the following: raw data sharing services, raw data privacy protection services, wherein the data source can work alone or in cooperation with the network to protect the privacy included in or inferred from the raw data.

[0084] The data source parameters may include the capabilities of a given data source. The capabilities of a given data source may be one or more of the following: energy available for reporting data to be collected, maximum transmit power available for reporting data to be collected, maximum activation time of the data source, quality of data that the data source can provide, and quantity of data that the data source can provide.

[0085] Data source parameters may include contextual information for a given data source.

[0086] Information indicative of the data that a given data source is capable of providing may include one or more data parameters, non-limiting examples of which are described below.

[0087] Data parameters may include metadata. Metadata may provide information about the data available from a given data source. Metadata may describe one or more of the following: the source, size, format, summary, overview, and other characteristics of the data. Metadata helps provide information about the content of the data or data set.

[0088] The data parameter may include a data name. The data name may be one or more of the following: a string of data, an index of data, a hash of data (e.g., a hash code, a hash root), and may be indicated using a named data networking (NDN) scheme, an information centric networking (ICN) scheme, or other schemes.

[0089] The data parameter may include a data title. The data title may be one or more of: a string of data, an index of data, a hash of data (e.g., a hash code, a hash root), and may be indicated using a named data networking (NDN) scheme, an information centric networking (ICN) scheme, or other schemes.

[0090] The data parameter may include a data tag. The data tag may be one or more of: a string of data, an index of data, a hash of data (e.g., a hash code, a hash root), and may be indicated using a named data networking (NDN) scheme, an information centric networking (ICN) scheme, or other schemes.

[0091] The data parameters may include keywords, which may be specific words abstracted from the data, for example.

[0092] The data parameter may include the data semantics, such as a sentence or paragraph describing the data.

[0093] Data parameters may include data attributes. For example, in a relational database that organizes data into a table of one or more columns and rows, a column of the table may be a data attribute that represents a value attributed to an object.

[0094] Data parameters may include data features. Data features may be abstracted from data. Data features may correspond to features used to describe AI model input data during AI training or AI reasoning. Data features may be obtained from data through one or more of embedding, feature engineering, and representation learning methods.

[0095] The data parameters may include a data pattern, which may be a statistical distribution of the data and / or a graph of the data.

[0096] The data parameters may include data formats. The data formats may be normalized or standardized data structures of video (e.g., avi, flv), audio (e.g., MPEG-4, MIDI), images (e.g., jpg, png), databases, and text.

[0097] The data parameters may include application categories for which the data can be used. Application categories may also be referred to as service categories or usage categories. Application categories may indicate the purpose for which data from a given data source can be used, which may be one or more of the following: federated learning, AI training, federated AI reasoning, network resource management, UE mobility management, network operations management, network policy management, network access management, slice management, session management, enhanced mobile broadband (eMBB) services, ultra-reliable low latency communication (URLLC) services, massive machine-type communication (mMTC) services, vertical services, vehicle-mounted services, and sensing services.

[0098] Thus, the DRM module may act as a repository for information about available data sources, data that the available data sources may provide, and related details.

[0099] The data collection and preprocessing (DCP) module 300 is used to collect data from the data source and transmit the data to the data consumer. Therefore, the DCP module interacts with the data source to obtain data from the data source according to the data collection action. The DCP module 300 can also be used to perform data preprocessing when necessary. The data collection, preprocessing and transmission actions can be configured in response to instructions provided by the DSD module 500. The DCP module 300 can be used to transmit the collected or processed data to the data consumer 030.

[0100] To further define, as used herein, data collection actions may include, for example: collecting raw data from a data source; collecting raw data from a data source and preprocessing the collected raw data. Preprocessing may include tasks such as merging, filtering, cleaning, and normalizing the raw data. These tasks may be performed on the collected raw data or other data generated by other preprocessing actions.

[0101] The CM module 400 is used to determine (e.g., detect and maintain or track) at least one correlation between different data that can be obtained from a set of data sources. These correlations can be maintained in a correlation library 450, a database, or other data structure. The correlations are based at least in part on the representation of the data source provided by the DRM module 200.

[0102] The DSD module 500 is used to interact with the data consumer component 030 and the CM module 400 to determine the set of data collection actions to be performed to support the request from the data consumer. The DSD module 500 is also used to configure the DCP module 300 to perform a specified set of data collection actions. The DSD module 500 can select an appropriate or optimal set of data sources from the set 010 for performing the determined set of data collection actions. This selection can be made with comprehensive consideration of data relevance information, data access overhead, and the requirements of the data consumer.

[0103] The DSD module 500 may also be used to perform data query statement generation, for example as part of or after determining a set of data collection actions. Such query statements may be generated to indicate the raw data to be collected and the data source from which the raw data is to be collected. The DSD module 500 may also trigger a data query request to the DCP module 300.

[0104] In various embodiments, the DSD module 500 may also determine data acquisition and preprocessing rules implemented by the DCP module 300. This determination may be performed to indicate how to obtain the final target data from the raw data. For example, preprocessing rules may be determined to modify and clean up data that is not directly usable, filter irrelevant or redundant data, or a combination thereof.

[0105] The DSD module 500 may also be used to select one or more instances of the DCP module or its components (e.g., one or more points or one or more instances) for performing data collection and preprocessing. For example, such selection may be based on access overhead, data consumer mobility, communication link status, DCP capabilities, etc. or a combination thereof.

[0106] Access overhead may involve the DSD selecting one or more DCP module components that are closest to a data source and / or data consumer so that data access overhead (eg, data collection and / or data transfer overhead) may be potentially reduced.

[0107] Data consumer mobility may involve DSD selection. The coverage area may include one or more DCP module components in the area where the data consumers are typically located (e.g., mobile or fixed), thereby potentially facilitating the switching and exchange of DCP modules or data consumers.

[0108] The communication link state may involve the DSD selecting one or more DCP module components that have established a communication link with a data source or a data consumer, so that link establishment delays and overheads can be potentially reduced. Additionally or alternatively, the DSD module may select one or more DCP module components that have a better (e.g., more reliable, more stable, stronger signal, etc.) communication link with a data source or a data consumer, so that a higher link transmission speed can be achieved.

[0109] DCP capabilities may involve the DSD selecting one or more DCP module components with corresponding preprocessing capabilities for preprocessing raw data. Additionally or alternatively, the DSD may select one or more DCP module components with sufficient capabilities (e.g., sufficient computing resources, lower load, etc.) to perform one or more data acquisition and (if necessary) preprocessing tasks.

[0110] Once selected, the DSD module 500 may configure and activate the DCP module 300 or selected instances or components thereof according to the determined data acquisition and pre-processing rules.

[0111] The above modules can be flexibly deployed in different network layers. For example, the DRM module 200, the CM module 400 and the DSD module 500 and at least the DCP controller part of the DCP module 300 can be deployed in but not limited to the control plane (CP) 021. The DCP point device part of the DCP module 300 can be deployed in but not limited to the user plane (UP) 022 or the data plane. The DRM, CM, DSD and DCP modules can be deployed on the user equipment (UE) side, the network side or as part of the network function (NF) or deployed with a third party. The network side can include, for example, a radio access network (RAN) node or a core network (CN) function.

[0112] Figure 2AA set of data sources and different data representations of each of the data sources are shown. Data source 010a has corresponding data representations 011a, 011b, 011c; data source 010b has corresponding data representations 012a, 012b, 012c; data source 010c has corresponding data representations 013a, 013b. Some of the raw data indexed by the data representation indicated by the dashed line 113 in the raw data cannot be read by the DAM (i.e., the representation with the dashed line will index the raw data that cannot be parsed by the DAM). Some of the raw data indexed by the data representation indicated by the solid line 114 in the raw data can be read by the DAM (i.e., the representation with the dashed line will index the raw data that can be parsed by the DAM). Provide Figure 2A It is to illustrate the data representation captured and maintained by the DRM module. As used herein, data representation can also be referred to as a representation of a data source. Accordingly, as described above, a data representation can indicate information about the original data, information about the data source that provides the original data, or both. The original data can be indexed by the corresponding representation of its data source.

[0113] In various embodiments, the representation of the data source may be provided in the form of a hash code, a hash root, or named in a named data networking (NDN) scheme. Figure 2B Two modes are shown, wherein the representation of the data source can be obtained by these two modes. In the first mode 240, the data source 010 registers (241) with the DRM module 200 and reports the representation of the data item with appropriate granularity (e.g., columns, tables, and data sets). The representation to be reported can be configured (242) to the data source by the DRM module 200, and the DRM module can update (244) the representation library (e.g., knowledge base). In more detail, the configuration operation 242 can be omitted, and the DRM module 200 sends the representation configuration information to the data source 010 (e.g., as part of the request). The representation configuration information indicates that the DRM module instructs the data source to report (if the data source can provide relevant data indexed by the representation). The DRM module can maintain a local representation library and configure part of the representation in the representation library as a data source. Accordingly, the DRM module can generate a representation by requesting a report from the data source and receiving and processing the report (including data). Alternatively, the DRM module can receive the representation, for example, through a report.

[0114] Different application categories may be associated with different sets of data representations. The DRM module may obtain a corpus of data representations and may select and maintain useful data representations for each application category. The mapping between representations and original data may be reported (243) to the DRM module 200 by the data source 010. The original data may be indexed by its representation. If the data source cannot provide relevant data indexed by the representation, the data source may not perform the reporting (243).

[0115] In the second mode 250, if the original data can be parsed and analyzed by the DRM module 200, the data representation can be extracted by the DRM module using methods such as representation learning (embedding) or data analysis (e.g., by an autoencoder). Accordingly, the DRM module 200 can have the ability to read data in different formats (e.g., RDBMS, HDFS files and CSV files, text, images, sounds, audio, video, transmission data, wireless signal data, radio wave signal data) from different data sources and extract corresponding data representations from the original data.

[0116] According to the second mode 250, the data source 010 sends 251 raw data to the DRM module. The DRM module 200 then extracts data representations from the raw data based on data profiling or representation learning 252. The trained representation learning model or data profiling model can be learned by the DRM itself or pre-configured into the DRM by other parties. Thus, the DRM module can receive the raw data and process the received raw data into representations.

[0117] After the first mode or the second mode, the DRM module 200 may send a dependency management request to the CM module 400 in the form of a dependency management request 260, including a representation of the data it has collected. The message 260 may include a data source ID and a representation of the data source. The CM module may build (451) or modify its data dependency library based at least in part on the information in the dependency management request 260.

[0118] It should also be noted that the DRM module 200 can obtain the data source representation with the assistance of the DCP module. For example, the DCP module can interact with the data source, collect data from the data source, and then send the data to the CM module.

[0119] According to various embodiments, after obtaining data representations (representations of data sources), the DAM correlates the data representations. To facilitate this, the CM module is used to detect correlations in the data representations and maintain (track) the detected correlations, such as in a data correlation information library (e.g., correlation library 450). More specifically, the CM module can determine (e.g., track) correlations between different data that can be obtained from the data source based at least in part on the data representations collected by the DRM module. In various embodiments, the correlations between different data indexed by the data representations can be indicated or further evaluated by the correlations of the data representations.

[0120] In various embodiments, the detected and maintained correlations include correlations between different data. Additionally or alternatively, the detected and maintained correlations may include correlations between data and services, applications, usages, or a combination thereof.

[0121] In various embodiments, at least one of the correlations between different data includes an indication of the coherence between two or more instances of such different data, which may be raw data indexed by data representations. Coherence may include, but is not necessarily limited to, one or more of the following: a degree of equality or inequality; a degree of similarity or dissimilarity; a degree of inclusion or exclusion of dependency; and a degree of transitive correlation. The degree of equality or inequality may indicate the degree to which the values ​​of different data instances (items) are equal or unequal. The degree of similarity or dissimilarity may indicate the degree to which the attributes (such as data features or data patterns) of different data instances are similar or dissimilar. The degree of inclusion or exclusion of dependency may indicate the degree or range to which a data attribute set of one data instance in the data instance is included or not included in (e.g., excluded from) a data attribute set of another data instance in the data instance. The degree of inclusion or exclusion of dependency may indicate to what extent the data attribute sets of two data instances are disjoint or mutually exclusive. The degree of transitive correlation may correspond to the degree (level) of correlation transmitted from other fields, such as sequence correlation, combination correlation. For example, two data items may be serially related because their values ​​both represent a time field. For another example, two data items may be combinatorially related because they may be combined together for tasks such as AI training tasks. For another example, if each of the first data set and the second data set has a correlation relationship with a third data set, there may be a transitive correlation between the first data set and the second data set. That is, the two data sets may be related through the third data set; the transitive correlation between the first data set and the second data set is related through the third data set.

[0122] In various embodiments, the correlation between data and a service, application, usage, or a combination thereof may be or result from an indication that the data is available for a particular service, application, or usage.

[0123] In various embodiments, the CM module constructs a correlation graph, such as a knowledge graph, or an equivalent data structure. The data structure indicates the correlation between data sources, such as Figure 3A As shown. Figure 3A In, with Figure 2A Similarly, there may be different types of vertices, such as vertices 010a, 010b, 010c indicating data sources and vertices 011a, 011b, 011c, 012a, 012b, 012c, 013a, 013b indicating data representations. In addition, some vertices 011a, 011b, 012a, 012b, 013a may refer to data representations 114 of raw data whose indexes can be read by the DAM (i.e., data representations of raw data whose indexes can be parsed by the DAM), while other vertices 011c, 012c, 013b may refer to data representations 113 of raw data whose indexes cannot be read by the DAM (i.e., data representations of raw data whose indexes cannot be parsed by the DAM).

[0124] like Figure 3A As further shown in FIG. 1 , the edges in the correlation graph or chart can be weighted, wherein the edge weight corresponds to a correlation level (value), a correlation type, or a combination thereof. For example, edge weight 220a indicates that there is a content correlation between data representations R#1 011a and R#4 012b of two different data sources #1 010a and data source #2 010b, with a correlation level of 0.86. The correlation level can be a correlation coefficient between –1 and 1. A correlation coefficient close to 1 indicates a strong correlation, close to –1 indicates a strong non-correlation, and close to 0 indicates a lack of correlation. Another edge weight 220b indicates that there is a semantic correlation between data representations R#3 011c and R#6 012c, with a correlation level of 0.43. Another edge weight 220c indicates that there is a semantic correlation between data representations R#3 011c and R#8 013b, with a correlation level of 0.25. The correlation between data indexed by the data representations can be indicated or further evaluated by the correlation of the data representations.

[0125] The CM module can detect (one or more) correlation levels using one or more methods in a variety of methods. In some embodiments, the correlation level can be detected by content evaluation, wherein the CM module parses the provided original data, data representations, or a combination thereof. Accordingly, the DRM module can be used to send one or more data representations and data received from one or more data sources to the CM module, and the CM module can evaluate the correlation level of the representation by parsing (processing) both the original data and at least one data representation. Accordingly, the data management module (i.e., DRM) can determine the correlation level of one or more correlations of one or more data representations by parsing (processing) both the received data and at least one data representation. The correlation level between the data indexed by the data representation can be indicated or further evaluated by the correlation level of the data representation.

[0126] In some embodiments, the relevance level can be detected by a semantic-based assessment, such as natural language processing (NLP) or word embedding methods. For example, this can be performed in a case where the CM module cannot parse the data (e.g., raw data) and can only assess the relevance of the data based on the data representation. In some embodiments, the relevance level can be detected by reading preconfigured information. For example, the DRM module can maintain a representation library, and the CM module can be preconfigured to set the relevance level between data representations based on the contents of the representation library.

[0127] Figure 3B The example content 451 of the correlation library provided by the embodiment is shown. The correlation information library may indicate a data source, a correlation representation 455 of the data source, a correlation type 456, and a description 457. The correlation between different data sources may be represented, for example, by a triple parameter. The format of the triple may be (data source 1, correlation type, data source 2), indicating that data source 1 and data source 2 have a correlation of the type indicated by the correlation type.

[0128] For example, the first entry 455a [(DS1, DS2); (R1, R4), (R3, R6); 0.86, 0.43] means that there are related representations of data source 1 and data source 2, and the related representation pairs are representation 1 and representation 4, and representation 3 and representation 6, respectively, and the correlation levels of these two pairs are 0.86 and 0.43, respectively, and the correlation type is similarity 456a. A corresponding description 457a is also provided to indicate that the entry is used to correlate data between data source 1 and data source 2. In addition, data sources can be dynamically registered and unregistered with the system. CM can incrementally introduce and update the correlation library, for example, using inference rules (e.g., learning-based models) and automatic learning methods.

[0129] For another example, the second entry 455b [(DS1, DS3); (R3, R8); 0.25] means that there are related representations of data source 1 and data source 3, the related representation pair is representation 3 and representation 8, the correlation level of the pair is 0.25, and the correlation type is including dependency 456b. A corresponding description 457b is also provided to indicate that the entry is used to correlate data between data source 1 and data source 3. It should be noted that the content 451 reflects the following. Figure 3A The situation shown.

[0130] In various embodiments, data sources can dynamically register and unregister with the system. Accordingly, the CM module can incrementally import and update the correlation library. This can be performed, for example, using inference rules (e.g., learning-based models) and automatic learning methods.

[0131] In various embodiments, the CM module may be used to reduce high-dimensional correlation information to low-dimensional correlation information. This may be done to extract more important, more efficient, or more useful information. The CM module may maintain different representation correlations for different application categories. For example, Figure 3C yes Figure 3A 220 d , 220 e , 220 f , 220 g , 220 h , 220 i , and 220 j are set to 1 for completeness. Figure 3C This representation of the correlation graph shown corresponds to Figure 3A The original correlation graph shown. The original correlation graph can be simplified to a smaller scale correlation graph with fewer vertices and edges. Therefore, a simplified correlation graph can be provided. For example, the simplification can be performed using graph neural network (GNN) processing, graph embedding, link prediction, graph classification, node classification or other machine learning (ML) methods.

[0132] For example, Figure 3C , Figure 3D and Figure 3E As shown, the dependency graph may include vertices of different granularities. The dependency graph may include vertices indicating data sources and data representations (e.g., Figure 3A , Figure 3C and Figure 3D ), indicating only the vertices that represent the data, or indicating only the vertices that indicate the data source (e.g., as shown in Figure 3E ). The dependency graph may include different numbers of vertices, which indicate one or more of: a portion of a data source; a data representation selected from the entire data source; an entire representation library. Each edge weight indicates a level of dependency between two vertices (e.g., Figure 3C, where the relevance levels 220a, 220b, 220c, 220d, 220e, 220f, 220g, 220h, 220i, 220j are shown using edge weights. Other formats may also be used. For example, the relevance levels may be represented by the distance between vertices rather than by the distance between vertices. Figure 3D , and dependencies can be shown between different sources without the need for Figure 3E Representation shown.

[0133] Different graphs can provide different levels of flexibility and accuracy for use in the subsequent data discovery process. This can help make the data discovery process faster and / or more efficient.

[0134] Closely related representations or data sources can be embedded in the same group or cluster.

[0135] Figure 3D The embodiment provides Figure 3C A simplified correlation graph derived from the graph of . Figure 3D In a graph, there are data sources and data representations. For example, vertices are placed into a coordinate system where the horizontal and vertical coordinates are normalized. Various embedding methods can be used to map vertices into the coordinate system. Figure 3C The edge weights used in can be expressed as the distance between vertices. Figure 3D In the data, the distance between vertices R1 011a and R4 012b corresponds to the distance between vertices R1 011a and R4 012b using Figure 3C The edge weights in 220a show the same correlation level of 0.86. Similarly, in Figure 3D In the data, the distance between vertices R3 011c and R8 013b corresponds to the distance between vertices R3 011c and R8 013b using Figure 3C 0120c. The distance between vertices can be a function of the correlation level, with vertices that are closer to each other having a higher correlation than pairs of vertices that are farther apart from each other. A closer distance between pairs of vertices means a higher level of correlation. In this format, the closest vertices, such as data representation vertices R1 011a and R4 012b, can be divided into a group 017. In another example, data representation vertices R3 011c, R6 012c, and R8 013b can be divided into another group 0118. Vertices indicating data sources and / or data representations with the highest correlation can be classified in the same group.

[0136] Figure 3E Another embodiment provides a Figure 3C A simplified correlation graph derived from the graph of . Figure 3E In a graph of , there is a data source, but no data representation. Figure 3DSimilarly, the distances between (data source) vertices correspond to corresponding levels of relevance between these vertices. Figure 3E The distance 220s between the data source vertices DS1 010a and DS2 010b in corresponds to the level of correlation between these same data source vertices. Figure 3C In the example, the relevance level can be inferred from the relevance level of one or more of their data representations, for example, from the relevance level of the edge weight 1 220e between the data source vertex DS1 010a and its data representation vertex R1 011a, which also has a relevance level of 0.86 220a with the data representation vertex R4 012b, which also has a relevance level of 1 220q with its data source vertex DS2 010b. Figure 3E As shown, vertices indicating data sources DS1 010a and DS2 010b and having a shorter distance 220s (higher relevance level) between each other may be classified in the same group 019. Vertices indicating data sources DS1 010a and DS3 010c and having a longer distance 220u between each other may not be included in the same group, as their relevance level is not high enough. Similarly, vertices indicating data sources DS2 010b and DS3 010c and having a longer distance 220t between each other may not be included in the same group, as their relevance level is not high enough.

[0137] A dependency graph may only include data sources, even though their corresponding data representations are an important factor in assessing the relevance of data sources. A simplified graph may be stored to reduce storage space, and its dependency information may be retrieved more easily and / or faster when needed. A simplified graph may also help speed up computations and reduce complexity.

[0138] As mentioned above about Figure 2BAs described, the DRM module may send a relevance management request (e.g., message 260), also referred to as an evaluation request, to the CM module to request the CM module to evaluate the relevance of two or more data representations. Therefore, the DRM module interacts with the CM module to start the CM module to perform the detection and maintenance of the relevance between the different data that can be obtained from the data source set. These messages may include one or more parameters. The parameters may indicate one or more data representations, and these one or more data representations, for example, indicate that the relevance information (e.g., relevance level) of the indicated data representation will be evaluated. The parameters may include a relevance type, which indicates that the relevance information of the representation will be evaluated using a specified relevance type. The parameters may include an indication that the CM module will determine at least one relevance, for example, based on the indicated relevance type. The parameters may include an application category identifier, which, for example, indicates the application category to which the data representation belongs, or indicates that the CM module will determine the relevance information between the data representation and the specified application category, or a combination thereof. The parameters may include a data source identifier, which is used to identify one or more data sources that provide the data representation. The parameters may include one or more memory addresses of the raw data indexed by the data representation, such that based on the memory addresses, if the CM has appropriate access rights, the CM module may be used to obtain the raw data and then evaluate the data relevance information using a content-based evaluation method. That is, the parameters may indicate one or more computer memory addresses storing the raw data of the one or more data representations. The relevance between the different data obtainable from the set of data sources may be indicated or further evaluated by the relevance of the data representations.

[0139] In various embodiments, the DSD module is used to receive requests from data consumers and assist the data consumers in determining data sources that contain the required data. If necessary, the DSD module can also select, configure and activate one or more appropriate DCP instances, controllers, points or combinations thereof to perform data acquisition, data transmission and (if necessary) data preprocessing. The request from the data consumer can be used to request data. Accordingly, the DSD module can determine the data requested by the data consumer. The determination can be made based on information in the request and at least one data representation. Determining the data can include collecting raw data, or collecting and processing raw data.

[0140] Figure 4 , Figure 5A and Figure 5BThe operation of the DSD module 500 provided by an embodiment of the present invention is shown. In various embodiments, the DSD module is used to facilitate the selection of (e.g., the best) data source 010 (e.g., data source instances 010a, 010b, 010c). Doing so can help data consumers discover and select (e.g., the most suitable) data sources and data items for a given task. This can include, for example, taking into account the representation correlation information (e.g., obtained from the CM module through the correlation library 450), data access overhead (e.g., Figure 4 In the case of the data consumer 511) and the requirements of the data consumer (as transmitted from the data consumer 030 via (e.g., data discovery) request 031), discover and select a data source that accommodates relevant data sets required by multiple parties of the data consumer or multiple consumers. For example, the requirements of the data consumer may include data dependencies required by multiple parties of the data consumer, or application categories of data to be used by the data consumer.

[0141] In various embodiments, the DSD module is used to perform data query statement generation. A data query statement is generated to indicate the type of raw data to be collected from a specified data source. The DSD module can generate a query plan as a result of interacting with a data consumer component. The query plan can indicate the data to be collected and the members of the (one or more) data sources from which the data is to be collected. The DSD module can determine the data sources and data representations involved to meet the requirements of the data consumer. The DSD module can determine which raw data types to collect and which data sources to collect these types of raw data from. Based on these determinations, the DSD module can generate a query statement and trigger a data query request including the query statement to the DCP module.

[0142] In various embodiments, the DSD module is used to determine the data collection and (if necessary) data preprocessing rules to be implemented. The data preprocessing rules may indicate how to obtain the target data to be provided to the data consumer from the raw data (collected from the data source). For example, one or more data preprocessing rules may specify how to modify and clean up data that is not directly available, and how to filter out irrelevant or redundant data.

[0143] For example, data preprocessing rules may include one or more rules for stitching together data that is not directly available. The rules may specify operations for finding intermediate data items to be used to stitch (merge) together data that is not directly available. This operation may be performed, for example, when a data consumer requires a composite data set that cannot be obtained from any single data source. Thus, data from two, three, or more data sources may be merged together to form combined data. Rules for filtering, cleaning, and normalizing data may be similarly determined.

[0144] In various embodiments, the DSD module is used to select one or more DCP module instances to call to perform data acquisition and (when necessary) data preprocessing. The selection can be based on one or more of the following: data access overhead, communication link status between the data source and the data consumer, location of the DCP module and the data consumer, DCP module capabilities (e.g., category IDs supported by the DCP module, data transfer or computational load of the DCP module), mobility of the data consumer, and mobility of the DCP module. The DSD module can be informed of the capabilities of the DCP module through a registration operation or through notifications from the DCP module instance. The selection can be based at least in part on the data acquisition action to be performed.

[0145] The DSD module may also be used to configure and activate the selected DCP module (eg, a selected instance thereof) to perform data acquisition and data preprocessing actions. This may be accomplished, for example, via one or more configuration instruction messages.

[0146] Figure 4 , Figure 5A and Figure 5B The operation of the DAM and its modules provided by the embodiment of the present invention is shown. Figure 4 , Figure 5A and Figure 5B As shown, the DSD module 500 can interact with the data consumer 030 by receiving (e.g., data discovery) request 031 from the data consumer 030. The DSD module 500 can also process the content of the request 031, which can indicate an application category identifier, which indicates the category for which the result of the data collection action can be used. The request 031 can indicate one or more characteristics (e.g., data representation) of the data required by the data consumer component 030. The request 031 can indicate the number of (e.g., related) data sets required by the data consumer component 030. The number of data sets can be, for example, the number of different data records indexed by different IDs. The request 031 can indicate the dependencies of the data sets required by multiple parties (e.g., using AI model type indications such as VFL or HFL, or in terms of data ID and feature alignment) and the required correlation between these data sets. The request 031 can indicate the address of (e.g., one or more) devices of the data consumer, which are designated to receive the results of a set of data collection actions. The request 031 can indicate one or more final target data requirements (e.g., data format, data structure) of the results of a set of data collection actions to be sent to the data consumer. Based on these parameters, the DSD module can configure the DCP module to perform data acquisition actions.

[0147] The DSD module 500 may send a correlation information request 551 to the CM module 400 (e.g., via a message). The correlation information request 551 may specify one or more types of correlations requested by the data consumer component 030 (e.g., via request 031). The correlation information request 551 may include a request to the CM module 400 to identify the specified one or more types of correlations between members of different data available from a data source (e.g., a set of networked data sources) 030. The correlation information request 551 may include an application category identifier indicating an application to be associated with the desired correlation information. The correlation information request 551 may include a representation of one of the representations of the data source associated with the desired correlation information. The correlation information request 551 may include a correlation type associated with the desired correlation information.

[0148] CM module 400 may send a relevance information response 552 to DSD module 500, which may include some or all of the corresponding data relevance information requested by relevance information request 551. CM module 400 may obtain the data relevance information from or through relevance repository 450, for example.

[0149] The DSD module 500 can select the most appropriate data based on at least some of the following: data relevance information received from the CM module 400 via the relevance information response 552, the data access overhead 511, and the data requirements of the data consumer component 030 (e.g., as specified by the (e.g., data discovery) request 031).

[0150] The DSD module 500 may check (514) whether the (raw) data may be used by the data consumer component 030. If so, no data pre-processing (516a) is required, and the DSD module 500 may generate a corresponding data query statement 517a. If not, the raw data may be pre-processed (516b) so that it may be used by the data consumer component 030, in which case the DSD module 500 may generate a (possibly different) corresponding data query statement 517b, and request data pre-processing (518), e.g., by constructing pre-processing rules 520, which may be used by the DCP module 300 to perform the data pre-processing. The DSD module 500 may configure the DCP module 300 to perform a set of data acquisition actions based at least in part on the (e.g., data discovery) request 031 received from the data consumer component 030.

[0151] Based on the set of data collection actions, the DSD module 500 may configure the DCP module 300 to select (560a) appropriate (one or more) DCP module instances to perform the set of data collection actions, which may include (if necessary) data preprocessing. The DSD module 500 may configure (560b) the (one or more) DCP module instances to perform one or more data collection actions in the set of data collection actions.

[0152] The DSD module 500 may send a data collection and (if necessary) pre-processing request (message) 518 to the DCP module 300 (e.g., via the DCP controller 310) to activate the DCP module instance to collect and (if necessary) pre-process data. The data collection and (if necessary) pre-processing request 518 may include one or more configuration parameters, as described below.

[0153] The configuration parameters may include an identifier of a DCP point device (i.e., a DCP module instance) to be configured and activated. The configuration parameters may include an indication of whether the acquired data needs to be preprocessed. The configuration parameters may include an indication of the raw data to be collected from a specified member of a set of (e.g., networked) data sources. A data query statement (e.g., 517a or 517b) may be used to indicate which raw data should be collected from which data sources, such as indicating (one or more) specific data representations of the representation of the data to be collected, indicating a data source address of the address of the data source providing the data. For example, a data query statement may have the format of [(representation #1, data source #A); (representation #2, data source #B); ...].

[0154] Configuration parameters may include the address of one (or each) data consumer component device to which the DCP module is to forward output. Configuration parameters may include an indication of the requirements of the final target data to be sent to the data consumer component, such as the type of data (e.g., data format), the (optional) number of data sets, and the (optional) dependencies of the data sets (e.g., in terms of ID or feature alignment). Configuration parameters may include an indication of one or more rules (e.g., pre-processing rules 520) to be applied by data pre-processing the acquired (raw) data.

[0155] The preprocessing rules may indicate how to preprocess the acquired data to obtain the final target data. Preprocessing may include (and the associated rules may indicate) one or more of the following: merging the acquired data (e.g., concatenating); filtering the acquired data (e.g., removing useless or redundant data); cleaning the acquired data, normalizing the acquired data, or both; specifying one or more indications of a portion of the acquired data to which the associated (one or more) preprocessing rules are to be applied; specifying one or more conditions that trigger the implementation of the associated (one or more) preprocessing rules.

[0156] The preprocessing rules may indicate one or more conditions (e.g., a sequence of steps, timestamps, trigger events, index data) to be satisfied before the DCP module (or one or more devices or instances thereof) can perform data preprocessing. An example of a preprocessing rule may be: [Step 1 (or indicated by a trigger event or timestamp): concatenate data from data sources A and B; Step 2 (or indicated by a trigger event or timestamp): filter redundant data indexed by representations in the new data acquired in step 1; Step 3 (or indicated by a trigger event or timestamp): clean up the new data acquired in step 2; ...].

[0157] The configuration parameters may include one or more indications of data dependencies between different representations involved in the representation of the data source, the indications being used for pre-processing the acquired data.

[0158] In various embodiments, each of the one or more devices or instances of the DCP module may be operable to provide its capability indication to the DSD module (e.g., via the DSD module). Such capability indication may be used by the DSD module to configure the DCP module based at least in part on the capability indication.

[0159] In various embodiments, the DCP module 300 (or one of its devices or instances) may be unable to perform data acquisition and (if necessary) pre-processing for any of a variety of reasons (e.g., when the DCP module fails to successfully acquire target data, or the DCP module denies a request from the DSD module, e.g., due to data communication overload or computational overload 531a). In this case, the DCP module 300 (e.g., via the DCP controller 310) may send a fault response (message) 531 to the DSD module 500 indicating the fault. In response to receiving the fault response 531, the DSD module 500 may begin to repeat (532) the previously described data acquisition actions to find suitable data and / or another available DCP module instance to perform (one or more) data acquisition actions.

[0160] For example, Fig. 6AAs shown, participant A (of the data consumer component) may require target data 523 (represented by feature 1 524a, feature 2 524b, and feature 3 524c of ID1 541, ID2 542, and ID3 543), for example, for horizontal federated learning (HFL). The DSD module may discover first data 010e from a first data source and second data 010f from a second data source, which together (at least) constitute the required data. Additionally or alternatively, the data discovery may be performed by a DCP module (or one or more of its devices, points, or instances), which may be configured by the DSD module to obtain data, which may include pre-processing the data as required by participant A, if necessary. Neither the first data source nor the second data source can provide the required target data 523 alone. However, the first original data 010e of the first data source and the second original data 010f from the second data source may be pre-processed to obtain the required target data 523. The DSD module may configure one or more data preprocessing rules for execution by the DCP module (or one of its devices or instances) to preprocess the first data 010e and the second data 010f that are not directly available by combining them together. This combination produces partial data A 547a (represented by feature 1 524a of ID1 541, ID2 542, and ID3 543), all of which are part of the desired target data 523. This combination also produces partial data B 547b (represented by feature 2 524b of ID1 541, ID2 542, and ID3 543), all of which are two repetitions of a part of the desired target data 523 (one from data 010e and one from data 010f). Since the data in partial data B 547b is repeated twice, the preprocessing may include deleting the duplicate data and retaining only one copy to be included in the desired target data 523. This combination also produces partial data C 547c (represented by feature 3 524c of ID1 541, ID2 542, and ID3 543), all of which are part of the desired target data 523. This combination also produces partial data D 547d (represented by feature 2 524b and feature 3 524c of ID4 544, ID5 545, and ID6 546), none of which is part of the desired target data 523. Since the data in partial data D 547d is not required for the target data 523, it can be deleted during preprocessing.

[0161] For example, Figure 6BAs shown, the correlation graph 520b may include information about several data sources and one or more representations thereof, such as: a representation of location data 014a, cell data 014b, and timestamp data 014c that may be provided by data source #1 010g; a representation of traffic data 015a, location data 015b, tracking area (TA) data 015c, and subscription permanent identifier (SUPI) data 015d that may be provided by data source #2 010h; a representation of quality of service (QoS) data 016a and time data 016b that may be provided by data source #3 010i. Some of the representations may be correlated. For example, location data 015b may be correlated with location data 014a, with a correlation value of 0.95 220g; TA data 015c may be correlated with cell data 014b, with a correlation value of 0.92 220h; and timestamp data 014c may be correlated with time data 016b, with a correlation value of 0.92 220i.

[0162] In an exemplary embodiment, the data consumer component may require target data 525 including flow data 015a and QoS data 016a. The DSD module may configure the DCP module (or one of its devices or instances) to obtain the required target data 525 using the correlation graph 520b. The flow data 015a may be obtained from data source #2 010h, and the QoS data 016a may be obtained from data source #3 010i, which do not have a direct correlation according to the correlation graph 520b. However, as described above, the known correlations represented by the data sources may be used to combine their corresponding data to obtain the required target data 525.

[0163] Continuing with the above example, the desired traffic data 015a may be correlated with the desired QoS data 016a using the correlation graph 520b as follows: the traffic data 015a may be combined (e.g., concatenated) with the location data 015b because both are representations of the same data source #2 010h; the location data 015b may be combined with the location data 014a of the data source #1 010g because they have a relatively high correlation value of 0.95 220g (combining the location data 015b with the location data 014a is preferred over combining the TA data 015c with the cell data 014b because the former has a higher correlation level of 0.95 220g than the latter's correlation level of 0.92 220h); the location data 014a may be combined with the timestamp data 014c because both are representations of the same data source #1 010g; the timestamp data 014c may be combined with the time data 016b of the data source #3 010i because they have a relatively high correlation value of 0.95 220i; Finally, the time data 016b can be combined with the QoS data 016a since both are representations of the same data source #3 010i.

[0164] The combined raw data from all the above combinations can then be processed by the DCP module (or one of its devices or instances) configured accordingly by the DSD module, for example as described above with reference to Fig. 6A The target data is pre-processed in the manner described above to obtain the required target data 525.

[0165] The functions of the DCP module may include at least one of the following: acquiring data from one or more data sources; (if necessary) preprocessing the data according to the data preprocessing rules configured by the DSD module; and transmitting the required target data to the data consumer component.

[0166] like Figure 4 , Figure 5A and Figure 5BAs shown, the DCP module 300 may include a DCP controller 310 and one or more DCP points (also referred to as DCP module devices or DCP module instances) 320, such as a first DCP point 321a and a second DCP point 321b. The one or more DCP points 320 include a DCP point anchor 322. A DCP point anchor is a DCP point that transfers desired target data to a data consumer component, such as in a service flow. Other non-anchor DCP points typically do not interact directly with data consumer components. Other (one or more) DCP points may send collected and pre-processed data to the DCP point anchor, which then transfers the desired target data to the data consumer component. The DCP point anchor may be the only entity used to transfer the desired target data to the data consumer component. The use of the DCP point anchor may at least help reduce the complexity of target data transfer to the data consumer component, potentially help with unified scheduling and management or transmission of target data, or a combination thereof.

[0167] DCP points 320 respond to configuration instructions from DCP controller 310. The configuration instructions cause or configure DCP points 301 to collectively perform a set of data collection actions.

[0168] The DCP controller 310 may be deployed in a control plane (CP) 021, and the DCP point 320 may be deployed in a user plane (UP) 022 or a data plane. Several DCP points may work together to serve a specific data consumer component. As the DSD module configures the DCP module, one of the DCP points 320 is used as a DCP point anchor point 322 to provide a set of data collection action results to one or more devices of the data consumer component. Therefore, the DCP module is used to provide the data consumer with the results of the data collection action.

[0169] like Figure 4 , Figure 5A and Figure 5B As further shown in FIG. 5 , the DCP controller 310 may receive a data collection and (if necessary) data pre-processing request 518 from the DSD module 500. The data collection and (if necessary) pre-processing request 518 may indicate the data to be collected, the data source to which the data may correspond, and the desired target data.

[0170] Based on the data collection and (if necessary) pre-processing request 518, the DCP controller 310 may also configure (e.g., determine and optimize) pre-processing rules 520 (for the DSD module itself). This may occur if the pre-processing rules 520 were not included in the data collection and (if necessary) pre-processing request 518, or if the included pre-processing rules 520 need to be optimized to minimize communication and / or computational overhead. Accordingly, the DCP controller or another DCP device may determine (e.g., optimize) resource scheduling to support data collection, pre-processing, or both.

[0171] In various embodiments, when configuring a DCP point (e.g., Figure 4 321 in the DCP controller), the DCP controller may select one or more specific DCP points from the plurality of DCP points, and select a DCP point anchor point to transmit the desired target data to the data consumer component. For example, such selection may be based on one or more of the following: DCP point location; mobility of the data consumer component (e.g., to ensure that the coverage area of ​​the selected DCP point anchor point includes mobile or stationary data consumer components, thereby reducing data switching or exchange); minimizing data access overhead (e.g., data acquisition overhead, data transmission overhead); one or more communication link states (e.g., selecting a DCP point as an anchor point that has established or has a potentially better (e.g., more reliable, stronger, faster) communication link with the data source and / or data consumer component, for example, to increase data exchange speed and / or minimize link establishment delay and overhead); capabilities of a specific DCP point (e.g., the DCP point has the required pre-processing capabilities, sufficient computing resources, lower load); and combinations thereof.

[0172] In various embodiments, when configuring a DCP point (e.g. Figure 4 321 in the example, the DCP controller 310 may perform optimization of resource scheduling to support the execution of a set of data collection actions. As a result of such optimization, the DCP controller may configure a communication and data preprocessing execution strategy that enables a specific one or more DCP points to collect data from corresponding data sources while minimizing operation overhead (e.g., bandwidth and latency).

[0173] In various embodiments, when configuring a DCP point (e.g., Figure 4 321 in the DCP controller, the DCP controller can configure and activate specific one or more DCP points and pass data collection and preprocessing requirements (messages) (e.g., Figure 4 and Figure 5B Assign (one or more) separate tasks for setting data acquisition and preprocessing tasks to specific DCP points) as shown in 518a in FIG.

[0174] In various embodiments, the data collection and preprocessing request (message) can cause at least one DCP point device to perform one or more data collection tasks, data preprocessing tasks, or both. These tasks can be configured based on the content of the data collection and preprocessing request (message).

[0175] In various embodiments, the data collection and pre-processing request (message) may include one or more parameters described below.

[0176] The parameters may include an identifier of one of the DCP point devices (instances) to be configured and activated.

[0177] These parameters may include an indication of whether data pre-processing is required.

[0178] These parameters may include a data query statement indicating the type of raw data to be collected from a specified data source in a set of (e.g., networked) data sources, and may, for example, have the format of [(representation #1, data source #A); (representation #2, data source #B); ...], which indicates that data indexed by representation 1 is collected from data source #A, data indexed by representation 2 is collected from data source #B, and so on.

[0179] The parameters may include an address of a device to which the at least one of the DCP point devices is to forward the output. If the forwarding DCP point device is not a DCP point anchor point, the device may be another DCP point device. If the forwarding DCP point device is a DCP point anchor point, the device may be a data consumer component device.

[0180] These parameters may include an indication of requirements for final target data to be sent to the data consumer component.

[0181] These parameters may include data preprocessing rules, indicating how to preprocess the collected raw data to obtain the final or required target data. These data preprocessing rules may include one or all of the following: one or more specific preprocessing actions (e.g., data concatenation, data filtering, data cleaning, etc.); required data representation (e.g., indexing the data to be preprocessed); (one or more) conditions that must be met before performing a given preprocessing action (e.g., sequence of steps, timestamp, triggering event). For example, these data preprocessing rules may be [Step 1 (or indicated by a triggering event or timestamp): concatenate data from data sources A and B; Step 2 (or indicated by a triggering event or timestamp): filter redundant data, which is indexed by the representation in the new data obtained in step 1; Step 3 (or indicated by a triggering event or timestamp): clean the new data obtained in step 2; ...].

[0182] In various embodiments, the data source may also be a DCP point device (instance).

[0183] An example of a data collection and preprocessing requirement (message) may be: [DCP point #1: collect data representing index 1 from data source A, preprocessing rule 1, timestamp / trigger event / step number 1, data receiver 1 (e.g., another DCP point); ...; DCP point anchor: collect data representing index n from data source N, preprocessing rule n, timestamp / trigger event / step number n, data receiver n (i.e., data consumer component)]. In other words, DCP point #1 needs to collect data representing index 1 from data source A, and DCP point #1 preprocesses the collected data using preprocessing rule 1 when timestamp / trigger event / step number 1 occurs, and then outputs the preprocessed data to another DCP point 1; similar or different instructions are used for one or more other DCP points...; DCP point anchor needs to collect data representing index n from data source N, and DCP point anchor preprocesses the collected data using preprocessing rule n when timestamp / trigger event / step number n occurs, and then transmits the final target data to receiver n, i.e., data consumer component.

[0184] like Figure 5B As further shown in FIG. 1 , the DCP controller 310 may send a data collection request to request each data source (e.g., the first data source 010a, the second data source 010b, the third data source 010c) to send data to a (e.g., specific) DCP point device (e.g., DCP point 321a or DCP point anchor 322). The data collection request may include one or more parameters, such as data representation, DCP point ID.

[0185] The DCP point and / or DCP point anchor collects raw data from one or more data sources and (if necessary) pre-processes the raw data into final (required) target data. The DCP anchor then sends the final (required) target data to the data consumer component.

[0186] According to the above embodiment, the DAM platform includes: a DRM function for obtaining a representation of a data source (for example, used as an index of raw data) to indicate the type of data that can be provided by different data sources; a CM function for managing data source relevance; a DSD function for selecting relevant data sets for multiple parties and determining data preprocessing rules; and a DCP module for performing data preprocessing of data that is not directly available, irrelevant or redundant.

[0187] In addition, based on the above, the DAM platform can determine the correlation between the available data accordingly and determine the data requested by the data consumer. The determination of the data requested by the data consumer can be performed based on the request itself and the acquired and calculated information (such as the representation of the data source and the determined correlation). Determining the data requested by the data consumer can also include generating a query plan and determining the requested data based on the execution of the query plan. Based on the request of the data consumer, the determined data can be provided to the data consumer by the DAM platform.

[0188] In various embodiments, possible features or advantages of the systems and methods described herein include one or more of the following: obtaining (e.g., capturing and managing) information about data sources and their corresponding data representations; creating and maintaining correlations of representations of data sources, and using the representations and the correlations to obtain (e.g., discover, collect) raw data; if the raw data is not directly usable by the data consumer component, further using the representations and the correlations to pre-process the raw data to obtain the required target data, and transmitting the required target data to the data consumer component, which can help improve data availability and reduce data collection overhead. Possible features or advantages include assisting the data consumer component in discovering and selecting the (e.g., most) appropriate data set in the following situations: when the data consumer component does not clearly define the required target data; and when multiple parties of the data consumer require multiple related required target data sets.

[0189] In some cases, multiple parties of a data consumer component may require multiple related target datasets. The required target datasets may not always be clearly defined by the data consumer component, and therefore, the DAM may receive limited information about the required target datasets.

[0190] Traditionally, multiple parties of data consumer components may require a target dataset for federated learning (FL). The FL computing platform can be provided, for example, by a service provider (e.g., NET4AI) that provides (network) connectivity and AI computing services. There are usually multiple FL participants (i.e., parties) and a collaborator in FL. Each FL participant uses a training dataset to train a sub-model. Although the training dataset of each FL participant cannot be made public to other FL participants or FL collaborators, each FL participant can send trained intermediate parameters (e.g., gradient values) to the FL collaborator. The FL collaborator can then aggregate all intermediate parameters received from multiple FL participants into one or more aggregated parameters, and then send the one or more aggregated parameters to each FL participant. Based on the one or more aggregated parameters received from the FL collaborator, each FL participant can update its local sub-model. The above process can be repeated until the sub-model training is completed.

[0191] FL can be classified into horizontal federated learning (HFL) and vertical federated learning (VFL). There may be significant limitations on the FL training datasets of multiple FL participants. For example, in HFL, HFL participants need target datasets with less data ID overlap and more data feature overlap; while in VFL, VFL participants need target datasets with more data ID overlap and less data feature overlap.

[0192] There may be many potential data sources for FL, however, the system is often unable to determine which potential data sources to combine while achieving the most dataset intersection (e.g., correlation) to obtain the best AI training performance. It may be technically impractical and time- and resource-consuming to have these systems try all possible combinations of all potential data sources to select the best data source that will provide the desired target dataset.

[0193] In FL, for the purpose of obtaining the desired target data, it may be beneficial not to treat multiple FL participants as independent, since their respective desired target datasets may be related. The desired target datasets are related (e.g., in terms of ID and feature alignment in VFL and HFL, respectively), rather than independent.

[0194] For example, Fig. 7A shows the HFL scenario, while Figure 7B A VFL scenario is shown. Fig. 7A , participant A 931 and participant B 932 need to have corresponding datasets 931a and 932a with overlapping features, i.e., each dataset 931a and 932a includes entries describing the same features 901, 902, 903. However, in this case, the overlap between the IDs is small or zero. For example, dataset 931a has a set of IDs 911, 912, 913 that are disjoint from the set of IDs 914, 915, 916 of dataset 932a (each ID identifies a different dataset entry).

[0195] On the contrary, Figure 7B , participant C 933 and participant D 934 need to have corresponding datasets 933a and 934a with overlapping IDs, i.e., each dataset 933a and 934a includes entries identified by the same set of IDs 911, 912, 913, 914, 915, 916. However, in this case, the overlap between features is small or zero. For example, dataset 933a has feature 901 that does not intersect with feature sets 902, 903 of dataset 934a.

[0196] In various embodiments of the present invention, the DAM can obtain (e.g., search, select, pre-process, transfer) the required target data, taking into account that the required target data sets may preferably be related rather than independent. For example, this can enable the DAM to satisfy target data constraints regarding data IDs, data features, or a combination thereof.

[0197] For example, refer to Figure 8 , the DAM 100 interacts with a controller 034 of an external system 040 by receiving queries 034a and providing responses 034b. The interaction can be performed as an interaction with a data consumer, as described elsewhere herein. It should be noted that a single controller 034 interacts with the DAM on behalf of multiple parties (i.e., FL participants 035a and 035b). As described elsewhere herein, the DAM 100 can cause data sources 010a, 010b to transmit (033a, 033b) data to FL participants 035a and 035b (usually through the DAM 100 or its modules, such as DCP modules). The FL participants 035a and 035b can use the transmitted data to participate in a federated learning exercise involving FL collaborators 036, which will be readily understood by those skilled in the art. It should be noted that the data transmitted (033a) by DAM100 (or coordinated by DAM 100) to the first FL participant 035a can be related to the data transmitted (033b) by DAM 100 (or coordinated by DAM100) to the second FL participant 035b, which is beneficial to federated learning.

[0198] Fig. 9 An exemplary implementation provided by an embodiment of the present invention is shown, wherein optimal target data sets are obtained for multiple parties of the data consumer components, and the required target data sets of the data consumer components are related.

[0199] refer to Fig. 9 , one or more data sources (e.g., the first data source 010a and the second data source 010b of the data source set 010) each send a representation management request 1141 to request the DRM module 200 to manage the data representation information of the data source. The representation management request 1141 may be similar to Figure 2B The DRM module 200 may include a registration message 241 for registering a given data source. The message may include a data representation, which may include one or more parameters. The parameters may include information indicating the given data source itself, information indicating the data that the given data source can provide, or both, as described elsewhere herein. The DRM module 200 may select the most useful representation for each of the one or more related application categories from the representation corpus.

[0200] Subsequently, the DRM module 200 sends a dependency management request 1160 to the CM module 400 to request the CM module to evaluate the dependency of the representations. The dependency management request 1160 may be associated with Figure 2B The correlation management request 1160 may be similar or identical to the correlation management request 260 of the present invention. The correlation management request 1160 may include one or more parameters, such as the parameters described elsewhere in this document regarding the correlation management request 260. In response to the request, the CM module 400 uses content evaluation, semantic-based evaluation, or by reading pre-configuration information as described elsewhere in this document to detect or evaluate the correlation of the data representation. Different types of correlations (e.g., equality / inequality, similarity / dissimilarity, inclusion / exclusion of dependencies, or transitive correlations) can be evaluated for different applications, services, or uses. The data correlation information generated by the CM module 400 is provided as the content of a correlation library that can be generated, maintained, and stored by the CM module. The original data correlation information library can be further simplified into a small-scale library to facilitate subsequent efficient data discovery and selection.

[0201] The data consumer 030 sends a data discovery request 1132 to the DSD module 500 to discover and select data suitable for a specific use. The data discovery request 1132 can be Figure 4 and Figure 5A 1132. In this embodiment, it is assumed that data consumer 030 requests data for multiple parties (e.g., party A 038a and party B 038b), wherein the required target data sets of the multiple parties are related. Data consumer 030 may send data discovery request 1132 and receive a response through controller 037. Data discovery request 1132 may include one or more parameters described below.

[0202] One or more parameters may include an application category identifier (ID) indicating the category for which the results of the data collection action are to be used.

[0203] The one or more parameters may include an indication of one or more characteristics of the target data required by the multiple parties of the data consumer. Such one or more characteristics may constitute a basic data representation (e.g., one or more data characteristics) of the required target data. In addition to these necessary data representations, the target data required by the multiple parties of the data consumer may also include other representations. However, the other representations cannot be known in advance by the data consumer and therefore cannot be provided to the DSD by the data consumer.

[0204] The one or more parameters may include a number of related data sets required by the data consumer (eg, a number of parties corresponding to the data consumer).

[0205] One or more parameters may include an indication of dependencies of target data sets required by multiple parties, and indicate the required correlation between target data sets required by multiple parties. For example, the dependency may be indicated by an AI model type (e.g., a VFL model or an HFL model) or a correlation type. For example, for a VFL model type, the dependency is that multiple data sets should have more data ID overlap and less data feature overlap; and for an HFL model type, the dependency is that multiple data sets should have less data ID overlap and more data feature overlap.

[0206] The one or more parameters may include an address of each device of a counterpart of the data consumer that is designated to receive results of the set of data collection actions.

[0207] The one or more parameters may include one or more final target data requirements (eg, data format, data structure) of the results of a set of data collection actions to be sent to the data consumer.

[0208] The one or more parameters may include a desired target data set size, which indicates an amount of data desired by a data consumer (eg, a particular party or parties).

[0209] The DSD module 500 may determine the required target data based on at least one or more of the above parameters. In various embodiments, the data consumer may provide the DSD module with multiple data features (not all) of the target data set required by multiple parties (i.e., multiple FL participants). For ease of explanation, this embodiment considers providing one data feature of the target data set required by multiple parties to the DSD module. For example, if the data consumer indicates that the dependency of the target data set is a VFL model type, the target data selected by the DSD module for the data consumer should consist of multiple data sets, where (ideally) all data sets have overlapping data IDs. And only one feature of the data features of the data set includes the provided data feature, and all other features of the data set do not include the provided data feature. In another example, if the data consumer indicates that the dependency of the target data set is a HFL model type, the target data selected by the DSD module for the data consumer should consist of multiple data sets, and (ideally) no data set has an overlapping ID, and all data features of the data set include the provided data feature.

[0210] The DSD module 500 then sends a relevance information request 1153 to the CM module 400. The relevance information request 1153 is used to request data relevance information and may include one or more parameters, such as those described elsewhere herein. The relevance information request 1153 may be associated with, for example, Figure 5A The described relevance information request 551 and, for example, Fig.11The described dependency information request 1151 is similar or identical.

[0211] CM module 400 then sends a correlation information response 1154 to DSD module 500. Correlation information response 1154 may be associated with, for example, Figure 5A The described correlation information response 552 or for example about Fig.11 The described relevance information response 1152 is similar or identical.

[0212] The relevance information response 1154 may include (or be followed by) data relevance information provided by the CM module in response to the relevance information request. Accordingly, the DSD module obtains the relevance information from the CM module. Based on the relevance information provided by the CM module (in addition to, for example, received data requirements, data access overhead, etc.), the DSD module may discover and select the most appropriate data source that can provide (at least) the relevant data required by multiple parties of the data consumer.

[0213] The DSD module 500 can determine the appropriate data source and data representation to meet the request of the data consumer. The DSD module can generate a data query statement and send the data query statement to the DCP module 300. The query statement can indicate which raw data should be collected from which data sources. For example, the DSD module can select a data source that can provide a data set with more data ID overlap and less data feature overlap for the VFL model type; and the DSD module can select a data source with less data ID overlap and more data feature overlap for the VFL model type.

[0214] like Fig.10As shown, if there are two FL participants (parties) (not shown) and the basic data feature indicated by the data consumer is location 014a, the DSD module can select data source #1 010j and data source #3 010m for VFL because data source #1 010j and data source #3 010m have data sets with more relevant data ID representations, such as ID 014d of data source #1 010j and C-RNTI 016c (a type of ID) of data source #3 010m, with a correlation level of 0.85 220p, and no relevant data feature representations. Similarly, the DSD module may select data source #1 010j and data source #2 010k for HFL because data source #1 010j and data source #2 010k have data sets with fewer relevant data ID representations, the only one being ID 014d of data source #1 010j and SUPI 015c (a type of ID) of data source #2 010k, which have a correlation level of 0 (zero) 220m, while having more relevant data feature representations, such as location 014a of data source #1 010j and location 015a of data source #2 010k, with a correlation level of 0.95 220j, and such as unit 014b of data source #1 010j and TA015b of data source #2010k, with a correlation level of 0.92 220k.

[0215] like Fig. 9 As further shown in the figure, the DSD module 500 sends a data collection and preprocessing request 1119 to the DCP module 300 (e.g., its DCP controller) to activate the DCP point device (i.e., instance) to collect data for multiple parties. For example, in this step of this embodiment, it is assumed that the original data can be used directly by the data consumer (its multiple parties) without preprocessing, but in other examples, preprocessing may be required, and may at least include the features of preprocessing as described elsewhere in this document. Before sending the data collection and preprocessing request 1119 to the DCP module, the DSD module may perform DCP (e.g., module point, device, instance, anchor point) selection, as described elsewhere in this document. The data collection and preprocessing request 1119 may include one or more parameters as described below. When the DCP module does not perform preprocessing, it may be referred to as a data collection and delivery (DCD) module.

[0216] The parameters of the data collection and preprocessing request 1119 may include indication information indicating that the data consumer (eg, multiple parties) may directly use the raw data and does not require preprocessing.

[0217] The parameters of the data collection and preprocessing request 1119 may include a data query statement, which includes the data to be collected and the corresponding appropriate data source (i.e., indicating which raw data should be collected from which data sources), including one or more of the following parameters: data representation (e.g., data features, data ID); data source address; data size. The format of the data query statement may be, for example: [(representation #1, data source #A, data size #C); (representation #2, data source #B, data size #D); ...].

[0218] The parameters of the data collection and pre-processing request 1119 may include corresponding addresses of multiple parties of the data consumer to receive the target data.

[0219] The parameters of the data collection and preprocessing request 1119 may include a mapping between multiple addresses of data sources and data consumers, indicating which data collected from a particular data source should be transmitted to which party of the data consumer.

[0220] Then return to reference Fig. 9 , the DSD module 500 sends to the data consumer 030 (eg, its controller 037) Data Now respond 1132b, to inform the data consumer that the data discovery and collection are successful. The data discovery response 1132b may include one or more parameters, for example, indicating one or more of the following: application category identifier; data representation; data size.

[0221] The DCP module 300 may establish a secure tunnel to collect data from the data source and transmit the data to the data consumer. The DCP module collects data from the data source and transmits the data to each of the data consumers through the secure tunnel. In various embodiments, the DCP module may only collect data related to the data representation indicated by the DSD module in the data collection and preprocessing request 1119.

[0222] The DCP module 300 may establish and configure secure tunnels to collect (1166a, 1166b) data from the first data source 010a and the second data source 010b, respectively, and transmit (1171a, 1171b) the data to the A side 038a and the B side 038b of the data consumer 030. The DCP module 300 collects data representing the index received in association with the data collection and preprocessing request 1119 from the data sources, and (if necessary) preprocesses the collected raw data based on preprocessing rules that may also be received in association with the data collection and preprocessing request 1119 or that may be constructed by the DCP module 300 itself, as described elsewhere herein.

[0223] Subsequently, the DCP module transmits the data from the first data source 010a and the second data source 010b respectively (1171a, 1171b) to the A side 038a and the B side 038b of the data consumer 030 through the secure tunnel.

[0224] In various embodiments of the present invention, the above embodiments may be used to help multiple data consumers discover and select appropriate data when multiple data consumers need related data sets based on the maintained knowledge of the data source (i.e., data source representation and relevance) and limited indication information received from the data consumer (e.g., some data features, data relevance types).

[0225] Fig.11 An exemplary implementation provided by an embodiment of the present invention is shown, in which data collection and data synthesis are performed. For this embodiment, it is assumed that the raw data provided by the data source cannot be directly used by the data consumer, so data preprocessing is performed. For example, the data consumer may need a composite data set that cannot be obtained from any single data source. In this case, the DAM can splice or merge the required data together based on information from multiple data sources. Accordingly, the DAM collects raw data from the data source, preprocesses the raw data, and then transmits the preprocessed data to the data consumer.

[0226] refer to Fig.11 , one or more data sources of set 010 (e.g., 010a and 010b) each send a management request 1141, which Indicates management request 1141 may include a request for the DRM module 200 to manage data representation information of a data source, and the DRM module receives the request. The representation management request 1141 may be similar to Figure 2B The DRM module 200 may include a registration message 241 for registering a data source. The message may include a data representation, which may include one or more parameters. The parameters may include information indicating the data source itself, information indicating the data that the data source can provide, or both, as described elsewhere herein. The DRM module 200 may select the most useful representation for each of the one or more relevant application categories from the representation corpus.

[0227] Subsequently, the DRM module 200 sends the CM module 400 Dependency Management Request 1160, to request the CM module to evaluate the relevance of the representation. The relevance management request 1160 can be used with Figure 2BThe correlation management request 1160 may be similar or identical to the correlation management request 260 of the present invention. The correlation management request 1160 may include one or more parameters, such as the parameters described elsewhere in this document regarding the correlation management request 260. In response to the request, the CM module 400 uses content evaluation, semantic-based evaluation, or by reading pre-configuration information as described elsewhere in this document to detect or evaluate the correlation of the data representation. Different types of correlations (e.g., equality / inequality, similarity / dissimilarity, inclusion / exclusion of dependencies or transitive correlations) can be evaluated for different applications, services, or uses. The data correlation information generated by the CM module 400 is provided as content of a correlation library that can be maintained and stored by the CM module. The CM module can generate content and store the content in the correlation library. The original data correlation information library can be further simplified into a small-scale library to facilitate subsequent efficient data discovery and selection.

[0228] The data consumer 030 sends a data discovery request 1131 to the DSD module 500 to discover and select data suitable for a specific use. The data discovery request 1131 can be used with Figure 4 and Figure 5A 038a). In this embodiment, for the sake of clarity, it is assumed that data consumer 030 is one party (e.g., party A 038a) requesting data. However, in other embodiments, data can be requested for multiple parties, such as party B 038b. Data consumer 030 can send data discovery request 1131 through controller 037 and receive a response.

[0229] The data discovery request 1131 may include one or more parameters, such as in conjunction with other Figure 4 and Figure 5A For example, the parameters of the data discovery request 1131 may include an application category identifier, a base data representation, addresses of the parties of the data consumer to whom the data is to be sent (e.g., the address of participant A 038a), the final target data requirement, and a desired data size indicating the amount of data to be provided.

[0230] For example, suppose party A 038a is Figure 8 Participant A 931 in the horizontal federated learning (HFL) scheme of A. In this case, the parameter indicating the basic data representation may indicate that data feature 1 901, data feature 2 902, and data feature 3 903 are the data required by party A 038a.

[0231] The DSD module 500 is used to determine the final target data based at least in part on the provided application ID and the basic data representation parameters in the data discovery request 1131.

[0232] The DSD module 500 then sends a relevance information request 1151 to the CM module 400. The relevance information request 1151 is used to request data relevance information and may include one or more parameters, such as described elsewhere herein. The relevance information request 1151 may be associated with, for example, a reference to Figure 5A The described dependency information request 551 is similar or identical.

[0233] CM module 400 then sends a correlation information response 1152 to DSD module 500. Correlation information response 1152 may be associated with, for example, Figure 5A The described correlation information response 1152 may be similar or identical to the correlation information response 552. The correlation information response 1152 may include (or be followed by) the data correlation information provided by the CM module in response to the correlation information request. Accordingly, the DSD module obtains the correlation information from the CM module.

[0234] Based on information such as relevance information, data requirements received in association with the data discovery request 1131, data access overhead, etc., the CM module 400 discovers and selects the most appropriate data source that can provide the data required by the data consumer. The DSD module determines the data sources and data representations involved to satisfy the data consumer's request. The DSD module then generates a data query statement, which is then sent to the DCP module. The query statement indicates which raw data to collect and from which data sources the raw data is collected. For example, referring to Fig. 6A , the DSD module can find a first data source (providing data 010e) and a second data source (providing data 010f) that can provide the required target data 523. Neither data 010e nor data 010f itself provides all the required target data 523. That is, no data source has a directly available data set that should have all the features of data features 1, feature 2, and feature 3. In addition, the data IDs of the two data sets 010e and 010f are not aligned. Therefore, in an embodiment, the DAM will preprocess the original data 010e and 010f into the required target data 523. This may involve merging (connecting) certain parts of the data entries together and deleting other parts of the data. This preprocessing forms a composite data set that is the data required by the data consumer or one of its parties. The DSD module can construct data preprocessing rules for execution by the DCP module, which modify the non-directly available data to form the required target data. This may include merging data together, filtering irrelevant or redundant data, or a combination thereof.

[0235] Return to reference Fig.11, the DSD module 500 then sends a data collection and pre-processing request 1118 to the DCP module 300 (e.g., to its DCP controller). The data collection and pre-processing request 1118 can be related to the reference Figure 5B The DCP module 500 may be similar or identical to the data collection and preprocessing request 518 described above. The data collection and preprocessing request 1118 may activate the selected DCP module instance to collect data and, if necessary, preprocess the collected data. Before or simultaneously with sending the data collection and preprocessing request 1118, the DCP module 500 may select the DCP module instance to perform data collection and preprocessing (if necessary).

[0236] The data collection and preprocessing request 1118 may include one or more parameters, such as reference Figure 5B described.

[0237] As about Fig. 6A For example, the parameters of the data query statement indicated in the data collection and preprocessing request 1118 may be in the format of [(feature #1 and feature #2, data source #1); (feature #2 and feature #3, data source #2); ...]. This indicates instructions for collecting data indexed by feature #1 and feature #2 from data source #1 and collecting data indexed by feature #2 and feature #3 from data source #2, etc.

[0238] For example, the parameters of the data preprocessing rules indicated in the data acquisition and preprocessing request 1118 may indicate that the data will be preprocessed in a certain manner. This manner may indicate the preprocessing operation, the representation (index of the data to be preprocessed), and the conditions (e.g., sequence of steps, timestamp, triggering event) under which the preprocessing operation is to be performed. This may indicate that when the specified conditions are met, the data preprocessing operation will be performed by the DCP module.

[0239] For example, for Fig. 6A , the preprocessing rules can be:

[0240] a. Operation 1 (or indicated by a trigger event or timestamp): stitching together the data provided by both the first data source and the second data source;

[0241] b. Operation 2 (or indicated by a trigger event or timestamp): filter (discard) data with IDs (e.g., ID4 544, ID5 545, ID6 546) that do not appear in either of the two data sets 010e and 010f;

[0242] c. Operation 3 (or indicated by a trigger event or timestamp): retain a copy of the redundant data (eg, data 547b) and other copies;

[0243] d. Operation 4 (or indicated by a trigger event or timestamp): clean the data or transcribe the data into a desired data format;

[0244] For example, regarding Figure 6B , data source #2 010h can provide data 015a about network traffic, data source #3 010i can provide data 016a about network QoS, and the final target data required by the data consumer can be data about network traffic and QoS. In addition, the data 015a about network traffic provided by data source #2 010h and the data 016a about network QoS provided by data source #3 010i can be connected through the related data provided by data source #1. Therefore, in order to generate the final target data, the pre-processing rule can be: splice (merge) the data from data source #1, data source #2 and data source #3 together, wherein the representations (QoS 016a and flow 015a) will be spliced, the relevant intermediate representations (time 016b, timestamp 014c, position 014a, position 015b) are used to splice the data, and the intermediate data source (data source #1 010g) is used to determine the relevant intermediate representations, so as a result of the splicing using the correlation, the start and end data sources (flow 015a of data source #3 010h and QoS 016a of data source #3 010i) can be connected, as shown. Figure 6B This is shown as line 525 connecting target traffic 015a and QoS 016a.

[0245] Subsequent reference Fig.11 , the DSD module 500 sends to the data consumer 030 Data Discovery Response 1131b, to inform the data consumer that the data discovery and collection are successful. The data discovery response 1131b may include one or more parameters, such as indicating an application category identifier, data representation, and data size.

[0246] The DCP module 300 may establish and configure a secure tunnel to collect (1165a, 1165b) data from the data sources 010a, 010b and transmit (1170) the data to the A party 038a of the data consumer 030. The DCP module 300 collects the data representing the index received in association with the data collection and preprocessing request 1118 from the data source, and preprocesses the collected raw data based on the preprocessing rules also received in association with the data collection and preprocessing request 1118 or the preprocessing rules constructed by the DCP module 300 itself.

[0247] Subsequently, the DCP module transmits (1170) the final target data to the A party 038 of the data consumer 030 through a secure tunnel.

[0248] Accordingly, although the data provided by the data source may not necessarily be directly used by the data consumer, the embodiments of the present invention can pre-process the data to make it usable. The DAM determines which data sources to collect data from and the data representation to be provided. The DAM (for example, interactively or autonomously) assists the data consumer in generating a data query statement including the data to be collected and the related data sources to indicate the raw data to be collected and the data source from which the raw data is to be collected. In addition, the DAM can determine and implement data pre-processing rules to convert data that is not directly available into usable data. This approach can reduce the data processing overhead of the data consumer and can help to better utilize the raw data to increase the data potential. The proposed DAM platform can be used to provide facilities or business models for companies or networks that sell data sets. The cost of the provided data set can be charged per access (only useful data items are exposed and accessed) instead of selling the entire data set.

[0249] Fig.12 1 is a schematic diagram of an electronic device 1200 that can perform any or all of the steps of the above methods and features described herein, provided by different embodiments of the present invention. For example, a physical machine or server or other computing device can be configured as an electronic device. The electronic device can be a single integrated device or a device formed by independent but networked components, such as in a data center, a collection of data centers, or other facilities. The apparatus for performing embodiments of the present invention may include one or more electronic devices (e.g., Fig.12 ) or part thereof.

[0250] As shown, device 1200 includes a processor 1210 (e.g., a central processing unit (CPU) or a special processor, such as a graphics processing unit (GPU), or other such processor unit), a memory 1220, a non-transitory mass storage 1230, an I / O interface 1240, a network interface 1250, and a transceiver 1260, all of which are communicatively coupled via a bidirectional bus 1270. According to certain embodiments, any or all of the elements described may be utilized, or only a subset of the elements described may be utilized. In addition, device 1200 may include multiple instances of certain elements, such as multiple processors, memories, or transceivers. In addition, elements of a hardware device may be directly coupled to other elements without the bidirectional bus.

[0251] The memory 1220 may include any type of non-transitory memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or any combination thereof. The mass storage element 1230 may include any type of non-transitory storage device, such as a solid state drive, a hard disk drive, a disk drive, an optical drive, a USB drive, or any computer program product for storing data and machine executable program code. According to some embodiments, the memory 1220 or the mass storage 1230 may record statements and instructions thereon, which may be executed by the processor 1210 to perform any of the above-mentioned method operations.

[0252] Electronic devices configured according to the present invention may include hardware, software, firmware, or a combination thereof. Examples of hardware are computer processors, signal processors, ASICs, FPGAs, silicon photonic chips, and the like. The hardware may be electronic hardware, photonic hardware, or a combination thereof. An electronic device may be considered a computer because it performs operations corresponding to calculations, such as receiving and processing data, receiving and processing instructions, generating and storing data, providing outputs such as instructions, queries, or reports, or a combination thereof. Therefore, various techniques that are readily understood by those skilled in the art may be used to provide an electronic device. An electronic device may include a computer operably coupled to a memory (e.g., a non-transitory electronic memory). The memory may store computer program instructions that, when executed, cause the computer to perform operations as described herein.

[0253] It should be understood that although specific embodiments of the technology are described herein for illustrative purposes, various modifications may be made without departing from the scope of the technology. Accordingly, the specification and drawings are to be regarded only as an illustration of the invention as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations or equivalents within the scope of the invention. Specifically, the following is within the scope of the present technology: providing a computer program product or program element or program memory or storage device (e.g., magnetic or optical, tape or disk), etc., for storing machine-readable signals for controlling the operation of a computer according to the method of the present technology and / or constructing some or all of its components according to the system of the present technology.

[0254] The actions associated with the methods described herein may be implemented as coded instructions in a single computer program product. In other words, the computer program product is a computer readable medium on which software code is recorded to perform the methods when the computer program product is loaded into a memory and executed on a microprocessor of the wireless communication device. The computer readable medium may be non-transient because the information is not included in a transient propagation signal.

[0255] The actions associated with the methods described herein may be implemented as coded instructions in a plurality of computer program products. For example, a first portion of the method may be performed by one computing device, and a second portion of the method may be performed by another computing device, a server, etc. In this case, each computer program product is a computer-readable medium on which software code is recorded to perform the appropriate portion of the method when the computer program product is loaded into a memory and executed on a microprocessor of a computing device.

[0256] In addition, each step of the method can be executed on any computing device (e.g., a personal computer, a server, etc.), and according to one or more program elements, modules or objects generated from any programming language (e.g., C++, Java, etc.), or a portion of one or more program elements, modules or objects. In addition, each step or a file or object that implements each of the steps, etc., can be executed by dedicated hardware or a circuit module designed for this purpose.

[0257] Through the description of the above embodiments, the present invention can be implemented only by hardware, or by software and necessary general hardware platforms. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product.

[0258] It will be readily appreciated that throughout the foregoing discussion, the network functions and operations described above may correspond to methods for supporting the operation of a communication network (e.g., a 5G or 6G wireless communication network). The method may involve computer-implemented functions, i.e., functions implemented by one or more computing, communication, or memory components of a network infrastructure, or a combination thereof. These components may take various forms, such as specific servers or general-purpose computing, communication, or memory devices, or a combination thereof, which are used to provide the required functionality through virtualization technology. The method may involve the operation of one or more network components in order to improve the operation of the network. Therefore, in the case where the communication network is considered as a device, embodiments of the present invention may be directed to improving the internal operation of the communication network.

[0259] In addition, it will be readily understood that embodiments of the present invention relate to communication network systems or related devices thereof for performing the above-mentioned network functions and operations. Again, the system or device may include one or more computing, communication or memory components of the network infrastructure, or a combination thereof, which may take various forms, such as a specific server or a general-purpose computing, communication or memory device, or a combination thereof, which is used to provide the required functionality through virtualization technology. The various methods disclosed herein may be implemented on one or more real or virtual computing devices, such as devices within a communication network control plane, devices operating in a data plane, or a combination thereof. A computing device for implementing the method operations may include a processor operably coupled to a memory that provides instructions executed by the processor to perform the method as described herein.

[0260] Various embodiments of the present invention use one or both of real computer resources and virtual computer resources. Such computer resources utilize a set of one or more microprocessors at the hardware level, which are operably coupled to a corresponding set of memory components, and the memory components include stored program instructions executed by the microprocessors. Computing resources can be used to provide virtual computing resources at one or more virtualization levels. For example, one or more given general-purpose computer hardware platforms can be used to provide one or more virtual computers. Computer hardware (such as processor resources and memory, etc.) can also be virtualized to provide resources from which other virtual computers are built. A set of computing resources that can be allocated to provide various computing resources, which are in turn used to implement various computing components of the system, can be regarded as providing a distributed computing system, and its internal architecture can be configured in various ways.

[0261] It should be noted above that functions and modules can be given different names and instantiated in different ways. A given function can be merged or integrated with one or more other functions. A given function can be provided by multiple separate functional elements in collaboration.

[0262] Although the present invention has been described with reference to the specific features and embodiments of the present invention, it is apparent that various modifications and combinations may be made to the present invention without departing from the present invention. Accordingly, the specification and the accompanying drawings are only regarded as an explanation of the present invention as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention.

Claims

1. A networked computerized system, characterized in that: include: a data representation manager (DRM) module for obtaining at least one representation of one or more data sources, each representation corresponding to one of the one or more data sources, the representation being used to represent a characteristic of the data source; a data collection and preprocessing (DCP) module for interacting with members of the one or more data sources to obtain data from the members of the one or more data sources according to a set of data collection actions; a correlation manager (CM) module for determining at least one correlation between different data obtainable from the one or more data sources, the at least one correlation being based at least in part on the at least one representation; as well as Data Source Discovery (DSD) module for: interacting with a data consumer component and the CM module to determine the set of data collection actions to be performed to support a request from the data consumer; and The DCP module is configured to perform the set of data collection actions.

2. The system according to claim 1, characterized in that The set of data collection actions includes: Collecting raw data from the one or more data sources; or Raw data is collected from the one or more data sources, and the raw data collected from the one or more data sources is preprocessed.

3. The system according to claim 2, characterized in that The preprocessing includes implementing one or more of the following: merging the raw data; filtering the raw data; cleaning the raw data; and The raw data are normalized.

4. The system according to claim 1, characterized in that The at least one representation includes one or more of the following: information indicating the data source; and Information indicating the data that the data source can provide.

5. The system according to claim 4, characterized in that The information indicating the data source includes one or more of the following: a data source type; a category of applications to which data from the data source can be used; a location of the data source; The services that the data source can provide; capabilities of the data source; and context information of the data source.

6. The system according to claim 4, characterized in that The information indicating the data that the data source can provide includes one or more of the following: metadata; data name; data title; data tag; keyword; data semantics; data attribute; Data characteristics; The data schema; the data format; and the class of applications for which the data may be used.

7. The system according to any one of claims 1 to 6, characterized in that The DRM module is configured to generate the at least one representation at least in part by: requesting a report from the data source; and receiving and processing the report to obtain the at least one representation.

8. The system according to claim 7, characterized in that Requesting the report includes sending representation configuration information to the data source, the representation configuration information indicating a representation of data that the data source is to provide to the DRM module.

9. The system according to any one of claims 1 to 6, characterized in that The DRM module is used to receive data from the data source and process the received data into the at least one representation.

10. The system according to claim 9, characterized in that The DRM module is further configured to send the at least one representation and the received data from the one or more data sources to the CM module, wherein the CM module evaluates at least one correlation between parts of the received data by parsing the received data and the at least one representation.

11. The system according to any one of claims 1 to 6, characterized in that The DRM module is configured to receive the at least one representation.

12. The system according to any one of claims 1 to 11, characterized in that The at least one correlation between different data retrievable from the one or more data sources is an indication of coherence between two or more members of the different data.

13. The system according to claim 12, characterized in that The coherence reflects one or more of the following: the degree of equality or inequality; the degree of similarity or dissimilarity; This includes the degree of dependency or exclusion; and the degree of transitive dependencies.

14. The system according to claim 1, characterized in that The DRM module is used to interact with the CM module to start the CM module to perform the step of determining at least one correlation between the different data obtainable from the one or more data sources.

15. The system according to claim 14, characterized in that The interacting with the CM module includes: the DRM module sending an evaluation request to the CM module.

16. The system according to claim 15, characterized in that The evaluation request includes one or more of the following: one or more representations of the at least one representation; a correlation type to be used for said determining at least one correlation; an indication that the CM module is to evaluate dependency information for the one or more representations of the at least one representation; The CM module is to perform the instruction to determine at least one correlation based on the correlation type; an application category identifier, indicating an application category to which the at least one representation belongs, or indicating that the CM module is to determine at least one correlation between the at least one representation and the application category, or a combination of these two indications; a data source identifier identifying one or more members of the one or more data sources providing one or more representations of an association in the at least one representation; as well as One or more computer memory addresses storing raw data for the one or more representations of the at least one representation.

17. The system according to any one of claims 1 to 16, characterized in that The interaction with the data consumer component includes: A query plan is generated that indicates data to be collected and members of the one or more data sources from which the data is to be collected.

18. The system according to any one of claims 1 to 17, characterized in that Configuring the DCP module to perform the set of data collection actions includes: selecting one or more DCP module instances from a plurality of DCP module instances based on the set of data collection actions, and configuring the selected one or more DCP module instances from the plurality of DCP module instances.

19. The system according to any one of claims 1 to 18, characterized in that The DCP module is also used to provide the results of the set of data collection actions to the data consumer.

20. The system according to any one of claims 1 to 19, characterized in that The interacting with the data consumer component includes receiving and processing content of the request from the data consumer component, the content including parameters indicating one or more of the following: an application category identifier indicating a category to which the result of the data collection action is to be applied; an indication of one or more characteristics of the data desired by the data consumer; the number of relevant data sets required by the data consumer; an indication of dependencies of data sets required by the multiple parties, and an indication of required dependencies between said data sets required by the multiple parties; an address of a device of the data consumer, the device being designated to receive the results of the set of data collection actions; and One or more final target data requirements as a result of the set of data collection actions to be sent to the data consumer.

21. The system according to claim 20, characterized in that The method is also used to execute the configuration of the DCP module to perform the set of data collection actions based at least in part on the parameters.

22. The system according to any one of claims 1 to 21, characterized in that The DCP module includes a DCP controller and a plurality of DCP point devices. The DCP point devices respond to configuration instructions of the DCP controller. The configuration instructions enable the DCP point devices to jointly perform the set of data collection actions.

23. The system according to claim 22, characterized in that The DCP controller is deployed in a control plane of a network, and the DCP point device is deployed in a user plane or a data plane of the network.

24. The system according to claim 22, characterized in that Due to the configuration of the DCP module, one of the DCP point devices is used to act as an anchor device to provide results of the set of data collection actions to one or more devices of the data consumers.

25. The system according to claim 22, characterized in that The DCP controller is configured to perform one or more of the following: determining or optimizing one or more rules for preprocessing the acquired data; selecting one of the DCP point devices to perform the set of data collection actions; Configure and activate one of the DCP point devices; as well as Resource scheduling is optimized to support execution of the set of data collection actions.

26. The system according to claim 22, characterized in that The DCP controller is used to send a data collection and preprocessing requirement message to at least one of the DCP point devices, and the data collection and preprocessing requirement message enables the at least one of the DCP point devices to perform one or more data collection tasks, data preprocessing tasks, or both, and the tasks are configured based on the content of the data collection and preprocessing requirement message.

27. The system according to claim 26, characterized in that The data collection and preprocessing and request message parameters include one or more of the following: an identifier of one of the DCP point devices to be configured and activated; An indication of whether data preprocessing is required; A data query statement indicating the type of raw data to be collected from a specified one of the networked data sources set; an address of a device to which the at least one of the DCP point devices is to forward output; Requirements for final destination data to be sent to said data consumer; as well as Data preprocessing rules indicate how to preprocess the collected raw data.

28. The system according to any one of claims 1 to 27, characterized in that The DCP module includes one or more devices, each device being configured to provide a capability indication of each device to the DSD module, the DSD module performing the configuration of the DCP module based in part on the capability indication.

29. The system according to any one of claims 1 to 28, characterized in that The interaction with the CM module includes: sending a correlation information request to the CM module, the correlation information request specifying one or more types of correlations, and the correlation information request is a request to the CM module to identify the specified one or more types of correlations between members of the different data that can be obtained from the set of networked data sources.

30. The system according to claim 29, characterized in that The dependency information request includes one or more parameters for specifying the required dependency information, the one or more parameters including one or more of the following: an application category identifier indicating an application to be associated with the required correlation information; a representation of one of said representations of a data source associated with said desired relevance information; as well as A dependency type associated with the desired dependency information.

31. A system according to any one of claims 1 to 30, characterized in that Configuring the DCP module to perform the set of data collection actions includes: the DSD module providing one or more configuration parameters to the DCP module, the one or more configuration parameters including one or more of the following: The identifier of the DCP point device to be configured and activated; an indication of whether to pre-process the acquired data; an indication of raw data to be collected from a specified member of the set of networked data sources; The address of the device to which the DCP module is to forward the output; an indication of a requirement for final target data to be sent to said data consumer; an indication of one or more rules to be applied by said pre-processing of said acquired data; and An indication of one or more data dependencies between different representations involved in the representation of the data source, the indication being used for pre-processing the acquired data.

32. The system according to claim 31, characterized in that The indication of one or more rules to be applied by the pre-processing of the acquired data is used to indicate one or more of the following: one or more rules to be used to merge the acquired data; one or more rules to be used to filter the acquired data; one or more rules to be used to clean the acquired data, normalize the acquired data, or both; one or more indications of portions of the acquired data to which an association rule of the one or more rules is to be applied; as well as One or more conditions that trigger implementation of an associated rule among the one or more rules.

33. A data representation manager (DRM) networked computerized device, characterized in that Used for: Obtain at least one representation of one or more data sources, each representation corresponding to a respective data source of the one or more data sources, the representation being used to represent a characteristic of the data source; as well as Interacting with one or more other devices to support determining a correlation between different data available from the one or more data sources, the correlation based at least in part on the at least one representation of the data source.

34. A data collection and preprocessing (DCP) networked computerized device, characterized in that: Used for: interacting with one or more other devices to configure the DCP device to perform a set of data collection actions; as well as Interact with members of a set of one or more data sources to obtain data from the set of one or more data sources according to the set of data collection actions.

35. The system according to claim 34, characterized in that The DCP module is also used to pre-process the acquired data according to the set of data acquisition actions.

36. A Correlation Manager (CM) networked computerized device, characterized in that Used for: determining at least one correlation between different data retrievable from a set of networked data sources, the at least one correlation being based at least in part on at least one retrieved representation of a data source belonging to the set of networked data sources; as well as Interact with one or more other devices to determine a set of data collection actions to be performed to support a request from a data consumer.

37. A data source discovery (DSD) networked computerized device, characterized in that Used for: interacting with a data consumer component and one or more other devices to determine a set of data collection actions to be performed to support a request from a data consumer; and Another device is configured to perform the set of data collection actions.

38. A data management method, characterized in that: include: The data management module obtains at least one representation of one or more data sources, each representation corresponding to one of the one or more data sources, the representation being used to represent a characteristic of the data source; The data management module receives a request from a data consumer, the request being for requesting data; and The data management module determines the data requested by the data consumer based on the request information and the at least one representation.

39. The method according to claim 38, characterized in that The determining the data requested by the data consumer according to the request information and the at least one representation comprises: Collecting raw data from the one or more data sources according to the request information and the at least one representation; or According to the request information and the at least one representation, raw data is collected from the one or more data sources, and the raw data collected from the one or more data sources is pre-processed.

40. The method according to claim 39, characterized in that The preprocessing includes implementing one or more of the following: merging the raw data; filtering the raw data; cleaning the raw data; and The raw data are normalized.

41. The method according to claim 39 or 40, characterized in that The method further comprises: determining one or more rules for pre-processing the raw data; or Resource scheduling is determined to support the execution of the raw data acquisition or the raw data preprocessing.

42. The method according to claim 41, characterized in that The one or more rules include at least one of the following: one or more rules to be used to merge the raw data; one or more rules to be used to filter the raw data; one or more rules to be used to clean the raw data; one or more rules to be used to normalize the raw data; one or more indications of portions of the original data to which an association rule of the one or more rules is to be applied; as well as One or more conditions that trigger implementation of an associated rule among the one or more rules.

43. The method according to any one of claims 38 to 42, characterized in that The at least one representation includes one or more of the following: information indicating the data source; and Information indicating the data that the data source can provide.

44. The method according to claim 43, characterized in that The information indicating the data source includes one or more of the following: a data source type; a category of applications for which data from the data source can be used; a location of the data source; The services that the data source can provide; capabilities of the data source; and context information of the data source.

45. The method according to claim 43, characterized in that The information indicating the data that the data source can provide includes one or more of the following: metadata; data name; data title; data tag; keyword; data semantics; data attribute; Data characteristics; The data schema; the data format; and the class of applications for which the data may be used.

46. ​​The method according to any one of claims 38 to 45, characterized in that The data management module obtains at least one representation of one or more data sources including: requesting a report from the data source; and The report is received and processed to obtain the at least one representation.

47. The method according to claim 46, characterized in that Requesting the report includes sending representation configuration information to the data source, the representation configuration information indicating a representation of data that the data source is to provide to the DRM module.

48. The method according to any one of claims 38 to 45, characterized in that The data management module obtains at least one representation of one or more data sources including: Data is received from the data source and the received data is processed into the at least one representation.

49. The method according to claim 48, characterized in that The data management module determines at least one correlation between portions of the received data by parsing the received data from the one or more data sources and the at least one representation.

50. The method according to any one of claims 38 to 45, characterized in that The data management module obtains at least one representation of one or more data sources including: The data management module receives at least one representation of the one or more data sources.

51. The method according to any one of claims 38 to 50, characterized in that Also includes: The data management module determines at least one correlation between different data available from the one or more data sources, the at least one correlation being based at least in part on the at least one representation of a data source; as well as The data management module determines, according to the request and the at least one representation, that the data requested by the data consumer comprises: The data management module determines the data requested by the data consumer based on the request, the at least one representation, and the at least one correlation.

52. The method according to claim 51, characterized in that The at least one correlation between different data retrievable from the one or more data sources is an indication of coherence between two or more members of the different data.

53. The method according to claim 52, characterized in that The coherence reflects one or more of the following: the degree of equality or inequality; the degree of similarity or dissimilarity; This includes the degree of dependency or exclusion; and the degree of transitive dependencies.

54. The method according to any one of claims 38 to 53, characterized in that The data management module determines, according to the request and the at least one representation, that the data requested by the data consumer comprises: The data management module generates a query plan indicating data to be collected and members of the one or more data sources from which the data is to be collected; and The data management module determines the data requested by the data consumer according to the query plan.

55. The method according to any one of claims 38 to 54, characterized in that Also includes: The data management module provides the data requested by the data consumer to the data consumer.

56. The method according to any one of claims 38 to 55, characterized in that The request from the data consumer includes parameters indicating one or more of the following: an application category identifier indicating a category to which the result of the data collection action is to be applied; an indication of one or more characteristics of the data desired by the data consumer; the number of relevant data sets required by the data consumer; an indication of dependencies of data sets required by the multiple parties, and an indication of required dependencies between said data sets required by the multiple parties; an address of a device of the data consumer, the device being designated to receive the results of the set of data collection actions; as well as One or more final target data requirements as a result of the set of data collection actions to be sent to the data consumer.

Images