Feedback on the performance of the training data

Abstract

Example embodiments of the present disclosure relate to feedback on performance of training data. In a solution, a terminal device sends format information to a first network device, the format information indicating a format for feedback on performance of a data set, the data set being collected by the terminal device, the data set being used to train an artificial intelligence / machine learning, AI / ML, model. The terminal device then sends the data set to the first network device. The terminal device receives feedback information from the first network device on performance of the data set, the feedback information having the indicated format.

Description

Technical Field

[0001] Various exemplary embodiments of this disclosure generally relate to the telecommunications field, and more specifically to methods, apparatuses, devices, and computer-readable storage media for providing feedback on the performance of training data. Background Technology

[0002] A communication network can be used as a facility to enable communication between two or more communication devices or to provide communication device access to a data network. A mobile or wireless communication network is an example of a communication network. Communication devices can be served by an application server.

[0003] Communication networks can operate according to standards provided by organizations such as the 3rd Generation Partnership Project (3GPP) or the European Telecommunications Standards Institute (ETSI). Examples of standards provided by 3GPP are the so-called 3GPP standards for cellular technology generations, such as those for 4G, 5G, and 6G technologies. Summary of the Invention

[0004] In a first aspect of this disclosure, a terminal device is provided. The terminal device includes at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the terminal device to at least: send format information to a first network device, the format information indicating a format for feedback on the performance of a dataset, the dataset being collected by the terminal device and used for training an artificial intelligence / machine learning (AI / ML) model; send the dataset to the first network device; and receive feedback information from the first network device regarding the performance of the dataset, the feedback information having the indicated format. In a second aspect of this disclosure, a first network device is provided. The first network device includes at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the first network device to at least: receive format information from a terminal device, the format information indicating a format for feedback on the performance of a dataset, the dataset being collected by the terminal device and used to train an artificial intelligence / machine learning (AI / ML) model; receive the dataset from the terminal device; and send feedback information to the terminal device regarding the performance of the dataset, the feedback information having the indicated format.

[0005] In a third aspect of this disclosure, a method is provided. The method includes: sending format information to a first network device, the format information indicating a format for providing feedback on the performance of a dataset, the dataset being collected by a terminal device and used to train an artificial intelligence / machine learning (AI / ML) model; sending the dataset to the first network device; and receiving feedback information from the first network device regarding the performance of the dataset, the feedback information having the indicated format.

[0006] In a fourth aspect of this disclosure, a method is provided. The method includes: receiving format information from a terminal device, the format information indicating a format for providing feedback on the performance of a dataset, the dataset being collected by the terminal device and used to train an artificial intelligence / machine learning (AI / ML) model; receiving the dataset from the terminal device; and sending feedback information about the performance of the dataset to the terminal device, the feedback information having the indicated format.

[0007] In a fifth aspect of this disclosure, a terminal device is provided. The terminal device includes: components for sending format information to a first network device, the format information indicating a format for providing feedback on the performance of a dataset, the dataset being collected by the terminal device and used to train an artificial intelligence / machine learning (AI / ML) model; components for sending the dataset to the first network device; and components for receiving feedback information from the first network device regarding the performance of the dataset, the feedback information having the indicated format.

[0008] In a sixth aspect of this disclosure, a first network device is provided. The first network device includes: components for receiving format information from a terminal device, the format information indicating a format for providing feedback on the performance of a dataset, the dataset being collected by the terminal device and used to train an artificial intelligence / machine learning (AI / ML) model; components for receiving the dataset from the terminal device; and components for sending feedback information about the performance of the dataset to the terminal device, the feedback information having the indicated format.

[0009] In a seventh aspect of this disclosure, a computer-readable medium is provided. The computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to a third aspect.

[0010] In an eighth aspect of this disclosure, a computer-readable medium is provided. The computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the fourth aspect.

[0011] It should be understood that the summary portion is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] Some exemplary embodiments will now be described with reference to the accompanying drawings, in which: Figure 1 An example communication environment in which example embodiments of this disclosure may be implemented is shown; Figure 2 Signaling flows are shown for example procedures for providing feedback on the performance of training data, according to some example embodiments of this disclosure; Figure 3The signaling flow for another example process of providing feedback on the performance of training data, according to some example embodiments of this disclosure, is shown; Figure 4 An example signaling flow is shown as another example process for providing feedback on the performance of training data according to some example embodiments of this disclosure; Figure 5 A flowchart is shown illustrating a method implemented at a terminal device according to some example embodiments of the present disclosure; Figure 6 A flowchart is shown illustrating a method implemented at a first network device according to some example embodiments of the present disclosure; Figure 7 A flowchart is shown illustrating a method implemented at a second network device according to some example embodiments of the present disclosure; Figure 8 A flowchart is shown illustrating a method implemented at a terminal device according to some example embodiments of the present disclosure; Figure 9 A flowchart is shown illustrating a method implemented at a first network device according to some example embodiments of the present disclosure; Figure 10 A simplified block diagram of a device suitable for implementing example embodiments of the present disclosure is shown; and Figure 11 A block diagram of an example computer-readable medium according to some example embodiments of the present disclosure is shown.

[0013] Throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0014] The principles of this disclosure will now be described with reference to some exemplary embodiments. It should be understood that these embodiments are described for illustrative purposes only and to assist those skilled in the art in understanding and implementing this disclosure, without imposing any limitation on the scope of this disclosure. The embodiments described herein can be implemented in various ways other than those described below.

[0015] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0016] References to "an embodiment," "an embodiment," "an example embodiment," etc., in this disclosure indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment needs to include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, whether explicitly stated or not, it is assumed that its influence on such feature, structure, or characteristic in conjunction with other embodiments is within the knowledge of those skilled in the art.

[0017] It should be understood that although various elements may be described herein using prefixes such as “first,” “second,” etc., these elements should not be limited by these terms. These terms are used only to distinguish one element from another, and they do not restrict the order of the terms. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0018] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements is connected by “and” or “or”, means at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0019] As used herein, unless explicitly stated otherwise, the “response to A” action does not indicate that the action is performed immediately after “A” occurs, and may include one or more intermediate steps.

[0020] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms “comprising,” “including,” “having,” “having,” “including,” and / or “comprising” as used herein specify the presence of the stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0021] As used in this application, the term "circuit system" may refer to one or more of the following: (a) Hardware circuit implementation only (such as implementation in analog and / or digital circuits only), and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of analog and / or digital hardware circuitry with software / firmware, and (ii) Any part of a hardware processor having software (including digital signal processor(s) working together to enable a device (such as a mobile phone or server) to perform various functions), software, and memory), and (c) One or more hardware circuits and / or one or more processors, such as one or more microprocessors or a portion thereof, that require software (e.g., firmware) to operate, but the software may not be present when operation is not required.

[0022] This definition of "circuit" applies to all uses of the term in this application, including in any claim. As another example, as used herein, the term "circuit" also encompasses only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware implementation. For example, and if applicable to a particular claim element, the term "circuit" also encompasses a baseband integrated circuit or processor integrated circuit used in mobile devices or servers, cellular network devices, or other computing or networking devices.

[0023] As used herein, the term "communication network" refers to a network that conforms to any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), LTE-A Advanced (LTE-A), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. Furthermore, communication between terminal devices and network devices in a communication network can be performed according to any suitable generated communication protocol, including but not limited to first-generation (1G), second-generation (2G), 2.5G, 2.75G, third-generation (3G), fourth-generation (4G), 4.5G, fifth-generation (5G), 5.5G, sixth-generation (6G) communication protocols and / or any other currently known or future-developed protocols. Embodiments of this disclosure can be applied to a variety of communication systems. Given the rapid development in communications, there will naturally also be future types of communication technologies and systems that can implement this disclosure. The scope of this disclosure should not be limited to the aforementioned systems only.

[0024] As used herein, the term "network device" refers to a node in a communications network through which terminal devices access the network and receive services. Network devices can refer to base stations (BS) or access points (APs), such as Node B (NodeB or NB), evolved Node B (eNodeB or eNB), NR NB (also known as gNB), Remote Radio Unit (RRU), Radio Head (RH), Remote Radio Head (RRH), repeater, Integrated Access and Backhaul (IAB) node, low-power node (such as femtoseconds, picoseconds), non-terrestrial network (NTN) or non-terrestrial network equipment (such as satellite network equipment, low Earth orbit (LEO) satellites, and geostationary Earth orbit (GEO) satellites), spacecraft network equipment, etc., depending on the terminology and technology applied. In some example embodiments, the Radio Access Network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) at the IAB donor node. An IAB node includes a mobile terminal (IAB-MT) portion that behaves like a UE toward its parent node, and a DU portion that behaves like a base station toward the next-hop IAB node.

[0025] The term "terminal device" refers to any terminal device capable of wireless communication. As an example and not a limitation, a terminal device may also be referred to as a communication device, user equipment (UE), subscriber station (SS), portable subscriber station, mobile station (MS), or access terminal (AT). Terminal devices can include, but are not limited to, mobile phones, cellular phones, smartphones, Voice over IP (VoIP) phones, wireless local loop phones, tablets, wearable terminal devices, personal digital assistants (PDAs), portable computers, desktop computers, image capture terminal devices (such as digital cameras), gaming terminal devices, music storage and playback devices, in-vehicle wireless terminal devices, wireless endpoints, mobile stations, laptop embedded devices (LEEs), laptop devices (LMEs), USB dongles, smart devices, wireless customer premises equipment (CPEs), Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. The terminal device may also correspond to the mobile terminal (MT) portion of an IAB node (e.g., a relay node). In the following description, the terms "terminal device," "communication device," "terminal," "user equipment," and "UE" are used interchangeably.

[0026] As used herein, the terms “resource,” “transmission resource,” “resource block,” “physical resource block” (PRB), “uplink resource,” or “downlink resource” can refer to any resource used to perform communication, such as communication between a terminal device and a network device, including resources in the time domain, frequency domain, spatial domain, code domain, or any other combination of time, frequency, spatial, and / or code domain resources used to implement communication. In the following, unless explicitly stated otherwise, resources in both the frequency and time domains will be used as examples of transmission resources used to describe some exemplary embodiments of this disclosure. Note that the exemplary embodiments of this disclosure are equally applicable to other resources in other domains.

[0027] The core network functions described herein can be implemented as core network entities comprising a combination of hardware processing circuitry and software and / or firmware including machine-readable instructions, or software including machine-readable instructions executable by at least one processor of the hardware processing circuitry. The hardware processing circuitry includes at least one processor and at least one memory storing machine-readable instructions executable by at least one processor of the hardware processing circuitry. The processor includes any one or a combination of an accelerator, a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, a central processing unit, a graphics processing unit, and a tensor processing unit. The memory includes any one or a combination of volatile or non-volatile memory (e.g., flash memory, cache, random access memory (RAM), and / or read-only memory (ROM)). The memory stores machine-readable instructions for the software and / or firmware to be executed by at least one processor of the hardware processing circuitry. The machine-readable instructions are executable by at least one processor of the hardware processing circuitry, causing the hardware processing circuitry to perform the actions or operations of the methods described herein. For example, the session management function described in this paper can be implemented as a session management entity, and the session management policy control function described in this paper can be implemented as a session management policy control entity.

[0028] As used herein, the term "artificial intelligence / machine learning (AI / ML) model" can refer to a data-driven algorithm that applies AI / ML techniques to generate a set of outputs based on a set of inputs. In the context of this disclosure, the term "AI / ML model" may be used interchangeably with the terms "model," "AI model," and "ML model."

[0029] Figure 1An example communication environment 100 in which exemplary embodiments of the present disclosure may be implemented is illustrated. Communication environment 100 includes multiple terminal devices, such as terminal device 110-1 and terminal device 110-2 shown, which may be collectively referred to as terminal device 110 or individually as terminal device 110. Terminal device 110 may collect data for training one or more AI / ML models. As used herein, the term "training" for AI / ML models may refer to any type of training of the AI / ML model, including but not limited to pre-training, initial training, fine-tuning, updating, etc.

[0030] The communication environment 100 also includes a first network device 120, which can be used to collect training data from the terminal device 110. The first network device 120 may be, or may be included in, a Data Collection Entity (DCE) authorized to collect data from the terminal device 110. In some example embodiments, the first network device 120 may be, or may be included in a Radio Access Network (RAN) node or device, such as a base station, like a gNB. In some example embodiments, the first network device 120 may be, or may be included in a core network (CN) function. For example, a Network Data Analysis Function (NWDAF) is a core network function that collects data from the terminal device 110 via an intermediate application function (AF).

[0031] The first network device 120 can aggregate data from different terminal devices and forward the aggregated data for training AI / ML models. In some example embodiments, the communication environment 100 may include a second network device 130, which can be used to at least train AI / ML models. In such example embodiments, the first network device 120 can forward training data to the second network device 130. For example, the second network device 130 may be, or can be included in, a Management Service (MnS) producer for training AI / ML models, also referred to as a training MnS producer.

[0032] It should be understood that Figure 1 The number of devices and their connections shown are for illustrative purposes only and are not intended to be limiting. Communication environment 100 may include any suitable number of devices configured to implement the exemplary embodiments of this disclosure. Although not shown, it should be understood that one or more additional devices may be located in communication environment 100. Furthermore, in some exemplary embodiments, the first network device 120 may be used to collect training data and train AI / ML models.

[0033] In the following description, for illustrative purposes, some example embodiments are described in which terminal device 110 operates as a UE, first network device 120 operates as a DCE, and second network device 130 operates as an MnS producer. However, in some example embodiments, the operations described in connection with the terminal device can be implemented at the network device or other devices, and the operations described in connection with the network device can be implemented at the terminal device or other devices.

[0034] Communication in communication environment 100 can be implemented according to any suitable communication protocol, including but not limited to cellular communication protocols, wireless local area network communication protocols (such as IEEE 802.11), and / or any other currently known or future-developed protocols. Furthermore, communication can utilize any suitable wireless communication technology, including but not limited to: Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Frequency Division Duplex (FDD), Time Division Duplex (TDD), Multiple Input Multiple Output (MIMO), Orthogonal Frequency Division Multiple Access (OFDM), Discrete Fourier Transform Extended OFDM (DFT-s-OFDM), and / or any other currently known or future-developed technologies.

[0035] With the increasing prevalence of AI applications, there may be scenarios where data collected by end devices (e.g., UE) will be used to train AI / ML models. Typically, data collection entities can collect data from end devices, simply aggregate the data, and forward it to MnS producers for use in training AI / ML models.

[0036] For effective model training, it is essential to ensure that the data is sufficient and contains adequate features and variations. However, when data is collected from multiple UEs and aggregated, there is a high probability that data from some UEs may not have reached the required label.

[0037] Based on some example embodiments of this disclosure, a solution is proposed for determining the performance of data collected by a terminal device and feeding the performance back to the terminal device.

[0038] Using the solutions in the example embodiments of this disclosure, the terminal device can receive feedback information and learn about the performance of the collected data. Based on the feedback information, the terminal device can update its data collection strategy so that the data collected for training AI / ML models can be optimized.

[0039] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0040] Figure 2 Signaling flow 200 is shown as an example procedure for providing feedback on the performance of training data, according to some example embodiments of this disclosure. For example... Figure 2As shown, signaling flow 200 involves terminal device 110, first network device 120, and second network device 130. For discussion purposes, reference will be made to... Figure 1 Discuss signaling flow 200.

[0041] like Figure 2 As shown, terminal device 110 sends (206) a first dataset to first network device 120. The first dataset may be collected by terminal device 110 for training AI / ML models. Although the singular form is used for "AI / ML model", it should be understood that in some example embodiments, the first dataset may be used to train more than one AI / ML model.

[0042] The first dataset can be sent proactively by the terminal device 110 or upon request by the first network device 120. In some example embodiments, the first dataset can be sent proactively. For example, once the terminal device 110 has collected data, it can proactively send (206) the collected data (i.e., the first dataset) to the first network device 120. Alternatively, the terminal device 110 can proactively send (206) the collected data to the first network device 120 within a predefined or configured time period.

[0043] Alternatively or additionally, in some example embodiments, the first network device 120 may send (202) a request for training data to the terminal device 110. For example, the request may include information and / or features desired by the first network device 120. The terminal device 110 may receive (204) the request and, in response, send (206) a first dataset to the first network device. Additionally, in some example embodiments, the terminal device 110 may report its availability and capability in collecting and sharing data.

[0044] The first network device 120 receives (208) a first dataset from the terminal device 110. Then, the first network device 120 determines (210) the performance of the first dataset. For example, the performance of the first dataset can indicate whether the first dataset meets the requirements for model training.

[0045] In some example embodiments, the performance of a dataset may include the quality of the dataset. Quality can be evaluated from the perspective of the data in the dataset. Alternatively or additionally, the performance of a dataset may include the effectiveness of the dataset. Effectiveness can be evaluated from the perspective of the training results achievable by the dataset. For example, the effectiveness of a dataset can represent the effectiveness of training an AI / ML model. Furthermore, it should be understood that evaluation metrics other than or in addition to quality and effectiveness are possible.

[0046] In some example embodiments, the first network device 120 may determine multiple attributes of the first dataset based on data items included in the first dataset. These multiple attributes may represent factors that influence model training. For example, the multiple attributes may include missing data points, corrupted data points, etc. It should be understood that the fewer missing / corrupted data points, the better the first dataset.

[0047] Then, the first network device 120 can determine a score representing the performance of the first dataset based on an algorithm and multiple attributes. The score can be an absolute metric or a relative metric.

[0048] In some example implementations, the score can be an absolute metric determined by an algorithm and multiple attributes of a first dataset. The algorithm can be a function, a polynomial, a machine learning model trained for dataset evaluation, etc. As an example, the algorithm could be a cost function as follows.

[0049] (1) Where f can represent the first score of the dataset, which is the absolute measure of the dataset; m and c can represent the attributes of the dataset, and α and β can represent the parameters of the function, such as the corresponding weights of different attributes. For example, m can refer to missing data points, and c can refer to corrupted data points in the dataset. α and β can represent the weights of m and c, respectively. It can be assumed that α, β ≥ 1, α + β = 1. In some examples, the weights α and β can depend on the training results at the second network device 130 and can be changed dynamically. According to the function (1) above, the lower the value of f, the better the dataset.

[0050] It should be understood that the above functions are for illustrative purposes only and are not intended to impose any limitations.

[0051] Alternatively or additionally, in some example embodiments, the score may be a relative metric determined based on absolute metrics of multiple datasets and the corresponding absolute metrics. Considering that the terminal device 110 may not understand absolute metrics, the first network device 120 may obtain a second score, which is a relative metric, to represent the performance of the first dataset. That is, the score may be a relative goodness score of the UE dataset (RGSUD).

[0052] Specifically, in some example embodiments, the first network device 120 can receive training data from multiple terminal devices to obtain multiple datasets including a first dataset. Then, the first network device 120 can determine the corresponding absolute metrics of the multiple datasets, for example, based on the function (1) described above. For example, the first network device 120 can obtain the first dataset's... The second dataset and the third dataset Therefore, the first network device 120 can determine the relative measure of the first dataset based on the absolute measure of the first dataset and the corresponding absolute measures of multiple datasets. For example, the first network device 120 can determine... , and , Then, the second score can be determined based on the following equation.

[0053] (2) Where s can represent the second rating of the dataset, which is a relative measure. f can represent the absolute measure of the dataset. It can represent the largest absolute measure among multiple absolute measures of multiple datasets. It can represent the smallest absolute measure among multiple absolute measures of multiple datasets.

[0054] For example, if f=15, =10, =20, then the second score s can be 5.

[0055] In some example embodiments, the second rating representing the performance of the first dataset can be mapped to a scale (or in other words, a rating range) to make it easier for the terminal device 110 to understand. For example, if only the second rating "5" is transmitted to the terminal device 110, it may be meaningless. If scales [1-10] are provided simultaneously, it allows for more sensing for the terminal device 110, i.e., in scales [1-10], the second rating is 5, which is the average.

[0056] In some example embodiments, information associated with determining dataset performance may be provided by the second network device 130. For example... Figure 2 As shown, the second network device 130 can send (220) information related to the performance determination of the dataset for the AI / ML model to the first network device 120. The first network device 120 can receive (222) the information and determine (210) the performance based on the information.

[0057] In some example embodiments, the information associated with dataset performance determination may include an algorithm for dataset performance determination, such as a function to be used. Alternatively or supplementary, the information may include multiple dataset attributes to be considered in dataset performance determination, such as missing data points, corrupted data points, etc. As another alternative or supplementary method, the information may include one or more parameters to be used in the algorithm, such as the weights of missing data points, the weights of corrupted data points, etc. continue Figure 2The first network device 120 sends (212) feedback information about the performance of the first dataset to the terminal device 110. Accordingly, the terminal device 110 receives (214) the feedback information from the first network device 120.

[0058] In some example embodiments, the feedback information may include a score representing the performance of the first dataset. For example, a first score and / or a second score may be included in the feedback information. Alternatively or additionally, the feedback information may include a scale associated with the scoring scale, such as scales [1-10]. By providing the scale, the terminal device 110 can more easily understand the score. Alternatively or additionally, the feedback information may include an algorithm for determining the performance of the first dataset. Based on this algorithm, the terminal device 110 can obtain the reasons behind the score. Alternatively or additionally, the feedback information may include one or more attributes of the first dataset that degrade the performance of the first dataset. For example, if the first dataset is missing a relatively large number of data points, the feedback information may include an indication of missing data points. As another example, if the first dataset includes a relatively large number of corrupted data points, the feedback information may include an indication of corrupted data points. Alternatively or additionally, the feedback information may include one or more suggestions for improving the performance of the dataset. The terminal device 110 may optimize future datasets based on one or more suggestions.

[0059] It should be understood that the above feedback information is for illustrative purposes only and is not intended to impose any limitations. Based on the feedback information, the terminal device may be able to optimize its data collection strategy.

[0060] Feedback information can have any suitable format. In some example embodiments, feedback information can be a structured information object. That is, feedback information can be encapsulated in a specific format (e.g., JSON format) known to the first network device 120 and the terminal device 110. The first network device 120 can generate feedback information based on the format of the structured information object. Therefore, the terminal device 110 can determine the performance of the first dataset from the feedback information based on the format of the structured information object.

[0061] Alternatively or additionally, in some example embodiments, the feedback information may consist of unstructured information objects without a specific format. The first network device 120 can generate the feedback information by generating these unstructured information objects. Therefore, the terminal device can determine the performance of the first dataset by processing the unstructured information objects.

[0062] In some example embodiments, since processing unstructured information objects may require Natural Language Processing (NLP) capabilities, terminal device 110 may send an instruction to first network device 120 indicating its ability to process information expressed in natural language. First network device 120 may receive the instruction and generate an unstructured information object. For example, for a terminal device indicating NLP capabilities to first network device 120, first network device 120 may generate feedback information in the format of an unstructured information object.

[0063] Additionally, in some example embodiments, the first network device 120 may send an instruction to the terminal device 110 regarding the format of the feedback information. The terminal device 110 may receive the instruction from the first network device 120 and then interpret the feedback information based on the indicated format.

[0064] continue Figure 2 Terminal device 110 can update (216) the data collection strategy for the AI / ML model based on feedback information. By understanding the performance of the first dataset and the reasons behind it, terminal device 110 can optimize the data collection strategy. For example, if the feedback information indicates that the first dataset has a defect of missing data points, terminal device 110 can collect data by sampling more densely in the time domain.

[0065] Then, terminal device 110 can collect another dataset for the AI / ML model based on the updated data collection strategy. This updated data collection strategy can improve the performance of the additional dataset.

[0066] In some example embodiments, the first network device 120 may send (224) multiple datasets for model training to the second network device 130. For example, the first network device 120 may select datasets with good performance and send these datasets to the second network device 130.

[0067] Figure 3 Signaling flow 300 is shown as another example process for providing feedback on the performance of training data, according to some example embodiments of this disclosure. For example... Figure 3 As shown, signaling flow 300 involves terminal device 110 and first network device 120. For discussion purposes, reference will be made to... Figure 1 Discuss signaling flow 200. In signaling flow 330, terminal device 110 and first network device 120 interact with each other regarding the format of feedback information.

[0068] like Figure 3As shown, terminal device 110 sends (306) format information to first network device 120, the format information indicating the format used for feedback on the performance of the dataset. The dataset may be collected by terminal device 110 for training AI / ML models. First network device 120 receives (308) format information from terminal device 110. Based on this format information, it may notify first network device 120 of the preferred or supported format of feedback information by terminal device 110.

[0069] In some examples, the indicated format may include structured information objects and / or unstructured information objects. Structured information objects may refer to information objects with a specific format (e.g., JASON format). Unstructured information objects may refer to information objects that do not have a fixed structure or format, such as text, images, or information expressed in natural language. In some example embodiments, since processing unstructured information objects may require NLP capabilities, terminal device 110 can determine whether it has the capability to process information expressed in natural language. If terminal device 110 has the capability to process information expressed in natural language, it can determine the indicated format as an unstructured information object. If terminal device 110 does not have the capability to process information expressed in natural language, it can determine the indicated format as a structured information object.

[0070] Then, terminal device 110 sends (310) the dataset to first network device 120. In some example embodiments, format information may be sent to first network device 120 along with the dataset. Alternatively, in some example embodiments, format information may be sent to first network device 120 separately, such as... Figure 3 As shown.

[0071] In some example embodiments, format information may be sent by the terminal device 110 upon request. For example... Figure 3 As shown, the first network device 120 can send a (302) format request to the terminal device 110, the format information being used to determine the format for feedback of the dataset's performance. The terminal device 110 can receive the (304) format request from the first network device 120 and send (306) format information as a response to the format request.

[0072] In some example embodiments, the format request may indicate multiple candidate formats for performance feedback. For example, the multiple candidate formats may include structured information objects and unstructured information objects. The terminal device 110 may select a format from the multiple candidate formats and notify the first network device 120 of the selected format via format information.

[0073] In some example embodiments, the second terminal device 120 may send a request to the terminal device 110 for a dataset used for model training. A format request may include or be included in the request for the dataset. That is, the request for the format and the request for the dataset may be sent together or separately to the terminal device 110. Similarly, format information and the dataset may be sent together or separately to the first network device 120.

[0074] continue Figure 3 First network device 120 receives (312) the dataset from terminal device 110. Then, based on the dataset, first network device 120 can determine (314) the performance of the dataset. A method for determining the performance of the dataset can be found in [reference needed]. Figure 2 The content described.

[0075] The first network device 120 sends feedback information to the terminal device 110 regarding the performance of a dataset with an indicated format. The feedback information can be generated by the first network device 120 based on the format indicated in the format information.

[0076] In some examples, feedback information can include various types of information, such as scores representing the performance of the dataset, scales associated with the scores, algorithms used to determine the performance of the dataset, one or more attributes of the dataset that degrade the performance of the dataset, one or more suggestions for improving the dataset information, etc. Feedback information is similar to the information referenced above. Figure 3 The descriptions are similar.

[0077] Terminal device 110 receives (318) feedback information from first network device 120. Then, terminal device 110 can interpret (320) the feedback information based on the indicated format.

[0078] Figure 4 An example signaling flow 400 is shown, illustrating an example process for providing feedback on the performance of training data according to some example embodiments of this disclosure. Signaling flow 400 can be considered as... Figure 2 Example embodiment of signaling flow 200. Signaling flow 400 involves UE 401, DCE 402, and MnS producer 403. UE can be considered as an example of terminal device 110. DCE 402 can be considered as an example of first network device 120. MnS producer 403 can be considered as an example of second network device 130.

[0079] like Figure 4As shown, DCE 402 can send (410) a request for training data to UE 401. For example, the request may include information and / or features desired by DCE 402. In some example embodiments, the request for training data may also request format information from UE 401. In some example embodiments, the request may indicate multiple candidate formats, such as structured information objects, unstructured information objects, etc. UE 401 may receive (412) a request from DCE 402 and send (414) a response to the request. The response may include a dataset collected for model training. In some example embodiments, the response may also include format information indicating the format selected by UE 401. DCE 402 may receive (416) a response from UE 401.

[0080] In some example embodiments, MnS producer 403 may send (430) information associated with dataset performance determination to DCE 402. This information may include, for example, an algorithm for dataset performance determination, multiple attributes to be considered in the dataset performance determination, one or more parameters to be used in the algorithm, etc. DCE 402 may receive (432) information from MnS producer 403.

[0081] Then, DCE 402 can determine (418) the performance of the dataset, for example, based on the algorithm and multiple attributes of the dataset. After obtaining the performance of the dataset, DCE 402 can send (420) feedback information about the performance of the dataset to UE 401. UE 401 can receive (422) the feedback information and optimize the data collection strategy for future data collection.

[0082] Figure 5 A flowchart of an example method 500 implemented at a terminal device according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 Method 500 is described from the perspective of terminal device 110.

[0083] At box 510, terminal device 110 sends a first dataset collected by the terminal device for training an artificial intelligence / machine learning (AI / ML) model to the first network device.

[0084] At box 520, terminal device 110 receives feedback information from the first network device regarding the performance of the first dataset.

[0085] In some example embodiments, the feedback information may include at least one of the following: a score representing the performance of the first dataset, a scale associated with the score scale, an algorithm for determining the performance of the first dataset, one or more attributes of the first dataset that reduce the performance of the first dataset, or one or more suggestions for improving the performance of the dataset.

[0086] In some example embodiments, method 500 may further include: determining the performance of the first dataset from the feedback information based on the format of the structured information object.

[0087] In some example embodiments, method 500 may further include: determining the performance of the first dataset by processing unstructured information objects.

[0088] In some example embodiments, method 500 may further include sending to a first network device an indication of the terminal device's ability to process information expressed in natural language.

[0089] In some example embodiments, method 500 may further include: receiving an indication from a first network device regarding the format of the feedback information; and interpreting the feedback information based on the indicated format.

[0090] In some example embodiments, method 500 may further include: updating the data collection strategy for the AI / ML model based on feedback information; and collecting a second dataset for the AI / ML model based on the updated data collection strategy.

[0091] In some example embodiments, the first network device may include or be included in the core network entity used for training data collection.

[0092] Figure 6 A flowchart of an example method 600 implemented at a first network device according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 Method 600 is described from the perspective of the first network device 120 in the middle.

[0093] At box 610, the first network device 120 receives from the terminal device a first dataset collected by the terminal device for training an artificial intelligence / machine learning (AI / ML) model.

[0094] At box 620, the first network device 120 determines the performance of the first dataset.

[0095] At box 630, the first network device 120 sends feedback information about the performance of the first dataset to the terminal device.

[0096] In some example embodiments, the feedback information may include at least one of the following: a score representing the performance of the first dataset, a scale associated with the score scale, an algorithm for determining the performance of the first dataset, one or more attributes of the first dataset that reduce the performance of the first dataset, or one or more suggestions for improving the performance of the dataset.

[0097] In some example embodiments, a first network device may determine the performance of a first dataset by: determining multiple attributes of the first dataset based on the data items included in the first dataset; and determining a score representing the performance of the first dataset based on an algorithm and the multiple attributes.

[0098] In some example embodiments, multiple datasets including a first dataset can be received by a first network device, and the first network device can determine the score by: determining an absolute metric corresponding to the performance of the first dataset based on an algorithm and multiple attributes; and determining a relative metric corresponding to the performance as the score based on the absolute metric of the first dataset and the corresponding absolute metrics of the multiple datasets.

[0099] In some example embodiments, method 600 may further include: receiving information from a second network device associated with a dataset performance determination for an AI / ML model, wherein the performance of the first dataset is determined based on the received information.

[0100] In some example embodiments, the information associated with dataset performance determination may indicate at least one of the following: the algorithm used for dataset performance determination, multiple dataset attributes to be considered in dataset performance determination, or one or more parameters to be used in the algorithm.

[0101] In some example embodiments, method 600 may further include: generating feedback information based on the format of a structured information object.

[0102] In some example embodiments, method 600 may further include generating feedback information by generating an unstructured information object.

[0103] In some example embodiments, method 600 may further include: receiving from the terminal device an indication of the terminal device's ability to process information expressed in natural language.

[0104] In some example embodiments, method 600 may further include sending an instruction to a terminal device regarding the format of the feedback information.

[0105] In some example embodiments, the first network device may include or be included in the core network entity used for training data collection.

[0106] Figure 7A flowchart of an example method 700 implemented at a second network device according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 Method 700 is described from the perspective of the second network device 130 in the middle.

[0107] At box 710, the second network device 130 sends information to the first network device related to the determination of dataset performance for artificial intelligence / machine learning AI / ML models.

[0108] At box 720, the second network device 130 receives from the first network device multiple datasets collected by multiple terminal devices for training AI / ML models.

[0109] In some example embodiments, the information associated with dataset performance determination may indicate at least one of the following: the algorithm used for dataset performance determination, multiple dataset attributes to be considered in dataset performance determination, or one or more parameters to be used in the algorithm.

[0110] In some example embodiments, a terminal device capable of performing any method 500 (e.g., Figure 1 The terminal device 110 may include components for performing corresponding operations of method 500. The device can be implemented in any suitable form. For example, the device can be implemented in a circuit or software module. The terminal device can be implemented as or included in... Figure 1 In terminal device 110.

[0111] In some example embodiments, the terminal device includes: a component for sending a first dataset collected by the terminal device for training an artificial intelligence / machine learning (AI / ML) model to a first network device; and a component for receiving feedback information from the first network device regarding the performance of the first dataset.

[0112] In some example embodiments, the feedback information may include at least one of the following: a score representing the performance of the first dataset, a scale associated with the score scale, an algorithm for determining the performance of the first dataset, one or more attributes of the first dataset that reduce the performance of the first dataset, or one or more suggestions for improving the performance of the dataset.

[0113] In some example embodiments, the terminal device may also include a component for determining the performance of the first dataset from the feedback information based on the format of the structured information object.

[0114] In some example embodiments, the terminal device may also include a component for determining the performance of the first dataset by processing unstructured information objects.

[0115] In some example embodiments, the terminal device may also include a component for sending an indication to a first network device of the terminal device's ability to process information expressed in natural language.

[0116] In some example embodiments, the terminal device may further include: a component for receiving an instruction from a first network device regarding the format of the feedback information; and a component for interpreting the feedback information based on the indicated format.

[0117] In some example embodiments, the terminal device may further include: a component for updating a data collection strategy for an AI / ML model based on feedback information; and a component for collecting a second dataset for the AI / ML model based on the updated data collection strategy.

[0118] In some example embodiments, the first network device may include or be included in the core network entity used for training data collection.

[0119] In some example embodiments, a first network device capable of performing any of the methods in method 600 (e.g., Figure 1 The first network device 120 may include components for performing the corresponding operations of method 600. The device may be implemented in any suitable form. For example, the device may be implemented in a circuit or software module. The first network device may be implemented as or included in... Figure 1 The first network device in the network is 120.

[0120] In some example embodiments, the first network device includes: components for receiving a first dataset collected by a terminal device for training an artificial intelligence / machine learning (AI / ML) model; components for determining the performance of the first dataset; and components for sending feedback information about the performance of the first dataset to the terminal device.

[0121] In some example embodiments, the feedback information may include at least one of the following: a score representing the performance of the first dataset, a scale associated with the score scale, an algorithm for determining the performance of the first dataset, one or more attributes of the first dataset that reduce the performance of the first dataset, or one or more suggestions for improving the performance of the dataset.

[0122] In some example embodiments, a first network device may determine the performance of a first dataset by: determining multiple attributes of the first dataset based on the data items included in the first dataset; and determining a score representing the performance of the first dataset based on an algorithm and the multiple attributes.

[0123] In some example embodiments, multiple datasets including a first dataset can be received by a first network device, and the first network device can determine the score by: determining an absolute metric corresponding to the performance of the first dataset based on an algorithm and multiple attributes; and determining a relative metric corresponding to the performance as the score based on the absolute metric of the first dataset and the corresponding absolute metrics of the multiple datasets.

[0124] In some example embodiments, the first network device may further include a component for receiving information from the second network device associated with a performance determination of a dataset for an AI / ML model, wherein the performance of the first dataset is determined based on the received information.

[0125] In some example embodiments, the information associated with dataset performance determination may indicate at least one of the following: the algorithm used for dataset performance determination, multiple dataset attributes to be considered in dataset performance determination, or one or more parameters to be used in the algorithm.

[0126] In some example embodiments, the first network device may further include a component for generating feedback information based on the format of a structured information object.

[0127] In some example embodiments, the first network device may further include a component for generating feedback information by generating unstructured information objects.

[0128] In some example embodiments, the first network device may further include a component for receiving from the terminal device an indication of the terminal device's ability to process information expressed in natural language.

[0129] In some example embodiments, the first network device may further include a component for sending instructions to the terminal device regarding the format of the feedback information.

[0130] In some example embodiments, the first network device may include or be included in the core network entity used for training data collection.

[0131] In some example embodiments, a second network device capable of performing any of the methods in method 700 (e.g., Figure 1 The second network device 130 may include components for performing the corresponding operations of method 700. This device can be implemented in any suitable form. For example, it can be implemented in a circuit or software module. The second network device can be implemented as or included in... Figure 1 The second network device 130 in the system.

[0132] In some example embodiments, the second network device includes: components for sending information to the first network device associated with a dataset performance determination for an artificial intelligence / machine learning (AI / ML) model; and components for receiving from the first network device multiple datasets collected by multiple terminal devices for training the AI / ML model.

[0133] In some example embodiments, the information associated with dataset performance determination may indicate at least one of the following: the algorithm used for dataset performance determination, multiple dataset attributes to be considered in dataset performance determination, or one or more parameters to be used in the algorithm.

[0134] Figure 8 A flowchart of an example method 800 implemented at a terminal device according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 Method 800 is described from the perspective of terminal device 110.

[0135] At box 810, terminal device 110 sends format information to first network device. The format information indicates the format used to provide feedback on the performance of the dataset, which is collected by the terminal device and used to train an artificial intelligence / machine learning (AI / ML) model.

[0136] At box 820, terminal device 110 sends a dataset to the first network device.

[0137] At box 830, terminal device 110 receives feedback information about the performance of the dataset from a first network device, the feedback information having the indicated format.

[0138] In some example embodiments, method 800 may further include: receiving a format request from a first network device, the format request being used to determine a format for feedback of the performance of the dataset, wherein format information is sent as a response to the format request.

[0139] In some example embodiments, the format request may indicate multiple candidate formats for performance feedback, and the indicated format is selected from the multiple candidate formats.

[0140] In some example embodiments, a formatting request may include or be included in a request for a dataset.

[0141] In some example embodiments, the indicated format may include at least one of a structured information object or an unstructured information object.

[0142] In some example embodiments, formatting information may be sent along with the dataset to a first network device.

[0143] In some example embodiments, method 800 may further include: determining whether the terminal device has the ability to process information expressed in natural language; determining the indicated format as an unstructured information object based on the determination that the terminal device has the ability to process information expressed in natural language; and determining the indicated format as a structured information object based on the determination that the terminal device does not have the ability to process information expressed in natural language.

[0144] In some example embodiments, feedback information may include at least one of the following: a score representing the performance of the dataset, a scale associated with the score scale, an algorithm for determining the performance of the dataset, one or more attributes of the dataset that reduce the performance of the first dataset, or one or more suggestions for improving the performance of the dataset.

[0145] Figure 9 A flowchart of an example method 900 implemented at a first network device according to some example embodiments of the present disclosure is shown. For discussion purposes, [the following will be discussed]. Figure 1 Method 900 is described from the perspective of the first network device 120 in the middle.

[0146] At box 910, the first network device 120 receives format information from the terminal device, which indicates the format used to provide feedback on the performance of the dataset collected by the terminal device for training an artificial intelligence / machine learning (AI / ML) model.

[0147] At box 920, the first network device 120 receives a dataset from the terminal device.

[0148] At box 930, the first network device 120 sends feedback information about the performance of the dataset to the terminal device, the feedback information having the indicated format.

[0149] In some example embodiments, method 900 may further include: sending a format request to a terminal device to determine a format for feedback of the performance of the dataset, wherein format information is received as a response to the format request.

[0150] In some example embodiments, the format request may indicate multiple candidate formats for performance feedback, and the indicated format is selected from the multiple candidate formats.

[0151] In some example embodiments, a formatting request may include or be included in a request for a dataset.

[0152] In some example embodiments, the indicated format may include at least one of a structured information object or an unstructured information object.

[0153] In some example implementations, format information can be received from the terminal device along with the dataset.

[0154] In some example embodiments, feedback information may include at least one of the following: a score representing the performance of the dataset, a scale associated with the score scale, an algorithm for determining the performance of the dataset, one or more attributes of the dataset that reduce the performance of the first dataset, or one or more suggestions for improving the performance of the dataset.

[0155] In some example embodiments, a terminal device capable of executing any method 800 (e.g., Figure 1 The terminal device 110 may include components for performing corresponding operations of method 800. The device may be implemented in any suitable form. For example, the device may be implemented in a circuit or software module. The terminal device may be implemented as or included in... Figure 1 In terminal device 110.

[0156] In some example embodiments, the terminal device includes: a component for sending format information to a first network device, the format information indicating a format for providing feedback on the performance of a dataset collected by the terminal device for training an artificial intelligence / machine learning (AI / ML) model; a component for sending the dataset to the first network device; and a component for receiving feedback information from the first network device regarding the performance of the dataset in the indicated format.

[0157] In some example embodiments, the terminal device may further include: a component for receiving a format request from a first network device, the format request being used to determine a format for feedback of the performance of the dataset, wherein format information is sent as a response to the format request.

[0158] In some example embodiments, the format request may indicate multiple candidate formats for performance feedback, and the indicated format is selected from the multiple candidate formats.

[0159] In some example embodiments, a formatting request may include or be included in a request for a dataset.

[0160] In some example embodiments, the indicated format may include at least one of a structured information object or an unstructured information object.

[0161] In some example embodiments, formatting information may be sent along with the dataset to a first network device.

[0162] In some example embodiments, the terminal device may further include: a component for determining whether the terminal device has the ability to process information expressed in natural language; a component for determining the indicated format as an unstructured information object based on the determination that the terminal device has the ability to process information expressed in natural language; and a component for determining the indicated format as a structured information object based on the determination that the terminal device does not have the ability to process information expressed in natural language.

[0163] In some example embodiments, feedback information may include at least one of the following: a score representing the performance of the dataset, a scale associated with the score scale, an algorithm for determining the performance of the dataset, one or more attributes of the dataset that reduce the performance of the first dataset, or one or more suggestions for improving the performance of the dataset.

[0164] In some example embodiments, a first network device capable of performing any of the methods in method 900 (e.g., Figure 1 The first network device 120 may include components for performing the corresponding operations of method 900. The device may be implemented in any suitable form. For example, the device may be implemented in a circuit or software module. The first network device may be implemented as or included in... Figure 1 The first network device in the network is 120.

[0165] In some example embodiments, the first network device includes: a component for receiving format information from a terminal device, the format information indicating a format component for providing feedback on the performance of a dataset collected by the terminal device for training an artificial intelligence / machine learning (AI / ML) model; a component for receiving the dataset from the terminal device; and a component for sending feedback information with the indicated format to the terminal device regarding the performance of the dataset.

[0166] In some example embodiments, the first network device may further include: a component for sending a format request to a terminal device, the format request being used to determine a format for feedback of the performance of the dataset, wherein format information is received as a response to the format request.

[0167] In some example embodiments, the format request may indicate multiple candidate formats for performance feedback, and the indicated format is selected from the multiple candidate formats.

[0168] In some example embodiments, a formatting request may include or be included in a request for a dataset.

[0169] In some example embodiments, the indicated format may include at least one of a structured information object or an unstructured information object.

[0170] In some example implementations, format information can be received from the terminal device along with the dataset.

[0171] In some example embodiments, feedback information may include at least one of the following: a score representing the performance of the dataset, a scale associated with the score scale, an algorithm for determining the performance of the dataset, one or more attributes of the dataset that reduce the performance of the first dataset, or one or more suggestions for improving the performance of the dataset.

[0172] Figure 10 This is a simplified block diagram of a device 1000 suitable for implementing exemplary embodiments of the present disclosure. The device 1000 can be provided to implement a communication device, for example, as... Figure 1 The terminal device 110, the first network device 120, or the second network device 130 are shown. As shown, device 1000 includes one or more processors 1010, one or more memories 1020 coupled to processor 1010, and one or more communication modules 1040 coupled to processor 1010.

[0173] Communication module 1040 is used for bidirectional communication. Communication module 1040 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interface can represent any interface necessary for communication with other network elements. In some example embodiments, communication module 1040 may include at least one antenna.

[0174] As a non-limiting example, processor 1010 can be any type suitable for a local technology network and can include one or more of the following: general-purpose computer, special-purpose computer, microprocessor, digital signal processor (DSP), and processor based on a multi-core processor architecture. Device 1000 can have multiple processors, such as application-specific integrated circuit chips that are time-dependent on a clock that synchronizes with the main processor.

[0175] Memory 1020 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 1024, electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), optical disc, laser disc, and other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, random access memory (RAM) 1022 and other volatile memories that will not persist for the duration of a power outage.

[0176] Computer program 1030 includes computer-executable instructions that are executed by an associated processor 1010. The instructions of program 1030 may include instructions for performing operations / actions of some example embodiments of this disclosure. Program 1030 may be stored in memory (e.g., ROM 1024). Processor 1010 can perform any suitable actions and processes by loading program 1030 into RAM 1022.

[0177] The exemplary embodiments of this disclosure can be implemented by program 1030, enabling device 1000 to execute as described in the reference. Figures 2 to 4 Any process discussed in this disclosure. Exemplary embodiments of this disclosure may also be implemented by hardware or a combination of software and hardware.

[0178] In some example embodiments, program 1030 may be tangibly contained in a computer-readable medium, which may be included in device 1000 (such as in memory 1020) or other storage device accessible by device 1000. Device 1000 may load program 1030 from the computer-readable medium into RAM 1022 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. As used herein, the term "non-transitory" is a limitation of the medium itself (i.e., tangible, not tactile), rather than a limitation of the persistence of data storage (e.g., RAM versus ROM).

[0179] Figure 11 An example of a computer-readable medium 1100 is shown, which may be in the form of a CD, DVD, or other optical storage disc. The computer-readable medium 1100 has a program 1030 stored thereon.

[0180] Generally, various embodiments of this disclosure can be implemented in hardware or special-purpose circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, and others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. Although various aspects of embodiments of this disclosure are shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, special-purpose circuitry or logic, general-purpose hardware, or controllers or other computing devices, or some combination thereof, as non-limiting examples.

[0181] Some exemplary embodiments of this disclosure also provide at least one computer program product tangibly stored on a computer-readable medium, such as a non-transitory computer-readable medium. The computer program product includes computer-executable instructions that execute in a device on a target physical or virtual processor, such as those included in a program module, to perform any of the methods described above. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a particular task or implement a particular abstract data type. In various embodiments, the functionality of a program module can be combined or split among program modules as needed. The machine-executable instructions for a program module can execute within a local or distributed device. In a distributed device, a program module can reside on both local and remote storage media.

[0182] Program code used to perform the methods of this disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that, when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a stand-alone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0183] In the context of this disclosure, computer program code or related data may be carried by any suitable carrier wave to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carrier waves include signals, computer-readable media, etc.

[0184] Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable media can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0185] Furthermore, although the operations are described in a specific order, this should not be construed as requiring that such operations be performed in the specific order shown or sequentially, or requiring that all the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure, but rather as descriptions of features that may be specific to particular embodiments. Unless explicitly stated otherwise, certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, unless explicitly stated otherwise, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0186] Although this disclosure has been described in language specific to structural features and / or methodological actions, it should be understood that the disclosure as defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

Claims

1. A terminal device, comprising: At least one processor; as well as At least one memory stores instructions that, when executed by the at least one processor, cause the terminal device to at least: Send format information to a first network device, the format information indicating a format for feedback of the performance of a dataset, the dataset being collected by the terminal device, the dataset being used to train an artificial intelligence / machine learning (AI / ML) model; Send the dataset to the first network device; as well as Feedback information regarding the performance of the dataset is received from the first network device, the feedback information having the indicated format.

2. The terminal device according to claim 1, wherein the terminal device is further configured to: A format request is received from the first network device, the format request being used to determine the format for feedback of the performance of the dataset, wherein the format information is sent as a response to the format request.

3. The terminal device of claim 2, wherein the format request indicates a plurality of candidate formats for feedback on the performance, and the indicated format is selected from the plurality of candidate formats.

4. The terminal device of claim 2, wherein the format request includes a request for the dataset or is included in the request for the dataset.

5. The terminal device of claim 1, wherein the indicated format includes at least one of the following: Structured information objects, or Unstructured information objects.

6. The terminal device according to claim 1, wherein the format information is sent to the first network device together with the dataset.

7. The terminal device according to claim 1, wherein the terminal device is further configured to: Determine whether the terminal device has the ability to process information expressed in natural language; Based on the determination that the terminal device has the capability to process information expressed in natural language, the indicated format is determined to be an unstructured information object; and Based on the determination that the terminal device does not have the ability to process information expressed in natural language, the indicated format is determined to be a structured information object.

8. The terminal device according to claim 1, wherein the feedback information includes at least one of the following: A score representing the performance of the dataset. The scale associated with the rating, An algorithm used to determine the performance of the dataset. One or more attributes of the dataset, the one or more attributes of the dataset degrade the performance of the dataset, or One or more suggestions for improving dataset performance.

9. A first network device, comprising: At least one processor; as well as At least one memory stores instructions that, when executed by the at least one processor, cause the first network device to at least: Receive format information from the terminal device, the format information indicating the format used to provide feedback on the performance of the dataset, the dataset being collected by the terminal device and used to train an artificial intelligence / machine learning (AI / ML) model; Receive the dataset from the terminal device; as well as Feedback information regarding the performance of the dataset is sent to the terminal device, the feedback information having the indicated format.

10. The first network device according to claim 9, wherein the first network device is further configured to: A format request is sent to the terminal device, the format request being used to determine the format for feeding back the performance of the dataset, wherein the format information is received as a response to the format request.

11. The first network device of claim 10, wherein the format request indicates a plurality of candidate formats for feedback of the performance, and the indicated format is selected from the plurality of candidate formats.

12. The first network device of claim 10, wherein the format request includes or is included in the request for the dataset.

13. The first network device of claim 9, wherein the indicated format includes at least one of the following: Structured information objects, or Unstructured information objects.

14. The first network device of claim 9, wherein the format information is received from the terminal device together with the dataset.

15. The first network device according to claim 9, wherein the feedback information includes at least one of the following: A score representing the performance of the dataset. The scale associated with the rating, An algorithm used to determine the performance of the dataset. One or more attributes of the dataset, the one or more attributes of the dataset degrade the performance of the dataset, or One or more suggestions for improving dataset performance.

16. A method for communication, comprising: Send format information to a first network device, the format information indicating a format for feedback of the performance of a dataset collected by a terminal device, the dataset being used to train an artificial intelligence / machine learning (AI / ML) model; Send the dataset to the first network device; as well as Feedback information regarding the performance of the dataset is received from the first network device, the feedback information having the indicated format.

17. A method for communication, comprising: Receive format information from the terminal device, the format information indicating the format used to provide feedback on the performance of the dataset, the dataset being collected by the terminal device and used to train an artificial intelligence / machine learning (AI / ML) model; Receive the dataset from the terminal device; as well as Feedback information regarding the performance of the dataset is sent to the terminal device, the feedback information having the indicated format.

18. A terminal device, comprising: A component for sending format information to a first network device, the format information indicating a format for feedback on the performance of a dataset collected by a terminal device, the dataset being used to train an artificial intelligence / machine learning (AI / ML) model; Components for sending the dataset to the first network device; as well as A component for receiving feedback information about the performance of the dataset from the first network device, the feedback information having the indicated format.

19. A first network device, comprising: A component for receiving format information from a terminal device, the format information indicating a format for feedback on the performance of a dataset collected by the terminal device and used to train an artificial intelligence / machine learning (AI / ML) model; Components for receiving the dataset from the terminal device; as well as A component for sending feedback information about the performance of the dataset to the terminal device, the feedback information having the indicated format.

20. A computer-readable medium comprising instructions stored thereon, the instructions being configured to cause a device to perform at least the method of claim 16 or the method of claim 17.

Images