Information transmission method, device and equipment

Abstract

The present application discloses an information transmission method, an apparatus, and a device in the field of communication technology. The method of the embodiment of the present application includes: a first device transmitting first information corresponding to first accuracy information of a first model to a second device or a third device, the first information being used to describe information used to obtain the first accuracy information, wherein the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority from Chinese Patent Application No. 202310087252.1 filed in China on January 16, 2023, and from Chinese Patent Application No. 202310102348.0 filed in China on February 8, 2023, the entire contents of which are incorporated herein by reference. [Technical Field]

[0002] The present application relates to the field of communications technology, and more particularly to information transmission methods, devices and equipment. [Background technology]

[0003] With the development of technology, some network elements have been introduced into communication networks to perform intelligent data analysis, generating data analytics (also known as inferential data results) for some tasks, which can assist devices inside and outside the network in making policy decisions, using artificial intelligence (AI) models to enhance the degree of intelligent device policy decisions.

[0004] Currently, to ensure the accuracy of data analysis results, the accuracy in actual application (AiU) or inference (inference) stage is calculated. However, in certain scenarios, errors are likely to occur in the AiU obtained by the above method, which can lead to problems such as erroneously stopping the use of correct inference results or continuing to use incorrect inference results. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide an information transmission method, device and apparatus, which can solve the problem that the accuracy of model inference in the conventional method has a relatively large error, which affects the use of the inference results. [Means for solving the problem]

[0006] According to a first aspect, there is provided an information transmission method, the information transmission method comprising: transmitting, by the first device to the second device or the third device, first information corresponding to first accuracy information of the first model, the first information being used to describe information used to obtain the first accuracy information; Here, the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model.

[0007] According to a second aspect, there is provided an information transmission device, the information transmission device comprising: a first transmission module for transmitting first information corresponding to first accuracy information of the first model to a second device or a third device, the first information being used to describe information used to obtain the first accuracy information; Here, the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model.

[0008] According to a third aspect, there is provided an information transmission method, the information transmission method comprising: the second device triggering execution of the first model by the first device; and receiving, by the second device, first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0009] According to a fourth aspect, there is provided an information transmission device, the information transmission device comprising: a first processing module for triggering the first device to execute the first model; and a first receiving module for receiving first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0010] According to a fifth aspect, there is provided an information transmission method, the information transmission method comprising: transmitting the first model from the third device to the first device; and receiving, by the third device, first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0011] According to a sixth aspect, there is provided an information transmission device, the information transmission device comprising: a second transmitting module for transmitting the first model to the first device; and a second receiving module for receiving first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0012] According to a seventh aspect, there is provided a communications device, the communications device comprising a processor and a memory, the memory storing a program or instructions operable to run on the processor, the program or instructions being operable when executed by the processor to implement the method of the first aspect, or to implement the method of the third aspect, or to implement steps of the method of the fifth aspect.

[0013] According to an eighth aspect, there is provided a communications device, the communications device including a processor and a communications interface, wherein the communications interface is adapted to transmit first information corresponding to first accuracy information of a first model to a second device or a third device, the first information being adapted to describe information used to obtain the first accuracy information; Here, the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model.

[0014] According to a ninth aspect, there is provided a communications device, the communications device comprising a processor and a communications interface, wherein the processor is adapted to trigger a first device to execute a first model; The communication interface is used to receive first information corresponding to first accuracy information of the first model transmitted from the first device, and the first information is used to describe information used to obtain the first accuracy information.

[0015] According to a tenth aspect, there is provided a communications device, the communications device including a processor and a communications interface, wherein the communications interface is used to transmit a first model to a first device, and the communications interface is further used to receive first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0016] According to an eleventh aspect, there is provided an information transmission system, the information transmission system including a first device, a second device, and a third device, the first device being used to perform steps of the method according to the first aspect, the second device being used to perform steps of the method according to the third aspect, and the third device being used to perform steps of the method according to the fifth aspect.

[0017] According to a twelfth aspect, there is provided a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, performs the method of the first aspect, or the method of the third aspect, or the steps of the method of the fifth aspect.

[0018] According to a thirteenth aspect, there is provided a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor running a program or instruction to implement the method of the first aspect, or the method of the third aspect, or the method of the fifth aspect.

[0019] According to a fourteenth aspect, there is provided a computer program / program product, the computer program / program product being stored on a storage medium, the computer program / program product being executed by at least one processor to implement the method according to the first aspect, or to implement the method according to the third aspect, or to implement the steps of the method according to the fifth aspect. [Effects of the Invention]

[0020] In an embodiment of the present application, after the first device sends the first information to the second device (the device that triggers the first device to execute the first model) or the third device (the device that generates the first model), the second device or the third device can know the information of the data used to obtain the first accuracy information of the first model, and thereby combine it with the first information for subsequent processing, thereby avoiding the problem of the deviation of the first accuracy information being too large, which affects the use of the inference result. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram of a wireless communication system. [Figure 2] This is a flowchart of AI data analysis. [Figure 3] 1 is a schematic diagram of an information transmission method according to an embodiment of the present application; [Figure 4] 1 is a schematic diagram of the application of the method of the embodiment of the present application. [Figure 5] 2 is a second schematic diagram of the application of the method of the embodiment of the present application. [Figure 6] This is the third schematic diagram of the application of the method in the embodiment of the present application. [Figure 7] This is the fourth schematic diagram of the application of the method in the examples of the present application. [Figure 8] 2 is a second schematic diagram of an information transmission method according to an embodiment of the present application. [Figure 9] 3 is a third schematic diagram of an information transmission method according to an embodiment of the present application. [Figure 10] FIG. 4 is a schematic diagram of the module structure of the device corresponding to FIG. 3. [Figure 11] FIG. 9 is a schematic diagram of the module structure of the device corresponding to FIG. 8. [Figure 12] FIG. 10 is a schematic diagram of the module structure of the device corresponding to FIG. 9. [Figure 13] 1 is a structural schematic diagram of a communication device according to an embodiment of the present application; [Figure 14] 1 is a structural schematic diagram of a terminal according to an embodiment of the present application; [Figure 15] FIG. 2 is a structural schematic diagram of a network-side device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0022] The following clearly describes the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application fall within the scope of protection of the present application.

[0023] The terms "first," "second," etc. in the specification and claims of this application are intended to distinguish between similar objects and are not intended to describe a particular order or sequence. It should be understood that terms used in this manner are interchangeable where appropriate, so that the embodiments of this application may be performed in an order other than that illustrated or described herein. Furthermore, the objects distinguished by "first" and "second" are generally of the same type and do not limit the number of objects; for example, the first object may be one or more. Furthermore, "or" in the specification and claims indicates at least one of the connected objects; for example, "A or B" covers three solutions, i.e., Solution 1: including A but not B; Solution 2: including B but not A; and Solution 3: including both A and B. The term "instructions" in the specification and claims of this application may be an explicit instruction or an implicit instruction. Here, an explicit instruction may be understood as the sender explicitly informing the receiver of the operation or result of the request that needs to be performed in the instruction sent, and an implicit instruction may be understood as the receiver making a judgment based on the instruction sent from the sender and determining the result of the operation or result of the request that needs to be performed based on the judgment result.

[0024] It should be noted that the techniques described in the embodiments of the present application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be applied to other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-Carrier Frequency Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in the embodiments of the present application are always used interchangeably, and the described techniques may be used in the above-mentioned systems and radio technologies, as well as other systems and radio technologies. The following description describes a New Radio (NR) system for illustrative purposes, and NR terminology is used in most of the following description. However, these techniques may also be used in applications other than NR system applications, such as sixth generation (6G) networks. th This may be applied to 6G (6th Generation) communication systems.

[0025] 1 shows a block diagram of a wireless communication system to which an embodiment of the present application can be applied. The wireless communication system includes a terminal 11 and a network side device 12. Here, the terminal 11 may be a mobile phone, a tablet personal computer, a laptop computer (also called a notebook computer), a personal digital assistant (PDA), a palmtop computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, a vehicle user equipment (VUE), a pedestrian user equipment (PUE), a smart home (home devices with wireless communication capabilities, such as a refrigerator, a television, a washing machine, or furniture), a game console, a personal computer, a The wearable device may be a terminal-side device such as a smart computer (PC), a teller machine or a self-service machine, and the wearable device may be a smart watch, a smart bracelet, a smart earphone, a smart eyeglasses, a smart accessory (smart bracelet, smart hand chain, smart ring, smart necklace, smart ankle bracelet, smart anklet, etc.), a smart band, a smart clothing, etc. In addition to the above terminal devices, the wearable device may also be a chip within the terminal, such as a modem chip or a system-level chip (SoC). It should be noted that the specific type of the terminal 11 in the embodiments of the present application is not limited.The network side device 12 may include access network equipment or core network equipment, where the access network equipment may also be referred to as radio access network equipment, a radio access network (RAN), a radio access network function, or a radio access network unit. The access network equipment may include a base station, a wireless local area network (WLAN) access point, a WiFi node, etc. The base station may also be referred to as a Node B, an evolved Node B (eNB), an access point, a base transceiver station (BTS), a radio base station, a radio transceiver, a basic service set (BSS), an extended service set (ESS), a home B node, a home evolved B node, a transmitting receiving point (TRP), or any other appropriate term in the art. As long as the same technical effect is achieved, the base station is not limited to a specific technical term. For illustrative purposes, the embodiments of this application only take a base station in an NR system as an example, and do not limit the specific type of base station.Core network devices include core network nodes, core network functions, mobility management entities (MMEs), access and mobility management functions (AMFs), session management functions (SMFs), user plane functions (UPFs), policy control functions (PCFs), policy and charging rules functions (PCRFs), edge application server discovery functions (EASDFs), unified data management (UDMs), unified data repository (UDRs), home subscriber servers (HSSs), centralized network configuration (CNCs), network repository functions (NRFs), network exposure functions (NEFs), local NEFs (or L-NEFs), binding support functions (BSFs), and application functions (Application Node Functions). It should be noted that the embodiments of the present application only take core network equipment in an NR system as an example, and do not limit the specific type of core network equipment.

[0026] The following describes in detail the information transmission method according to the embodiments of the present application through several embodiments and its application scenarios in conjunction with the drawings.

[0027] For ease of understanding, the following is a description of some content progressions according to the embodiments of the present application: 1. Calculating the accuracy of AI models In the field of AI, during AI training, a model periodically calculates its accuracy, i.e., the accuracy in training (AiT). For example, AiT is the percentage of correct model predictions relative to the total number of predictions. During the training phase, a validation dataset contains input data and labels (label data), which are in correspondence with each other, with one set of input data corresponding to one (set) of labels. The predictions generated by the comparison model are compared with the current training labels to determine whether the current training is correct.

[0028] Similarly, the calculation of accuracy in use (AiU) during the reasoning phase of an AI is the same as the accuracy during the training phase, i.e., the number of correct guesses divided by the total number of guesses.

[0029] Second, the customer of the current task can also be referred to as the flow in which the consumer network function (consumer NF) obtains AI data analysis results (analytics) from the network data analytics function (NWDAF).

[0030] Functionally, NWDAF is A Model Training Logical Function (MTLF) for generating or training a model; It can be divided into two network elements: an Analytics Logical Function (AnLF) for inferring the generation of predictive information or the generation of analytical information.

[0031] The MTLF and AnLF may be deployed independently on different network element devices or may be co-located on the same network element device, such as the NWDAF, to provide both AI model training and model inference functions.

[0032] Specifically, as shown in Figure 2, 1. The MTLF (including the NWDAF containing MTLF) collects training data from the training data source device.

[0033] 2. MTLF trains Model A based on the training data.

[0034] 3. The Consumer NF sends a request message to the AnLF (NWDAF containing AnLF), which is used to request the AnLF to analyze or reason about a specific task, and this request message includes: Analytics identifier information (analytics ID), Analytics Filter Information, Includes information to be analyzed (Analytics Target), etc.

[0035] Here, this request message may be a Nnwdaf_AnalyticsSubscription_Subscribe message.

[0036] 4. AnLF requests a model from MTLF based on this request message. This message may be a Nnwdaf_MLModelProvision_Subscribe or Nnwdaf_MLModelInfo_Request message, which may include: -An analytics ID, which is information used to determine which model A should be used, and -Analytics Filter Information is included.

[0037] 5. The MTLF sends the information of Model A and the accuracy of the model (AiT) to the AnLF.

[0038] The message carrying this information may be a Nnwdaf_MLModelProvision_Notify or Nnwdaf_MLModelInfo_Response message.

[0039] NOTE: Steps 1 and 2 may be performed after step 4.

[0040] 6. AnLF determines the source device of the inference input data based on the received request message, inputs the data type information, and outputs related information such as the data type information.

[0041] 7. The AnLF obtains reasoning input data corresponding to the task, specifically, the AnLF may be used to send a request message for reasoning input data to the source device determined in step 6, and collect the reasoning input data corresponding to the task.

[0042] 8. AnLF makes inferences based on the acquired model A and inference input data, and obtains inference result data.

[0043] For example, the AnLF infers inference input data (e.g., numerical values ​​such as UE ID, time, and the current service status of the UE) based on model A corresponding to analytics ID=UE mobility, and obtains output data whose inference result data is the UE location.

[0044] AnLF can perform one inference calculation process to obtain one or more output result values, or AnLF can perform multiple inferences to obtain one or more output result values.

[0045] 9. The AnLF transmits the inference result data obtained by the inference to the consumer NF.

[0046] The inference result data may inform the consumer NF that the statistics or predicted values ​​obtained by inference from the model corresponding to the analytics ID are used to support the consumer NF in making a corresponding policy decision. For example, statistics or predicted values ​​corresponding to UE mobility may be used to support user paging optimization by the AMF.

[0047] 10. AnLF acquires label data corresponding to the inference result data.

[0048] The message carrying this label data may be an Nnf_EventExposure_subscribe message.

[0049] Specifically, the AnLF sends a label data request message to the source device of the label data determined in step 6, which includes type information of the label data, object information corresponding to the label data, time information (e.g., timestamp, time zone), etc., which are used to determine which label data to feed back to the source device of the label data.

[0050] 11.AnLF calculates AiU of model A based on the inference result data and label data.

[0051] 12. When the AnLF needs to transmit AiU to the consumer NF (for example, when it determines that the AiU does not meet the accuracy demand, or the accuracy is deteriorating, or the periodic transmission time has arrived), the AnLF transmits notification information to the consumer NF, and this notification information is used to notify that the AiU of the consumer NF model A does not meet the accuracy demand or is deteriorating, or is used to notify the AiU of the consumer NF model A.

[0052] Here, the message carrying this notification information may be Nnwdaf_AnalyticsSubscription_Notify.

[0053] 13. The consumer NF can perform the corresponding operation based on the AiU of the model A, for example, stop or suspend the use of the inference result data corresponding to the task.

[0054] 14. When the AnLF needs to transmit AiU to the MTLF (for example, when it determines that AiU does not meet the accuracy demand or has deteriorated, or that the periodic transmission time has arrived), the AnLF transmits notification information to the MTLF, which is used to notify that the accuracy of the MTLF model A does not meet the accuracy demand or has deteriorated, or to notify the accuracy of the consumer NF model A.

[0055] The notification information may be the same as the communication information in step 12 or may be different.

[0056] 15. The MTLF can perform the appropriate operations based on the AiU of model A, for example, re-training the model.

[0057] 16. After the MTLF retrains model A, the MTLF sends the retrained model A or the newly generated model B to the AnLF.

[0058] The information transmission method according to the embodiment of the present application, as shown in FIG. 3, includes the following steps: Step 301: A first device transmits to a second device or a third device first information corresponding to first accuracy information of a first model, the first information being used to describe information used to obtain the first accuracy information; Here, the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model.

[0059] Here, the first model is a model for inference corresponding to a specific task, or the first model is a model for inference corresponding to a specific analytics ID. The first accuracy information of the first model may be understood as model accuracy, or the accuracy of analysis using the first model, i.e., analytics accuracy, or accuracy provided by a first device (e.g., AnLF), or the accuracy of an analytics ID corresponding to the first model, i.e., accuracy for an Analytics ID or accuracy about an Analytics ID, or performance information of the first model, etc. The first information is used to describe information used to obtain the first accuracy information, i.e., the first information can explain or describe information of data used when obtaining the first accuracy information, where the data is inference input data.

[0060] It should be noted that the data may be understood as a sample or sample data. Each piece of data may be used to perform one inference with the first model. Each piece of data may include one or more parameters, and the one or more parameters are input parameters required for the first model to perform inference. Model Accuracy may be understood as machine learning (ML) model accuracy. It should be noted that information used to obtain the first accuracy information may be understood as information used to calculate the first accuracy information or information used to derive the first accuracy information.

[0061] It should be clarified that the second device triggering the first device to execute the first model may be understood as the second device requesting the first device to execute the first model, or the second device instructing the first device to execute the first model.

[0062] In this way, through step 301, after the first device sends the first information to the second device or the third device, the second device (the device that triggers the first device to execute the first model) or the third device (the device that generates the first model) can know the information of the data used to obtain the first accuracy information of the first model, and thereby combine it with the first information for subsequent processing, thereby avoiding the problem of the first accuracy information having too large a deviation, which affects the use of the inference results.

[0063] Optionally, the first information is: The number of data used and The number of times you perform the inference, The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0064] Here, since the above-mentioned first information is for describing information used to obtain the first accuracy information, it can be seen that: the number of data used, i.e., the number of inference input data used in the process of obtaining the first accuracy information, may be understood as the number of samples or as the sample space; the sampling time of the data used, i.e., the sampling time of the inference input data used in the process of obtaining the first accuracy information; the sampling domain of the data used, i.e., the sampling domain of the inference input data used in the process of obtaining the first accuracy information; the variance of the data used, i.e., the variance of the inference input data used in the process of obtaining the first accuracy information; and the mean value of the data used, i.e., the variance of the mean value of the inference input data used in the process of obtaining the first accuracy information.

[0065] It should be noted that since each data can be used to perform one inference using the first model, the number of data used to obtain the first accuracy information may be understood as the number of inferences performed to obtain the first accuracy information, or the number of inferences performed when obtaining the first accuracy information.

[0066] It should be clarified that in this application the number of times an inference is performed may be understood as the number of inference outputs or the number of analytics outputs.

[0067] It should be explained that in this application, performing inference means the same as making inference and may be substituted for each other, and thereafter will not be further explained.

[0068] It should be noted that the meanings of inference and analytics in this application are the same and may be substituted for each other, for example, the meanings of performing inference and performing analysis are the same, the meanings of the number of times inference is performed and the number of times analysis is performed are the same, the meanings of inference output and analysis output are the same, or the meanings of the number of times inference output and the number of times analysis output are the same, and thereafter, no further explanation is given.

[0069] If the number of data used in the first information is 10, after the first device transmits the first information to the second or third device, if the second or third device determines from the first information that there is a risk that the deviation between the first accuracy information of the first model and the actual accuracy is too large because the number of samples is too small or the number of inferences is too small, the second or third device will not perform the corresponding operation.

[0070] Of course, the first information may further include other information of the inference input data used in the process of obtaining the first accuracy information, which will not be listed here.

[0071] In one embodiment, the first device transmits the corresponding first information at the same time as transmitting the first accuracy information of the first model, or the first device transmits the corresponding first information before or after transmitting the first accuracy information of the first model.

[0072] Optionally, in this embodiment, the first device includes an analysis logic network element, or a network data analysis network element having an analysis logic function.

[0073] Here, the analysis logic network element is AnLF, and the network data analysis network element having analysis logic function is NWDAF containing AnLF.

[0074] Optionally, in this embodiment, the second device includes a terminal equipment (UE) or a consumer network element (NF).

[0075] Optionally, in this embodiment, the third device includes a model training logic network element or a network data analysis network element with model training logic functionality.

[0076] Here, the model training logic network element is a MTLF, and the network data analysis network element with the model training logic function is an NWDAF containing a MTLF.

[0077] Optionally, transmitting first information corresponding to first accuracy information of the first model from the first device to the second device or the third device includes: If the data used when the first device obtains the first accuracy information of the first model satisfies second information, the first device transmits the first information to the second device or the third device, and the second information is used to describe a requirement for the data to be used.

[0078] It should be noted that the data used when the first device obtains the first accuracy information of the first model satisfies the second information may also be understood as the number of times the first device performs inference when obtaining the first accuracy information of the first model satisfies the second information.

[0079] Here, the second information is used to describe the requirements for the data used, i.e., the second information can limit the transmission of the first information by explaining or describing the demand for data used when obtaining the first accuracy information, thereby avoiding invalid transmission of the first information.

[0080] It should be noted that the second information used to describe requirements for the data to be used may also be understood as the second information used to describe requirements for the number of times the inference is performed.

[0081] For example, if the second information includes a threshold for the number of pieces of data used, and this threshold is 1000, the first device transmits first information corresponding to the first accuracy information of the first model, such as the sampling time of the data used to obtain the first accuracy information of the first model, only when the number of pieces of data used to obtain the first accuracy information of the first model is equal to or greater than 1000. In this example, the number of pieces of data used being equal to or greater than 1000 may be understood to mean that the number of times inference is performed is equal to or greater than 1000.

[0082] Optionally, if the data used by the first model to perform inference satisfies third information, the first device transmits the inference results obtained using the first model to the second device or the third device, and the third information is used to describe the requirements for the data to be used.

[0083] It should be noted that the data used by the first model when performing inference satisfies the third information may be understood to mean that the number of times the first model performs inference satisfies the third information.

[0084] Here, the third information is used to describe the requirements for the data used, i.e., the third information can describe the demand for data used by the first model when performing inference and limit the transmission of the inference result. Here, the inference result is inference result data. In this way, the second device or the third device that receives the inference result can know that the data used by the first model when performing inference corresponding to this inference result satisfies the third information, and the second device or the third device can directly determine whether to use this inference result even without receiving the first information.

[0085] It should be noted that the third information used to describe requirements for the data to be used may also be understood to be used to describe requirements for the number of times the inference is performed.

[0086] For example, if the third information includes restriction information on the sampling area of ​​the data used, and if the restriction information on the sampling area is area 1, the first device will transmit the inference results obtained when the first model performs inference only if the area of ​​the data used when the first model performs inference belongs to area 1.

[0087] It should be noted that in this embodiment, if the first device transmits the first information based on the second information if the data used to obtain the first accuracy information of the first model satisfies the second information, the first device can transmit the inference result regardless of whether the data used to perform the inference by the first model satisfies the third information. If the first device transmits the inference result based on the third information if the data used to perform the inference by the first model satisfies the third information, the first device can transmit the first information regardless of whether the data used to obtain the first accuracy information of the first model satisfies the second information. At this time, the second device or the third device can determine whether to use the inference result based on the number of inferences.

[0088] In this embodiment, the second information and the third information may be configured by the first device itself, or may be configured by another device. Optionally, the method may further include: The first device receives the second information or the third information transmitted from the second device; or The method further includes the first device receiving the second information or the third information transmitted from the third device.

[0089] Optionally, the first device receiving the second information or the third information transmitted from the second device includes: receiving, by the first device, a first request transmitted from the second device; Here, the first request carries the second information or the third information, and the first request is used to trigger the first device to reason using the first model.

[0090] In this way, the first device responds to a request from the second device by transmitting at least one of the first information and the inference result. Of course, multiplexing the first request to carry the second information or the third information saves transmission overhead.

[0091] Here, the first request may be a Nnwdaf_AnalyticsSubscription_Subscribe message.

[0092] In this embodiment, the first request triggers inference for a specific task, and the first request includes at least one of an analytics ID, analytics filter information, and analytics target. The first device determines a first model based on the first request, for example, the first model can be determined by the analytics ID. The first device can determine a source device of inference input data corresponding to the first model, input data type information, output data type information, etc. In this embodiment, the second device triggers execution of the first model by the second device based on the first request, and the first model is a model corresponding to the analytics ID included in the first request.

[0093] Optionally, the first device receiving the second information or the third information transmitted from the third device includes: The first device receives model-related information including the second information or the third information transmitted from the third device; or The first device receives a second request sent from the third device, the second request carrying the second information or the third information, and the second request is used to request the first device to monitor the accuracy of the first model.

[0094] In this way, the first device responds to the request of the third device by transmitting at least one of the first information and the inference result. Of course, multiplexing the model-related information or the second request to carry the second information or the third information saves transmission overhead.

[0095] Here, the model-related information is fed back to the third device after the third device receives a fourth request sent from the first device, the fourth request being used to request the model-related information. The fourth request includes at least one of Analytics Filter Information and an analytics ID, and the analytics ID is used to determine the first model that needs to be used. The fourth request may be a Nnwdaf_MLModelProvision_Subscribe or Nnwdaf_MLModelInfo_Request message. However, the model-related information may also be fed back via a Nnwdaf_MLModelProvision_Notify or Nnwdaf_MLModelInfo_Response message. The model-related information includes information such as the analytics ID.

[0096] Here, the second request may be understood as a request from the third device to assist the first device in monitoring the accuracy of the first model. The second request may carry periodicity information for the first device to periodically transmit the first accuracy information of the first model to the third device. The second request may carry a predetermined condition for the first device to transmit the first accuracy information of the first model when the predetermined condition is satisfied.

[0097] Of course, the second information may be used to restrict the first device from transmitting the first accuracy information, and if the data used by the first device to obtain the first accuracy information of the first model satisfies the second information, the first device will transmit the first accuracy information to the second device or the third device.

[0098] In this embodiment, after receiving the first information, the second device can determine whether to use the inference results obtained by the first model based on the first information.

[0099] Specifically, the second device combines the first information with the first accuracy information of the first model to determine whether to use the inference result obtained by the first model. For example, the second device stops using the inference result when the accuracy of the first model decreases and the number of samples is smaller than a threshold value for the number of samples, or when the actual number of inferences is smaller than a threshold value for the number of inferences. Only when the number of samples is equal to or greater than the threshold value for the number of samples, or when the actual number of inferences is equal to or greater than the threshold value for the number of inferences, does the second device perform a corresponding operation, for example, use the inference result.

[0100] In this embodiment, after receiving the first information, the second device can determine whether to continue using the first device to perform inference based on the first information and the first accuracy information. If the accuracy of the first model decreases and the number of samples is smaller than a threshold value for the number of samples, or if the actual number of inferences is smaller than a threshold value for the number of inferences, the second device will not perform any operation, but will continue to perform inference using the first device using the first model. Only if the number of samples is equal to or greater than the threshold value for the number of samples, or if the actual number of inferences is equal to or greater than the threshold value for the number of inferences, will the second device perform a corresponding operation, such as stopping use of the first device.

[0101] In this embodiment, the third device determines second accuracy information of the first model based on the first information of N devices and the first accuracy information of the first model. Here, all of the N devices are devices that perform inference using the first model. The third device cooperates with the N devices and determines second accuracy information of the first model based on the first information of each device and the first accuracy information of the first model of each device. It should be noted that when all of the N devices perform inference using the first model, it may be understood that all of the N devices request the model from the third device using an analytics ID corresponding to the first model. Here, when the third device receives the first information of the first device and the first accuracy information of the first model, it requests the first information and the first accuracy information of the first model from at least one fifth device (a device among the N devices other than the first device, N is a positive integer greater than or equal to 2) via a third request. The third request may be understood to request at least one fifth device to assist in determining second accuracy information for the first model. Optionally, the third request may carry the second information described above, such that the at least one fifth device receiving the second information further transmits first information if the data used in obtaining the first accuracy information for the first model satisfies the second information. Optionally, the third request may further carry instruction information for instructing the transmission of the first information, facilitating the fifth device to transmit the first information to the third device based on the instruction information. Optionally, the third request may further carry instruction information for instructing the fifth device to immediately transmit the first information and the first accuracy information for the first model to the third device based on the instruction information, facilitating the fifth device to immediately transmit the first information and the first accuracy information for the first model to the third device based on the instruction information.

[0102] Here, the third request may be a Nnwdaf_MLModelMonitor_Subscribe message, and optionally, the value of a Reporting Period parameter included in this message is a special or specified value, for example, a value of 0, so as to facilitate the fifth device to immediately transmit the first information and the first accuracy information of the first model to the third device based on the special or specified value of this Reporting Period parameter.

[0103] Specifically, the second accuracy information Accuracy of the first model is

number

[0104] The second accuracy information may be referred to as global accuracy information or general accuracy information. The second accuracy may be understood as model accuracy, analytics accuracy, training accuracy, accuracy provided by a third device (e.g., MTLF), performance information of the first model, etc.

[0105] The third device determines whether the first model should be degraded based on the second accuracy information, or whether the first model needs to be updated based on the second accuracy information. For example, if the second accuracy information is smaller than a certain threshold, the third device determines that the first model should be degraded or that the first model needs to be upgraded.

[0106] The second accuracy information is accuracy information calculated based on the results fed back by multiple devices. The third device can determine the size of N according to the request. For example, the third device can request the number of devices participating in the accuracy calculation. For example, if the third device requests to calculate the second accuracy based on data from 10 devices, N is equal to 10. Or, if the third device requests to calculate the second accuracy based on data from 100 devices, N is equal to 100. Or, the third device can request information on data participating in the accuracy calculation. For example, if the third device requests to calculate the second accuracy based on data from 1000 devices, N is equal to 100. Or, the third device can request information on data participating in the accuracy calculation. For example, if the third device requests to calculate the second accuracy based on data from 1000 devices, N is equal to 100. If the total amount of data from the fifth device can exceed 1000, N is equal to 3; or if the total amount of data from the first device and the four fifth devices can exceed 1000, N is equal to 5; or the third device can request information on the number of inferences participating in the accuracy calculation. For example, if the third device requests to calculate the second accuracy based on 1000 inference results, N is equal to 3 if the total number of inferences from the first device and the two fifth devices can exceed 1000; or if the total number of inferences from the first device and the four fifth devices can exceed 1000, N is equal to 5. Of course, the third device can simultaneously request information on the number of devices participating in the accuracy calculation and the data participating in the accuracy calculation, or the third device can simultaneously request information on the number of devices participating in the accuracy calculation and the number of inferences participating in the accuracy calculation.

[0107] Optionally, in this embodiment, the method further comprises: the first device transmitting fourth information to a fourth device, the fourth information being used to describe a request for training data used to train a first model; or The method further includes the first device transmitting fifth information to a fourth device, the fifth information being used to describe information in the data that can be used when making inferences using the first model.

[0108] It should be noted that each training data can be used to perform one training run on the first model, and each training data may include one or more parameters, among which one or more parameters are input parameters required when performing the first model training run.

[0109] It should be explained that the fourth information is used to describe requirements for training data used in training the first model may also be understood as the fourth information being used to describe requirements for training times of training the first model.

[0110] It should be noted that the fifth information being used to describe information of data that can be used when making inferences using the first model may also be understood to be used to describe information of the number of times that inferences can be made when making inferences using the first model.

[0111] It should be mentioned that in this application, executing training means the same as conducting training and may be interchangeable, and will not be further explained thereafter.

[0112] Here, the fourth device may be a Network Repository Function (NRF).

[0113] Thus, when the device generating the first model sends a registration request to the NRF, it carries seventh information, which is used to describe information about the data used to train the first model. After the first device sends the fourth information to the NRF, the NRF can determine a corresponding third device for the first device based on the fourth information. However, the first device can carry the fifth information when sending a registration request to the NRF, which allows the NRF to determine that it can provide the necessary inference results to the second device based on the fifth information. The registration request may be an Nnrf_NFManagement_NFRegister message. The request sent by the first device to the NRF may be an Nnrf_NFDiscovery request.

[0114] It should be explained that the seventh information being used to describe information about data used when training the first model may also be understood as the seventh information being used to describe information about the number of training times performed when training the first model.

[0115] Optionally, in this embodiment, the first accuracy information of the first model is: a ratio of the number of correct predictions of the first model to the number of total predictions of the first model; the root mean square error of the first model; the recall rate of the first model; and and the F1 score of the first model.

[0116] For clarity, the recall rate may also be referred to as the recall degree. The recall rate is the ratio of the number of times a model correctly predicts a certain type to the number of data that actually represent that type, i.e., the proportion of samples that are correctly predicted as "A" among samples that are actually "A." This may be used to indicate whether the model's inference and prediction results are complete or comprehensive. For example, the recall rate may indicate how many samples are correctly predicted as "A" among samples that are actually "A," and the "A" may be any one type, such as the cell in which the UE is located or the network load level.

[0117] Recall rate = number of items predicted to be "A" and actually being "A" ÷ number of items that actually are "A".

[0118] The accuracy is the ratio of the number of times the model correctly predicts a certain type to the number of types for which the model prediction results are true, i.e., the ratio of samples correctly predicted as "A" to samples inferred (predicted) as "A," and may be used to represent the accuracy of model inference. For example, among samples predicted as "A" by the model, how many are true "A" samples, and the "A" may be any one type, such as the cell in which the UE is located or the network load level.

[0119] Accuracy = number of items predicted as "A" and actually "A" ÷ number of items predicted as "A" Precision and recall are trade-offs, i.e., the higher the precision, the lower the recall. In some scenarios, there is an F1 score that strikes a balance between precision and recall.

[0120] The F1 value is a comprehensive evaluation of precision and recall, and if both precision and recall are high, the F1 value will also be high.

[0121] The formula for F1 is:

number

[0122] Optionally, in this embodiment, the second information, the third information, the fifth information, or the sixth information corresponding to the first model is: A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and and a threshold for the average value of the data used.

[0123] It should be explained that each data can perform one inference by the first model, so the threshold number of data used may be understood as the threshold number of times to perform inference.

[0124] Optionally, in this embodiment, the fourth information or the seventh information is: The number of data used and The number of training runs, The sampling time period of the data used, and the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0125] It should be noted that the number of data used may be understood as the number of training runs, since each data can be trained once by the first model.

[0126] The application of the method of the embodiment of the present application will be described below in conjunction with a specific scenario.

[0127] As shown in Figure 4, the specific flow includes: Steps 1 and 2 are the same as steps 1 and 2 in FIG.

[0128] 3. The Consumer NF sends a first request to the AnLF, where the first request includes an analytics ID, analytics filter information, analytics target, etc. The first request further includes second information or third information. For example, the third information is a first threshold, i.e., a threshold for the number of data to be used, which may be understood as a threshold for the number of samples required, the minimum number of samples required, or the number of times inference is performed. Only when the number of samples in the AnLF is equal to or greater than this first threshold, the inference result is sent to the consumer NF, or only when the number of times inference is performed is equal to or greater than this first threshold. Of course, the first request does not have to include the second information or the third information.

[0129] This first request may be a Nnwdaf_AnalyticsSubscription_Subscribe message.

[0130] Step 4 is the same as step 4 in FIG.

[0131] 5. The MTLF sends model-related information and the accuracy of the model (AiT) to the AnLF.

[0132] The model-related information includes the second information or the third information. For example, the second information is a second threshold, i.e., a threshold for the number of data to be used, which may be understood as the number of samples required, the minimum number of samples required, or the number of times inference is performed. Only when the number of samples of the AnLF is equal to or greater than this second threshold, or the number of times the AnLF performs inference is equal to or greater than this second threshold, is the first information transmitted to the consumer NF. Of course, the model-related information does not have to include the second information or the third information.

[0133] NOTE: Steps 1 and 2 may be performed after step 4.

[0134] Steps 6 to 8 are the same as steps 6 to 8 in FIG.

[0135] 9. The AnLF transmits the inference result data obtained by the inference to the consumer NF.

[0136] The inference result data may be used to inform the consumer NF that the statistics or predicted values ​​obtained by the inference of the first model are used to support the consumer NF in making corresponding policy decisions. For example, the statistics or predicted values ​​corresponding to UE mobility may be used to support user paging optimization by the AMF.

[0137] In case 1, the AnLF transmits the inference result to the consumer NF only if the data used by the first model when performing inference satisfies third information (e.g., the number of data used (i.e., the number of samples) is equal to or greater than a first threshold, or the number of inferences performed is equal to or greater than a first threshold). This threshold may be a request from the consumer NF in step 3, or may be model-related information transmitted from the MTLF in step 5, or may be configured in the AnLF itself.

[0138] In case 2, the AnLF transmits first information (e.g., the number of data used, i.e., the number of samples or the number of times to perform inference) to the consumer NF, and the consumer NF can determine whether to use the inference result of the AnLF based on the number of samples or the number of times to perform inference. If the consumer NF determines that the number of samples or the number of times to perform inference is smaller than a third threshold, it does not use the inference result of the AnLF. Only if the consumer NF determines that the number of samples or the number of times to perform inference is equal to or greater than the third threshold, it starts to use the inference result of the AnLF. The third threshold is a condition for determining whether to use the inference result.

[0139] Here, it is sufficient if there is a restriction in either case 1 or case 2.

[0140] What you need to know is that if the restriction in case 1 does not exist, the AnLF will send the inference result, if any, to the consumer NF, and at this time, the consumer NF will decide whether to use the inference result based on the inference number.

[0141] Steps 10 and 11 are the same as steps 10 and 11 in FIG.

[0142] 12. When the AnLF needs to send first accuracy information (e.g., AiU) to the consumer NF (e.g., when it determines that the first accuracy information does not meet the accuracy demand, or the accuracy is decreasing, or the periodic transmission time has arrived), the AnLF sends the first information to the consumer NF.

[0143] The message carrying the first information may be used to notify the consumer NF that the first accuracy information of the first model does not meet the accuracy demand or is degraded, or may be used to notify the consumer NF of the first accuracy information of the first model.

[0144] This message may be Nnwdaf_AnalyticsSubscription_Notify.

[0145] 13. The consumer NF can perform a corresponding operation based on the first information and the first accuracy information of the first model. For example, if the consumer NF determines that the accuracy of the inference is decreasing and the number of samples is smaller than a threshold number of samples or the number of times inference is performed is smaller than a threshold number of inferences, it stops using the inference result. Only if the consumer NF determines that the number of samples is equal to or greater than the threshold number of samples or the number of times inference is performed is equal to or greater than the threshold number of inferences, it performs a corresponding operation and uses the inference result.

[0146] 14. When the AnLF needs to transmit first accuracy information to the MTLF (for example, when it determines that the first accuracy information does not meet the accuracy demand, or that the accuracy is deteriorating, or that the periodic transmission time has arrived), the AnLF transmits the first information to the MTLF.

[0147] The message carrying the first information may be used to notify the MTLF that the first accuracy information of the first model does not meet the accuracy demand or is degraded, or may be used to notify the consumer NF of the first accuracy information of the first model.

[0148] The first information that the AnLF transmits to the consumer NF and the MTLF may be the same or different.

[0149] Steps 15 and 16 are the same as steps 15 and 16 in FIG.

[0150] Also, as shown in FIG. 5 (in FIG. 5, step 2 corresponds to step 14 in FIG. 4, i.e., the AnLF in FIG. 4 is AnLF-1 in FIG. 5), the MTLF can further perform the following:

[0151] 1. Optionally, the MTLF sends a second request to AnLF-1 to request the AnLF to assist in monitoring the first accuracy information of the first model (assist ML Model Accuracy monitoring). Of course, the MTLF can consider other AnLFs using the first model as first devices and send the second request to them. Each AnLF can periodically send the first accuracy information of the first model to the MTLF, or send the first inference accuracy information when it finds that the first inference accuracy information of the first model is deteriorating.

[0152] Optionally, the MTLF includes second information or third information in these requests. For example, if the request includes second information, i.e., a threshold value for the number of data used or a threshold value for the number of inferences to be performed (e.g., a third threshold value), the AnLF reports the first accuracy information or the first information of the first model to the MTLF only if, when inferring using the first model, the number of samples used in inference or the number of inferences performed is equal to or greater than the third threshold value.

[0153] 2. When the AnLF-1 report period arrives or when the inference accuracy of the first model decreases, the AnLF-1 reports the first accuracy information of the first model to the MTLF and also reports first information (e.g., the number of data used when obtaining this first accuracy information or the number of inferences performed when obtaining this first accuracy information).

[0154] 3. When the MTLF determines that a predetermined condition is met, for example, when the first accuracy information of the first model of AnLF-1 has decreased, the MTLF sends a third request to other AnLFs that use the first model, requesting the AnLFs to immediately send the first accuracy information of the first model and the first information to the MTLFs.

[0155] 4. At least one AnLF sends the first accuracy information and the first information of each first model to the MTLF.

[0156] 5. The MTLF comprehensively judges the second accuracy information of the first model based on the first accuracy information and the first information of the first models of the multiple AnLFs, and determines whether retraining is necessary.

[0157] The second accuracy information of the first model is

number

[0158] Here, the third request may be a Nnwdaf_MLModelMonitor_Subscribe message, and optionally, the value of a Reporting Period parameter included in this message is a special or specified value, for example, a value of 0, so as to facilitate the fifth device to immediately transmit the first information and the first accuracy information of the first model to the third device based on the special or specified value of this Reporting Period parameter.

[0159] As shown in FIG. 6, after step 2 of FIG. 3, the MTLF may send a registration request to the NRF, which includes the analytics IDs supported by the MTLF, the MTLF's service range information, the supported analysis delay, and model filtering information (slice identifier, area of ​​interest). The registration request further includes seventh information for describing information about data used in training the first model. The registration request may be an Nnrf_NFManagement_NFRegister_request message. After receiving the registration request, the NRF sends a response message, which may be an Nnrf_NFManagement_NFRegister_respoonse message. The AnLF sends a discovery request to the NRF, which includes the analytics IDs, model filtering information, and fourth information, which is used to describe a request for training data used in training the first model or a request for the number of training times performed in training the first model. The discovery request may be an Nnrf_NFDiscovery_Request message. The NRF returns a response message containing the MTLF address based on the AnLF discovery request, which may be a Nnrf_NFDiscovery_Response message.

[0160] Of course, the registration request does not have to include the seventh information, and the discovery request does not have to include the fourth information.

[0161] Alternatively, before step 3 shown in FIG. 3 , as shown in FIG. 7 , the AnLF sends a registration request to the NRF, where the registration request includes an analytics ID supported by the AnLF, service range information of the AnLF, supported analysis delay, model filtering information (slice identifier, area of ​​interest (AOI)), etc. The registration request further includes fifth information, which is used to describe information on data that can be used when inferring using the first model or information on the number of inferences that can be performed when inferring using the first model. The registration request may be an Nnrf_NFManagement_NFRegister_request message. After receiving the registration request, the NRF sends a response message, which may be an Nnrf_NFManagement_NFRegister_respoonse message. The Consumer NF sends a discovery request to the NRF, where the discovery request includes an analytics ID, model filtering information, and information for describing a request for inference data to be used for inferring the first model. This discovery request may be an Nnrf_NFDiscovery_Request message. The NRF returns a response message including an AnLF address based on the discovery request of the Consumer NF. This response message may be an Nnrf_NFDiscovery_Response message.

[0162] In summary, the method of the embodiment of the present application achieves the accuracy of accurately evaluating model inference and avoids affecting the use of the inference results.

[0163] As shown in FIG. 8, the information transmission method according to the embodiment of the present application includes the following steps: Step 801: A second device triggers execution of a first model by a first device; In step 802, the second device receives first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0164] Here, the second device triggers the execution of the first model by the first device, i.e., the first model makes an inference, and after the first device sends the first information to the second device, the second device can know the information of the data used to obtain the first accuracy information of the first model or the information of the number of training times performed to obtain the first accuracy information of the first model, thereby combining it with the first information for subsequent processing, and avoiding the problem of the deviation of the first accuracy information being too large, which affects the use of the inference result.

[0165] Optionally, the first information is: The number of data used and the number of times the first model performs inference; The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0166] Optionally, receiving, by the second device, first information corresponding to first accuracy information of the first model transmitted from the first device includes: If the data used when the first device obtains first accuracy information of the first model satisfies second information, the second device receives the first information, and the second information is used to describe requirements for the data to be used.

[0167] It should be explained that the data used when the first device obtains the first accuracy information of the first model satisfies the second information may be understood as the number of inferences performed when the first device obtains the first accuracy information of the first model satisfies the second information. The method further includes the second device receiving the inference results obtained using the first model transmitted from the first device if the data used by the first model to perform inference satisfies third information, wherein the third information is used to describe requirements for the data to be used.

[0168] It should be noted that the data used by the first model when performing inference satisfies the third information may be understood to mean that the number of times the first model performs inference satisfies the third information.

[0169] Optionally, the method further comprises: The method further includes the second device transmitting the second information or the third information to the first device.

[0170] Optionally, the second device transmitting the second information or the third information to the first device includes: the second device sending a first request to the first device; Here, the first request carries the second information or the third information, and the first request is used to trigger the first device to reason using the first model.

[0171] Optionally, the method further comprises: The method further includes the second device determining, based on the first information, whether to use inference results obtained by the first model.

[0172] Optionally, the method further comprises: receiving, by the second device, first accuracy information of the first model corresponding to the first information transmitted from the first device; The second device further includes determining, based on the first accuracy information and the first information, whether to continue reasoning using the first device.

[0173] Optionally, the method further comprises: The method further includes the second device transmitting sixth information to the fourth device, the sixth information being used to describe a request for data that can be used when reasoning using the first model.

[0174] It should be noted that the sixth information used to describe requirements for data that can be used when reasoning using the first model may also be understood to mean that the sixth information used to describe requirements for the number of inferences that can be performed when reasoning using the first model.

[0175] Here, the second device transmits the sixth information to a fourth device (e.g., an NRF), so that the fourth device can provide the first device for adaptive reasoning to the second device, and the first device executes the first model to make the inference.

[0176] Optionally, the second information, the third information or the sixth information corresponding to the first model is: A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and and a threshold for the average value of the data used.

[0177] It should be explained that each data can perform one inference by the first model, so the threshold number of data used may be understood as the threshold number of times to perform inference.

[0178] Optionally, the second device comprises a terminal device or a consumer network element.

[0179] It should be noted that this method is realized in combination with the method performed by the first device, and the implementation manner of the embodiment of the above method can be applied to this method to achieve the same technical effect, and will not be further described here.

[0180] As shown in FIG. 9, the information transmission method according to the embodiment of the present application includes the following steps: Step 901: A third device transmits a first model to a first device; Specifically, it is the model information of the first model transmitted from the third device.

[0181] In step 902, the third device receives first information corresponding to the first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0182] Here, the third device transmits the first model to the first device, and the first device executes the first model and transmits the first information to the third device, so that the third device can know the information of the data used to obtain the first accuracy information of the first model, and thereby combine it with the first information for subsequent processing, thereby avoiding the problem of the deviation of the first accuracy information being too large, which affects the use of the inference results.

[0183] Optionally, the first information is: The number of data used and The number of times you perform the inference, The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0184] Optionally, receiving, by the third device, first information corresponding to first accuracy information of the first model transmitted from the first device includes: If the data used when the first device obtains first accuracy information of the first model satisfies second information, the third device receives the first information, and the second information is used to describe requirements for data to be used or requirements for the number of times inference is performed or requirements for the number of times inference is performed using the first model.

[0185] Optionally, the method further comprises: The method further includes the third device receiving the inference results obtained using the first model sent from the first device if the data used by the first model when performing inference satisfies third information, wherein the third information is used to describe requirements for the data to be used or the number of times to perform inference.

[0186] Optionally, the method further comprises: The third device may further transmit the second information or the third information to the first device.

[0187] Optionally, the third device transmitting the second information or the third information to the first device includes: the third device transmitting model-related information including the second information or the third information to the first device; or The third device sends a second request to the first device, the second request carrying the second information or the third information, and the second request is used to request the first device to monitor the accuracy of the first model.

[0188] Optionally, the method further comprises: The method further includes receiving, by the third device, first accuracy information of the first model corresponding to the first information transmitted from the first device.

[0189] Optionally, the method further comprises: The third device further includes determining second accuracy information of the first model based on the first information of the N devices and the first accuracy information of the first model; wherein the N devices include the first device and at least one fifth device that uses the first model; The N is a positive integer of 2 or more.

[0190] Optionally, the method further comprises: The method further includes the third device sending a third request to the at least one fifth device when receiving the first information of the first device and the first accuracy information of the first model, wherein the third request is used to request the first information of the at least one fifth device and the first accuracy information of the first model.

[0191] Optionally, the third request includes second information describing the request for data to be used or a request for the number of times to perform the inference.

[0192] Optionally, the third request may further carry instruction information for instructing transmission of the first information, facilitating the fifth device transmitting the first information to the third device based on the instruction information. Optionally, the third request may further carry instruction information for instructing immediate transmission, facilitating the fifth device immediately transmitting the first information and first accuracy information of the first model to the third device based on the instruction information.

[0193] Here, the third request may be a Nnwdaf_MLModelMonitor_Subscribe message, and optionally, the value of a Reporting Period parameter included in this message is a special or specified value, for example, a value of 0, so as to facilitate the fifth device to immediately transmit the first information and the first accuracy information of the first model to the third device based on the special or specified value of this Reporting Period parameter.

[0194] Optionally, the third device determines second accuracy information of the first model based on the first information of the N devices and the first accuracy information of the first model, formula

number

[0195] The third device determines whether the first model should be degraded based on the second accuracy information, or whether the first model needs to be updated based on the second accuracy information. For example, if the second accuracy information is smaller than a certain threshold, the third device determines that the first model should be degraded or that the first model needs to be upgraded.

[0196] Optionally, the method further comprises: The method further includes the third device transmitting seventh information to the fourth device, wherein the seventh information is used to describe information of data used when training the first model or information of the number of training rounds performed when training the first model.

[0197] Here, the third device transmits the seventh information to a fourth device (e.g., NRF) to facilitate the NRF recommending the third device to an appropriate first device by knowing information about the data used when it trains the first model.

[0198] Optionally, this seventh information is conveyed to the NRF via a registration request sent from the third device.

[0199] Optionally, the seventh information is: The number of data used and The number of training runs, The sampling time period of the data used, and the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0200] It should be noted that the number of data used may be understood as the number of training runs, since each data can be trained once by the first model.

[0201] Optionally, the method further comprises: The method further includes the third device determining whether to retrain the first model based on the second accuracy information.

[0202] For example, a fourth threshold corresponding to the second accuracy information is set in advance, and after the third device determines the second accuracy information, it compares the second accuracy information with the fourth threshold. If the second accuracy information is smaller than the fourth threshold, it retrains the first model to improve the accuracy of the first model inference; and if the second accuracy information is equal to or greater than the fourth threshold, it does not need to retrain the first model.

[0203] Optionally, the second information or third information corresponding to the first model is: A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and and a threshold for the average value of the data used.

[0204] It should be explained that each data can perform one inference by the first model, so the threshold number of data used may be understood as the threshold number of times to perform inference.

[0205] Optionally, the third device includes a model training logic network element or a network data analysis network element with model training logic functionality.

[0206] It should be noted that this method is realized in combination with the method performed by the first device, and the implementation manner of the embodiment of the above method can be applied to this method to achieve the same technical effect, and will not be further described here.

[0207] In the information transmission method according to the embodiment of the present application, the execution body may be an information transmission device. In the embodiment of the present application, the information transmission device according to the embodiment of the present application will be described by taking the execution of the information transmission method by the information transmission device as an example.

[0208] As shown in FIG. 10, an information transmission device 1000 according to an embodiment of the present application includes: a first transmission module 1010 for transmitting first information corresponding to the first accuracy information of the first model to a second device or a third device, the first information being used to describe information used to obtain the first accuracy information; Here, the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model.

[0209] After this device sends the first information to a second device (the device that triggers the first device to execute the first model) or a third device (the device that generates the first model), the second device or the third device can know the information of the data used to obtain the first accuracy information of the first model, and thereby combine it with the first information for subsequent processing, thereby avoiding the problem of the first accuracy information having too large a deviation, which affects the use of the inference result.

[0210] Optionally, the first information is: The number of data used and The number of times you perform the inference, The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0211] Optionally, the first transmitting module further comprises: If the data used when the first device obtains the first accuracy information of the first model satisfies the second information, it is used to send the first information to the second device or the third device, and the second information is used to describe the requirements for the data to be used or the requirements for the number of times to perform inference or the requirements for the number of times to perform inference using the first model.

[0212] Optionally, the device comprises: The device further includes a third transmission module for transmitting the inference results obtained using the first model to the second device or the third device if the data used by the first model when performing inference satisfies third information, wherein the third information is used to describe requirements for the data to be used or the number of times inference is performed.

[0213] Optionally, the device comprises: a third receiving module for receiving the second information or the third information transmitted from the second device; and a fourth receiving module for receiving the second information or the third information transmitted from the third device.

[0214] Optionally, the third receiving module further comprises: a first request transmitted from the second device; Here, the first request carries the second information or the third information, and the first request is used to trigger the first device to reason using the first model.

[0215] Optionally, the fourth receiving module further comprises: receiving model-related information transmitted from the third device, the model-related information including the second information or the third information; or receiving a second request sent from the third device, the second request carrying the second information or the third information, the second request being used to request the first device to monitor the accuracy of the first model;

[0216] Optionally, the device comprises: a fourth transmission module for transmitting fourth information to a fourth device, the fourth information being used to describe a request for training data to be used in training the first model or a request for a number of times to run the training; and a fifth transmitting module for transmitting fifth information to the fourth device, the fifth information being used to describe information of the data that can be used when making inferences using the first model.

[0217] Optionally, the first accuracy information of the first model comprises: a ratio of the number of correct predictions of the first model to the number of total predictions of the first model; the root mean square error of the first model; the recall rate of the first model; and and the F1 score of the first model.

[0218] Optionally, the second information, third information or fourth information corresponding to the first model is: A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and and a threshold for the average value of the data used.

[0219] Optionally, the fifth information is: The number of data used and The number of times you perform the inference, The sampling time period of the data used, and the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0220] Optionally, the first device includes an analysis logic network element or a network data analysis network element having an analysis logic function.

[0221] The information transmission device according to the embodiment of the present application can realize each process realized by the method embodiments of Figures 3 to 7 and achieve the same technical effects, and will not be further described here to avoid repetition.

[0222] As shown in FIG. 11, an information transmission device 1100 according to an embodiment of the present application includes: a first processing module 1110 for triggering a first device to execute a first model; and a first receiving module 1120 for receiving first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0223] This device triggers the execution of a first model by a first device, i.e., the first model makes an inference, and after the first device sends first information to a second device, the second device can know the information of the data used to obtain the first accuracy information of the first model, thereby combining it with the first information for subsequent processing, and can avoid the problem of the deviation of the first accuracy information being too large, which affects the use of the inference result.

[0224] Optionally, the first information is: The number of data used and The number of times you perform the inference, The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0225] Optionally, the first receiving module further comprises: If the data used when the first device obtains first accuracy information of the first model satisfies second information, the first information is used to receive the first information, and the second information is used to describe requirements for data to be used or requirements for the number of times inference is performed or requirements for the number of times inference is performed using the first model.

[0226] Optionally, the device comprises: The device further includes a fifth receiving module for receiving inference results obtained using the first model transmitted from the first device when the data used by the first model to perform inference satisfies third information, and the third information is used to describe requirements for the data to be used or the number of times to perform inference.

[0227] Optionally, the device comprises: The device further includes a sixth transmitting module for transmitting the second information or the third information to the first device.

[0228] Optionally, the sixth transmitting module further comprises: used to send a first request to the first device; Here, the first request carries the second information or the third information, and the first request is used to trigger the first device to reason using the first model.

[0229] Optionally, the device comprises: The system further includes a second processing module for determining whether to use inference results obtained by the first model based on the first information.

[0230] Optionally, the device comprises: a sixth receiving module for receiving first accuracy information of the first model corresponding to the first information transmitted from the first device; and a third processing module for determining whether to continue reasoning using the first device based on the first accuracy information and the first information.

[0231] Optionally, the device comprises: Further comprising a seventh transmitting module for transmitting sixth information to the fourth device, the sixth information being used to describe a request for data that can be used when making inferences using the first model or a request for the number of times that inferences can be performed.

[0232] Optionally, the second information, the third information or the sixth information corresponding to the first model is: A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and and a threshold for the average value of the data used.

[0233] Optionally, the second device comprises a terminal device or a consumer network element.

[0234] The information transmission device according to the embodiment of the present application can implement each process implemented by the method embodiment of Figure 8 and achieve the same technical effect, and will not be further described here to avoid repetition of description.

[0235] As shown in FIG. 12, an information transmission device 1200 according to an embodiment of the present application includes: a second transmitting module 1210 for transmitting the first model to the first device; and a second receiving module 1220 for receiving first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

[0236] This device transmits a first model to a first device, and the first device executes the first model and transmits first information to a third device, so that the third device can know the information of the data used to obtain the first accuracy information of the first model, and thereby combine it with the first information for subsequent processing, thereby avoiding the problem of the deviation of the first accuracy information being too large, which affects the use of the inference results.

[0237] Optionally, the first information is: The number of data used and The number of times you perform the inference, The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0238] Optionally, the second receiving module further comprises: If the data used when the first device obtains first accuracy information of the first model satisfies second information, the first information is used to receive the first information, and the second information is used to describe requirements for data to be used or requirements for the number of times inference is performed or requirements for the number of times inference is performed using the first model.

[0239] Optionally, the device comprises: The device further includes a seventh receiving module for receiving inference results obtained using the first model transmitted from the first device when the data used by the first model to perform inference satisfies third information, and the third information is used to describe requirements for the data to be used or the number of times to perform inference.

[0240] Optionally, the device comprises: The device further includes an eighth transmitting module for transmitting the second information or the third information to the first device.

[0241] Optionally, the eighth transmitting module further comprises: transmitting model-related information including the second information or the third information to the first device; or Sending a second request to the first device, the second request carrying the second information or the third information, the second request being used to request the first device to monitor the accuracy of the first model.

[0242] Optionally, the device comprises: The device further includes an eighth receiving module for receiving first accuracy information of the first model corresponding to the first information transmitted from the first device.

[0243] Optionally, the device comprises: a fourth processing module for determining second accuracy information of the first model based on the first information of the N devices and first accuracy information of the first model; wherein the N devices include the first device and at least one fifth device that uses the first model; The N is a positive integer of 2 or more.

[0244] Optionally, the device comprises: The device further includes a ninth sending module for sending a third request to the at least one fifth device when receiving the first information of the first device and the first accuracy information of the first model, wherein the third request is used to request the first information of the at least one fifth device and the first accuracy information of the first model.

[0245] Optionally, the third request includes second information to describe the request for the data to be used.

[0246] Optionally, the third request may further carry instruction information for instructing transmission of the first information, facilitating the fifth device transmitting the first information to the third device based on the instruction information. Optionally, the third request may further carry instruction information for instructing immediate transmission, facilitating the fifth device immediately transmitting the first information and first accuracy information of the first model to the third device based on the instruction information.

[0247] Here, the third request may be a Nnwdaf_MLModelMonitor_Subscribe message, and optionally, the value of a Reporting Period parameter included in this message is a special or specified value, for example, a value of 0, so as to facilitate the fifth device to immediately transmit the first information and the first accuracy information of the first model to the third device based on the special or specified value of this Reporting Period parameter.

[0248] Optionally, the fourth processing module further comprises: formula

number

[0249] The third device determines whether the first model should be degraded based on the second accuracy information, or whether the first model needs to be updated based on the second accuracy information. For example, if the second accuracy information is smaller than a certain threshold, the third device determines that the first model should be degraded or that the first model needs to be upgraded.

[0250] Optionally, the device comprises: Further included is a tenth transmission module for transmitting seventh information to the fourth device, wherein the seventh information is used to describe information of data used when training the first model or information of the number of training times performed when training the first model.

[0251] Optionally, the seventh information is: The number of data used and The number of training runs, The sampling time period of the data used, and the sampling area of ​​the data used; The distribution of data used and and the average value of the data used.

[0252] Optionally, the device comprises: and a fifth processing module for determining whether to retrain the first model based on the second accuracy information.

[0253] Optionally, the second information or third information corresponding to the first model is: A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and and a threshold for the average value of the data used.

[0254] Optionally, the third device includes a model training logic network element or a network data analysis network element with model training logic functionality.

[0255] The information transmission device according to the embodiment of the present application can implement each process implemented by the method embodiment of Figure 9 and achieve the same technical effect, and will not be further described here to avoid repetition.

[0256] Optionally, as shown in FIG. 13 , an embodiment of the present application further provides a communication device 1300, which includes a processor 1301 and a memory 1302. The memory 1302 stores a program or instruction that can run on the processor 1301. For example, if the communication device 1300 is a first device, the program or instruction executed by the processor 1301 can realize the steps of the information transmission method embodiment executed by the first device and achieve the same technical effect. If the communication device 1300 is a second device, the program or instruction executed by the processor 1301 can realize the steps of the information transmission method embodiment executed by the second device and achieve the same technical effect. If the communication device 1300 is a third device, the program or instruction executed by the processor 1301 can realize the steps of the information transmission method embodiment executed by the third device and achieve the same technical effect. To avoid repetition, further description will not be provided here.

[0257] An embodiment of the present application further provides a communication device, the communication device including: a processor; and a communication interface, wherein the communication interface is used to transmit first information corresponding to first accuracy information of a first model to a second device or a third device, the first information being used to describe information used to obtain the first accuracy information; Here, the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model.

[0258] This communication device embodiment corresponds to the method embodiment on the first device side, and the implementation processes and realization methods of the method embodiment can all be applied to this terminal embodiment and can achieve the same technical effects.

[0259] An embodiment of the present application further provides a communication device, the communication device including a processor and a communication interface, wherein the processor is used to trigger a first device to execute a first model; The communication interface is used to receive first information corresponding to first accuracy information of the first model transmitted from the first device, and the first information is used to describe information used to obtain the first accuracy information.

[0260] This communication device embodiment corresponds to the method embodiment on the second device side, and the implementation processes and realization methods of the method embodiments can all be applied to this terminal embodiment and can achieve the same technical effects.

[0261] An embodiment of the present application further provides a communication device, the communication device including a processor and a communication interface, wherein the communication interface is used to transmit a first model to a first device, the communication interface is further used to receive first information corresponding to first accuracy information of the first model transmitted from the first device, and the first information is used to describe information used to obtain the first accuracy information.

[0262] This communication device embodiment corresponds to the third device-side method embodiment, and the implementation processes and realization methods of the method embodiments can all be applied to this terminal embodiment and can achieve the same technical effects.

[0263] Specifically, FIG. 14 is a schematic diagram of a hardware structure for realizing a terminal, which is a second device in the embodiment of the present application.

[0264] The terminal 1400 includes at least some components such as, but not limited to, a radio frequency unit 1401, a network module 1402, an audio output unit 1403, an input unit 1404, a sensor 1405, a display unit 1406, a user input unit 1407, an interface unit 1408, a memory 1409 and a processor 1410.

[0265] As will be understood by those skilled in the art, the terminal 1400 may further include a power source (e.g., a battery) for powering each component, and the power source may be logically connected to the processor 1410 by a power management system, thereby enabling the power management system to realize functions such as charge / discharge management and power consumption management. The terminal structure shown in Figure 14 does not constitute a limitation on the terminal, and the terminal may include more or fewer components than those shown, or a combination of some components, or a different arrangement of components, which will not be further described here.

[0266] It should be understood that in the embodiment of the present application, the input unit 1404 may include a graphics processing unit (GPU) 14041 and a microphone 14042, and the graphics processor 14041 processes image data of still or video images captured by an image capture device (e.g., a camera) in a video capture mode or an image capture mode. The display unit 1406 may include a display panel 14061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1407 includes at least one of a touch panel 14071 and other input devices 14072. The touch panel 14071 is also called a touch screen. The touch panel 14071 may include two parts: a touch detection device and a touch controller. The other input devices 14072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control buttons, switch buttons, etc.), a trackball, a mouse, and a control lever, which will not be further described herein.

[0267] In the embodiment of the present application, the radio frequency unit 1401 can receive downlink data from the network side device and then transmit the data to the processor 1410 for processing, and can also transmit uplink data to the network side device. Generally, the radio frequency unit 1401 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc.

[0268] The memory 1409 may be used to store software programs or instructions and various data. The memory 1409 may include a first storage area that mainly stores programs or instructions and a second storage area for stored data. Here, the first storage area may store an operating system, an application program or instructions necessary for at least one function (e.g., an audio playback function, an image playback function, etc.), etc. The memory 1409 may include volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. Here, the nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct Rambus random access memory (DRRAM). Memory 1409 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

[0269] The processor 1410 may include one or more processing units. Optionally, the processor 1410 may integrate an application processor and a modem processor, where the application processor mainly processes operations related to the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication signals, e.g., a baseband processor. As can be appreciated, the modem processor may not be integrated into the processor 1410.

[0270] The terminal triggers the first device to execute the first model, i.e., the first model performs inference. In this way, after the first device sends the first information to the second device, the second device can know the information of the data used to obtain the first accuracy information of the first model, and can combine it with the first information for subsequent processing, thereby avoiding the problem of the first accuracy information being too large, which affects the use of the inference result.

[0271] Specifically, an embodiment of the present application further provides a network side device. As shown in Fig. 15, the network side device 1500 includes a processor 1501, a network interface 1502, and a memory 1503. Here, the network interface 1502 is, for example, a common public radio interface (CPRI).

[0272] Specifically, the network side device 1500 of the embodiment of the present application further includes instructions or programs stored in memory 1503 and capable of running on the processor 1501, and the processor 1501 can call the instructions or programs in the memory 1503, execute the first device side method, the second device side method, or the third device side method, and achieve the same technical effect, which will not be described further here to avoid repetition of description.

[0273] An embodiment of the present application further provides a readable storage medium, which stores a program or instruction, and when the program or instruction is executed by a processor, it can realize each process of the embodiment of the information transmission method performed by the first device, or realize each process of the embodiment of the information transmission method performed by the second device, or realize each process of the embodiment of the information transmission method performed by the third device, and achieve the same technical effect, which will not be described further here to avoid repetition.

[0274] Wherein, the processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0275] An embodiment of the present application further provides a chip, the chip including a processor and a communication interface, the communication interface coupled to the processor, the processor running a program or instruction, to realize each process of the embodiment of the information transmission method performed by the first device, or to realize each process of the embodiment of the information transmission method performed by the second device, or to realize each process of the embodiment of the information transmission method performed by the third device, and can achieve the same technical effect, and in order to avoid repetition of description, no further description will be given here.

[0276] It should be understood that the chips referred to in the embodiments of this application may be referred to as system level chips, system chips, chip systems, or system-on-chips.

[0277] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium, and which can be executed by at least one processor to realize each process of the embodiment of the information transmission method performed by the first device, or to realize each process of the embodiment of the information transmission method performed by the second device, or to realize each process of the embodiment of the information transmission method performed by the third device, and achieve the same technical effects, and will not be described further here to avoid repetition.

[0278] An embodiment of the present application further provides an information transmission system, which includes a first device, a second device, and a third device, wherein the first device is used to perform steps of an information transmission method performed by the first device, the second device is used to perform steps of an information transmission method performed by the second device, and the third device is used to perform steps of an information transmission method performed by the third device.

[0279] It should be noted that, in this specification, the terms "comprise," "include," "includes," or any other variations thereof are intended to cover the non-exclusive "comprise," whereby a process, method, article, or apparatus comprising a set of elements not only includes those elements, but also other elements not expressly listed or inherent in such process, method, article, or apparatus. Absent further limitations, an element defined by the phrase "comprises one of" does not preclude the presence of other identical elements in the process, method, article, or apparatus comprising that element. It should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may include performing functions in an essentially simultaneous manner or in the reverse order based on the functions involved. For example, the described method may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to some examples may be combined in other examples.

[0280] As will be apparent to those skilled in the art from the above description of the embodiments, the methods of the above embodiments can be realized in the form of software and a necessary general-purpose hardware platform. Of course, they can also be realized in hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical proposal of the present application, in substance or in part contributing to the prior art, may be embodied in the form of a computer software product, which is stored in a storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and includes a number of instructions for causing a terminal (which may be a mobile phone, computer, server, air conditioner, network device, etc.) to execute the methods described in each embodiment of the present application.

[0281] Although the embodiments of the present application have been described above in conjunction with the drawings, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not limiting. Those skilled in the art can take the teachings of the present application into account and implement many forms without departing from the spirit and scope of the claims, all of which fall within the scope of protection of the present application.

Claims

1. 1. A method for transmitting information, comprising: transmitting, by the first device to the second device or the third device, first information corresponding to first accuracy information of the first model, the first information being used to describe information used to obtain the first accuracy information; Here, the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model, an information transmission method.

2. The first information is The number of data used and The number of times you perform the inference, The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and 2. The method of claim 1, comprising at least one of: a mean value of the data used;

3. transmitting, by the first device to the second device or the third device, first information corresponding to first accuracy information of the first model; If the data used when the first device obtains the first accuracy information of the first model satisfies second information, the first device sends the first information to the second device or the third device, and the second information is used to describe a requirement for the data used; or 3. The method of claim 1 or 2, further comprising: if the number of times an inference is performed when the first device obtains first accuracy information of the first model satisfies second information, the first device transmitting the first information to the second device or the third device, wherein the second information is used to describe a requirement for the number of times an inference is performed.

4. The method comprises: If the data used by the first model to perform the inference satisfies third information, the first device transmits the inference result obtained by using the first model to the second device or the third device, and the third information is used to describe the requirements for the data to be used; or 4. The method of claim 1, further comprising: if the number of times the first model performs inference satisfies third information, the first device transmits to the second device or the third device an inference result obtained using the first model, wherein the third information is used to describe a requirement for the number of times inference is performed.

5. The method comprises: The first device receives the second information or the third information transmitted from the second device; or The method of claim 3 or 4, further comprising the first device receiving the second information or the third information transmitted from the third device.

6. The first device receiving the second information or the third information transmitted from the second device includes: receiving, by the first device, a first request transmitted from the second device; 6. The method of claim 5, wherein the first request carries the second information or the third information, and the first request is used to trigger the first device to reason using the first model.

7. The first device receiving the second information or the third information transmitted from the third device includes: The first device receives model-related information including the second information or the third information transmitted from the third device; or 6. The method of claim 5, comprising receiving, by the first device, a second request sent from the third device, the second request carrying the second information or the third information, and the second request being used to request the first device to monitor accuracy of the first model.

8. The method comprises: the first device transmitting fourth information to a fourth device, the fourth information being used to describe a requirement for training data to be used in training the first model or a requirement for a number of training times to be performed in training the first model; or 8. The method of claim 1, further comprising the first device transmitting fifth information to a fourth device, the fifth information being used to describe information on data that can be used when making inferences using the first model or information on the number of inferences that can be performed when making inferences using the first model.

9. The first accuracy information of the first model is: a ratio of the number of correct predictions of the first model to the number of total predictions of the first model; the root mean square error of the first model; the recall rate of the first model; and 9. The method of claim 1, wherein the first model is characterized by at least one of: an F1 score of the first model;

10. second information, third information, or fourth information corresponding to the first model, A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and 10. The method of claim 1, further comprising at least one of: a threshold for the average value of the data used;

11. The fifth information is: The number of data used and The number of times you perform the inference, The sampling time period of the data used, and the sampling area of ​​the data used; The distribution of data used and 9. The method of claim 8, comprising at least one of: an average value of the data used;

12. The method of claim 1 , wherein the first device comprises an analysis logic network element or a network data analysis network element having analysis logic functionality.

13. 1. A method for transmitting information, comprising: the second device triggering execution of the first model by the first device; and receiving, by the second device, first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

14. The first information is The number of data used and The number of times you perform the inference, The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and 14. The method of claim 13, comprising at least one of: an average value of the data used;

15. receiving, by the second device, first information corresponding to first accuracy information of the first model transmitted from the first device; If the data used when the first device obtains the first accuracy information of the first model satisfies second information, the second device receives the first information, and the second information is used to describe requirements for the data used; or 15. The method of claim 13 or 14, further comprising: if the number of times to perform inference when the first device obtains first accuracy information of the first model satisfies second information, the second device receiving the first information, and the second information being used to describe a requirement for the number of times to perform inference.

16. The method comprises: If the data used by the first model when performing inference satisfies third information, the second device receives the inference result obtained using the first model transmitted from the first device, and the third information is used to describe the requirements for the data used; or 16. The method of claim 13, further comprising: if the number of times the first model performs inference satisfies third information, the second device receives an inference result obtained using the first model transmitted from the first device, wherein the third information is used to describe the requirement for the number of times inference is performed.

17. The method comprises: The method of claim 15 or 16, further comprising the second device transmitting the second information or the third information to the first device.

18. The second device transmitting the second information or the third information to the first device includes: the second device sending a first request to the first device; 18. The method of claim 17, wherein the first request carries the second information or the third information, and the first request is used to trigger the first device to reason using the first model.

19. The method comprises:

19. The method of any one of claims 13 to 18, further comprising the second device determining, based on the first information, whether to use inference results obtained by the first model.

20. The method comprises: receiving, by the second device, first accuracy information of the first model corresponding to the first information transmitted from the first device; 20. The method of any one of claims 13 to 19, further comprising the second device determining whether to continue reasoning using the first device based on the first accuracy information and the first information.

21. The method comprises:

21. The method of claim 13, further comprising the second device transmitting sixth information to the fourth device, wherein the sixth information is used to describe a requirement for data that can be used when reasoning using the first model, or the sixth information is used to describe a requirement for the number of inferences that can be performed when reasoning using the first model.

22. second information, third information, or sixth information corresponding to the first model, A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and and a threshold for the average value of the data used.

23. 22. The method of any one of claims 13 to 21, wherein the second device comprises a terminal device or a consumer network element.

24. 1. A method for transmitting information, comprising: transmitting the first model from the third device to the first device; and receiving, by the third device, first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

25. The first information is The number of data used and The number of times you perform the inference, The sampling time of the data used, the sampling area of ​​the data used; The distribution of data used and 25. The method of claim 24, comprising at least one of: an average value of the data used;

26. receiving, by the third device, first information corresponding to first accuracy information of the first model transmitted from the first device; If the data used when the first device obtains the first accuracy information of the first model satisfies second information, the third device receives the first information, and the second information is used to describe a requirement for the data used; or 26. The method of claim 24 or 25, further comprising: if the number of times to perform inference when the first device obtains first accuracy information of the first model satisfies second information, the third device receiving the first information, and the second information being used to describe a requirement for the number of times to perform inference.

27. The method comprises: If the data used by the first model when performing inference satisfies third information, the third device receives the inference result obtained using the first model transmitted from the first device, and the third information is used to describe the requirements for the data used; or 27. The method of claim 24, further comprising: if the number of times the first model performs inference satisfies third information, the third device receives the inference results obtained using the first model transmitted from the first device, and the third information is used to describe the requirement for the number of inferences.

28. The method comprises:

28. The method of claim 26 or 27, further comprising the third device transmitting the second information or the third information to the first device.

29. The third device transmitting the second information or the third information to the first device includes: the third device transmitting model-related information including the second information or the third information to the first device; or 29. The method of claim 28, comprising the third device sending a second request to the first device, the second request carrying the second information or the third information, the second request being used to request the first device to monitor accuracy of the first model.

30. The method comprises:

30. The method of any one of claims 24 to 29, further comprising receiving, by the third device, first accuracy information of the first model corresponding to the first information transmitted from the first device.

31. The method comprises: The third device further includes determining second accuracy information of the first model based on the first information of the N devices and the first accuracy information of the first model; wherein the N devices include the first device and at least one fifth device that uses the first model; 31. The method of any one of claims 24 to 30, wherein N is a positive integer greater than or equal to 2.

32. The method comprises: The method of claim 31, further comprising the third device sending a request to the at least one fifth device upon receiving the first information of the first device and the first accuracy information of the first model, wherein the third request is used to request the first information of the at least one fifth device and the first accuracy information of the first model.

33. 33. The method of claim 32, wherein the third request includes second information to describe a request for data to be used.

34. The third device determines second accuracy information of the first model based on the first information of the N devices and first accuracy information of the first model, formula [Equation 6] calculating second accuracy information Accuracy of the first model by Here, Num ｉ is the number of data provided by the i-th device among the N devices or the number of inferences performed, and Accuracy ｉ 34. The method of any one of claims 31 to 33, wherein i is the first accuracy information of the first model of the i-th device among the N devices.

35. The method comprises:

35. The method of any one of claims 24 to 34, further comprising the third device transmitting seventh information to the fourth device, wherein the seventh information is used to describe information of data used when training the first model or information of the number of training rounds performed when training the first model.

36. The seventh information is The number of data used and The number of training runs, The sampling time period of the data used, and the sampling area of ​​the data used; The distribution of data used and 36. The method of claim 35, comprising at least one of: an average value of the data used;

37. The method comprises:

35. The method of any one of claims 31 to 34, further comprising the third device determining whether to retrain the first model based on the second accuracy information.

38. The second information or the third information corresponding to the first model is A threshold for the number of data used; A threshold for the number of times that inferences are performed; Restrictions on the sampling time period of the data used; Information on the sampling area limitations of the data used; The threshold for the variance of the data used, and 38. The method of any one of claims 24 to 37, comprising at least one of: a threshold on the average value of the data used.

39. 39. The method of any one of claims 24 to 38, wherein the third device comprises a model training logic network element or a network data analysis network element with model training logic functionality.

40. An information transmission device, a first transmitting module for transmitting first information corresponding to first accuracy information of the first model to a second device or a third device, the first information including the first transmitting module being used to describe information used to obtain the first accuracy information; Here, the second device is a device that triggers the first device to execute the first model, and the third device is a device that generates the first model, an information transmission device.

41. An information transmission device, a first processing module for triggering the first device to execute the first model; and a first receiving module for receiving first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

42. An information transmission device, a second transmitting module for transmitting the first model to the first device; and a second receiving module for receiving first information corresponding to first accuracy information of the first model transmitted from the first device, the first information being used to describe information used to obtain the first accuracy information.

43. A communications device comprising a processor and a memory, the memory storing a program or instructions operable on the processor, the program or instructions, when executed by the processor, realizing the information transmission method of any one of claims 1 to 12, or the information transmission method of any one of claims 13 to 23, or the steps of the information transmission method of any one of claims 24 to 39.

44. A readable storage medium having a program or instructions stored therein, the program or instructions being executed by a processor to implement the information transmission method of any one of claims 1 to 12, or the information transmission method of any one of claims 13 to 23, or the steps of the information transmission method of any one of claims 24 to 39.

Images