Methods and apparatus for transmitting information

Abstract

This application provides a method and apparatus for transmitting information. The method includes: a first network device sending a first type of feedback information to a second network device. The first type includes one or more of the following: model evaluation metrics, operation instruction information, and network performance metrics. The model evaluation metrics indicate the performance of the model, the operation instruction information indicates the operation to be performed based on the analysis results of the model, and the network performance metrics indicate the network metrics obtained based on the analysis results of the model. The first type is used to request feedback information; the first network device receives the feedback information from the second network device; and the first network device determines the performance of the model based on the feedback information. The method for transmitting information provided by this application enables the first network device to configure the type of feedback information according to its own needs and the capabilities of the second network device, thereby improving the flexibility of feedback.

Description

Technical Field

[0001] This application relates to the field of communications, and more specifically, to a method and apparatus for transmitting information. Background Technology

[0002] Management Data Analytics (MDA) services process and analyze network, service event, and status-related data (e.g., performance measurement data, Minimization of Drive Tests (MDT), alarm data, configuration data, network analysis data, etc.) and provide analysis reports. The functions of the MDAS producer primarily include two parts: Machine Learning (ML) model training and management data analysis. Due to the complexity of ML models and real-world networks, model performance (e.g., accuracy) gradually declines over time. Since MDAS analysis results are generated based on ML models, timely verification of the analysis results is necessary during the MDAS analysis service delivery process to ensure the effectiveness of the data analysis. In the MDA processing flow, the MDAS consumer receives the analysis report, verifies it, and provides feedback data to the MDAS producer.

[0003] However, in existing technologies, the MDAS consumer only verifies the model output in the analysis report. The MDASconsumer obtains the model's actual output value and feeds back the corrected model output, i.e., the actual output value. However, in practice, there are scenarios where the model output corresponds to a recommendation, and the MDAS consumer cannot feed back the model's actual output value; or where the model output corresponds to a predicted value, but the actual output value contains multiple data points, and feeding back all of them would waste resources; or where the model's actual output value exists on a third-party device, and the MDAS consumer cannot obtain it. For example, in some scenarios, the model's actual output value includes multiple data points, and feeding back all of them would waste resources. Another example is in energy-saving scenarios, where the MDAS producer provides the optimal energy-saving strategy suggestion to meet a certain objective, but the MDASconsumer cannot obtain the actual optimal energy-saving strategy and therefore cannot feed back the model's actual output value. Yet another example is in handover optimization scenarios, where the MDAS consumer can be the source base station (gNodeB, gNB). In standard handover procedures, the source gNB can only select the target gNB for handover based on radio signal quality, and cannot predict the actual Quality of Experience (QoE) of the User Equipment (UE) after handover to a specific target gNB, i.e., the true output value. In these scenarios, existing solutions may result in the MDAS consumer being unable to provide feedback on the model's true output value or having an inflexible feedback mechanism, leading to resource waste and impacting data analysis effectiveness. Summary of the Invention

[0004] This application provides a method and apparatus for transmitting information, and provides a flexible feedback mechanism that enables a second network device to determine feedback information based on a first type of feedback information sent by a first network device, and further, to send feedback information to the first network device. The first network device can determine the performance of the model based on the feedback information, and further, if the model performance is poor, adjust the data analysis strategy to ensure the effectiveness of the data analysis.

[0005] Firstly, a method for transmitting information is provided. A first network device sends a first type of feedback information to a second network device. The first type includes one or more of the following: model evaluation metrics, operation instruction information, and network performance metrics. The model evaluation metrics are used to indicate the performance of the model, the operation instruction information is used to indicate the operation to be performed based on the analysis results of the model, and the network performance metrics are used to indicate the network metrics obtained based on the analysis results of the model. The first type is used to request feedback information. The first network device receives the feedback information from the second network device. The first network device determines the performance of the model based on the feedback information.

[0006] The model evaluation metric may be obtained based on the model's predicted values ​​and the actual values, such as the difference between the predicted and actual values, or the average of the differences between the predicted and actual values. The operation instruction information may be an operation performed by the second network device based on the model's predicted values ​​or recommendations from the first network device, and may include specific operation information or timestamps. The network performance metric may be certain network performance metrics obtained after the second network device performs operations based on the model's predicted values ​​or recommendations from the first network device.

[0007] According to the communication method provided in this application, the first type of feedback information sent by the first network device to the second network device can be flexibly configured to allow the second network device to send the type of feedback information to the first network device. This enables the first network device to obtain feedback information according to its own needs and the feedback capability of the second network device, and to determine the performance of the model based on the feedback information. In some scenarios, it can also save transmission resources.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, after receiving feedback information, the first network device receives the model's actual values ​​or other data, such as network performance indicators, from the third network device based on the feedback information. Furthermore, the first network device can determine the model's performance based on the model's actual values ​​or other data, as well as the feedback information. This implementation allows the first network device to obtain more data from the third network device based on the feedback information, providing more data support for model evaluation and improving the model's evaluation effectiveness.

[0009] In conjunction with the first aspect, in certain implementations of the first aspect, the first network device receives first indication information from the second network device. This first indication information indicates a second type of feedback information supported by the second network device, i.e., the feedback capability of the second network device. The first network device receives the first indication information and determines a first type of feedback information based on the second type indicated by the first indication information. This implementation allows the first network device to configure the first type of feedback information according to the types of feedback information that the second network device can support, avoiding configuration errors that could lead to resource waste.

[0010] The first instruction information can be sent before the first network device sends the first type of feedback information, or it can be sent after the first network device sends the first type of feedback information, after the second network device discovers that the first type exceeds its own feedback capability.

[0011] The second type can also be the default, without requiring the instruction from the first instruction information.

[0012] Secondly, a method for transmitting information is provided, wherein a second network device receives a first type of feedback information from a first network device, the first type including one or more of the following: model evaluation metrics, operation instruction information, and network performance metrics, wherein the model evaluation metrics are used to indicate the performance of the model, the operation instruction information is used to indicate the operation to be performed based on the analysis results of the model, the network performance metrics are used to indicate the network metrics obtained based on the analysis results of the model, and the first type is used to request feedback information; the second network device sends feedback information to the first network device according to the first type of feedback information.

[0013] The model evaluation metric may be obtained based on the model's predicted values ​​and the actual values, such as the difference between the predicted and actual values, or the average of the differences between the predicted and actual values. The operation instruction information may be an operation performed by the second network device based on the model's predicted values ​​or recommendations from the first network device, and may include specific operation information or timestamps. The network performance metric may be certain network performance metrics obtained after the second network device performs operations based on the model's predicted values ​​or recommendations from the first network device.

[0014] According to the communication method provided in this application, the second network device receives a first type of feedback information from the first network device. The type of feedback information sent by the second network device to the first network device can be flexibly configured, so that the first network device can obtain feedback information according to its own needs. Furthermore, the performance of the model can be determined based on the feedback information. In some scenarios, transmission resources can also be saved.

[0015] In conjunction with the second aspect, in some implementations of the second aspect, the second network device itself is unable to generate feedback information corresponding to the first type. The second network device receives feedback information from the fourth network device and sends the feedback information to the first network device.

[0016] In conjunction with the second aspect, in some implementations of the second aspect, the second network device generates feedback information corresponding to the first type and sends the feedback information to the first network device.

[0017] In conjunction with the second aspect, in some implementations of the second aspect, the second network device receives the true value of the model from the fifth network device, generates feedback information based on the true value of the model and the first type, and sends the feedback information to the first network device.

[0018] In conjunction with the second aspect, in some implementations of the second aspect, the second network device sends first indication information to the first network device. This first indication information indicates a second type of feedback information supported by the second network device, enabling the first network device to determine a first type of feedback information based on this second type. This implementation allows the first network device to configure the first type of feedback information according to the types of feedback information supported by the second network device, avoiding resource waste or the inability of the second network device to provide feedback due to configuration errors.

[0019] The first instruction information can be sent before the first network device sends the first type of feedback information, or it can be sent after the first network device sends the first type of feedback information and the second network device discovers that the first type exceeds its own feedback capability.

[0020] The second type can also be the default, without requiring the instruction from the first instruction information.

[0021] Thirdly, a communication apparatus is provided, comprising units for performing the steps of the communication method described in the first aspect and its implementations.

[0022] In one design, the communication device is a communication chip, which may include input circuitry or interface for transmitting information or data, and output circuitry or interface for receiving information or data.

[0023] In another design, the communication device is a communication equipment (e.g., a first network device, etc.), and the communication chip may include a transmitter for sending information and a receiver for receiving information or data.

[0024] Fourthly, a communication apparatus is provided, comprising units for performing the steps of the communication method described in the second aspect and its implementations.

[0025] In one design, the communication device is a communication chip, which may include input circuitry or interface for transmitting information or data, and output circuitry or interface for receiving information or data.

[0026] In another design, the communication device is a communication equipment (e.g., a second network device), and the communication chip may include a transmitter for sending information and a receiver for receiving information or data.

[0027] Fifthly, a communication device is provided, comprising a processor and a memory for storing a computer program, the processor for calling and running the computer program from the memory, causing the communication device to perform the communication methods described in the first or second aspect and their respective implementations.

[0028] Optionally, the processor may be one or more, and the memory may be one or more.

[0029] Optionally, the memory may be integrated with the processor, or the memory may be separated from the processor.

[0030] Optionally, the communication device may also include a transmitter and a receiver.

[0031] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program (also referred to as code or instructions), which, when the computer program is run, causes a computer to execute the communication methods in the first or second aspect and their respective implementations described above.

[0032] In a seventh aspect, a computer-readable medium is provided that stores a computer program (also referred to as code or instructions) that, when executed on a computer, causes the computer to perform the communication methods described in the first or second aspect and their respective implementations.

[0033] Eighthly, a communication system is provided, comprising: at least one device according to any one of the third aspects and one device according to any one of the fourth aspects.

[0034] Ninthly, a chip system is provided, including a memory and a processor, the memory for storing a computer program and the processor for calling and running the computer program from the memory, causing a communication device equipped with the chip system to perform the communication methods described in the first or second aspect and their respective implementations.

[0035] The chip system may include input circuits or interfaces for transmitting information or data, and output circuits or interfaces for receiving information or data. Attached Figure Description

[0036] Figure 1 This is a flowchart of the verification and feedback mechanism for managing data analytics services.

[0037] Figure 2 This is an application scenario of the embodiments of this application.

[0038] Figure 3 This is a schematic diagram of an example of the data analysis effect feedback method provided in this application.

[0039] Figure 4 This is another schematic diagram of the data analysis effect feedback method provided in this application.

[0040] Figure 5 This is a schematic diagram of another example of the data analysis effect feedback method provided in this application.

[0041] Figure 6 This is a schematic diagram illustrating yet another example of the data analysis effect feedback method provided in this application.

[0042] Figure 7 This is a schematic diagram of an example of a communication device according to an embodiment of this application.

[0043] Figure 8 This is a schematic diagram of yet another example of a communication device according to an embodiment of this application. Detailed Implementation

[0044] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0045] The technical solutions of this application embodiment can be applied to various communication systems, such as: Global System of Mobile communication (GSM) system, Code Division Multiple Access (CDMA) system, Wideband Code Division Multiple Access (WCDMA) system, General Packet Radio Service (GPRS), Long Term Evolution (LTE) system, LTE Frequency Division Duplex (FDD) system, LTE Time Division Duplex (TDD) system, Universal Mobile Telecommunication System (UMTS), Worldwide Interoperability for Microwave Access (WiMAX) system, 5th Generation (5G) system, or New Radio (NR), etc.

[0046] The network device in this application embodiment can be a base station (BTS) in a Global System of Mobile communication (GSM) system or Code Division Multiple Access (CDMA), a base station (NodeB, NB) in a Wideband Code Division Multiple Access (WCDMA) system, an evolved NodeB (eNB or eNodeB) in an LTE system, a radio controller in a Cloud Radio Access Network (CRAN) scenario, or a relay station, access point, vehicle-mounted equipment, wearable device, or network equipment in a future 5G network or a future evolved PLMN network, etc. The embodiments of this application are not limited to these.

[0047] The network element or module in the embodiments of this application may include:

[0048] MDAS: Management Data Analysis Service, which provides specific types of management data analysis reports.

[0049] MDA: Managed Data Analysis, a logical entity that provides managed data analysis services, with functions for model training and managed data analysis.

[0050] Model training module: An internal functional module of MDA that can use training data to train models and generate usable models.

[0051] Management Data Analysis Module: An internal functional module of MDA that uses AI / ML technology to analyze management data (e.g., performance measurement data, minimized drive tests, alarm data, configuration data, network analysis data, etc.) and generate analysis reports.

[0052] MDAS producer: A provider of data analytics services that can offer MDAS to external parties.

[0053] MDAS consumer: Consumers who manage data analytics services can invoke MDAS.

[0054] Figure 1 This is a flowchart illustrating the verification and feedback mechanisms for managing data analytics services in existing technologies. For example... Figure 1 As shown, the steps of the MDAS verification and feedback mechanism are as follows:

[0055] Step 1: Analyze the configuration.

[0056] The MDAS producer sends a first configuration message to the MDAS consumer. This first configuration message contains analytics IDs, which indicate the types of analytics the MDAS producer can provide. If multiple types of analytics are available, a list of analytics IDs is included. The analytics ID is a unique identifier for each type of analytics; possible values ​​include: handover optimization, coverage issue analysis, and MDA-assisted energy saving.

[0057] Step 2: Analysis of report requests and responses.

[0058] The MDAS consumer sends an analytics report request message to the MDAS producer to request a specific analytics report. The message content includes an analytics ID, and optionally, the analytics area, report method, and reporting period. The MDAS producer returns a response message to the MDAS consumer, containing the analytics ID, indicating that the request has been received.

[0059] Step 3: Notification of the analysis report.

[0060] After receiving an analysis report request, the MDAS producer performs data analysis, prepares the analysis report, and sends an analysis report notification to the MDASconsumer. The message includes the analytics ID, the timestamp of analytics, the valid period of the analysis, the confidence level of the analysis results, and the specific analytics results, which are the outputs of the model, such as a predicted value or a recommendation.

[0061] Steps 4 and 5: Analysis report verification and feedback.

[0062] After receiving the analysis report from the MDAS producer, the MDAS consumer verifies the analysis results (analyticsresults). If the verification finds a discrepancy between the actual model output and the model output included in the analysis report, it sends a feedback notification to the MDAS producer, including the verified analysis identifier (analytics ID) and the corrected data (rectified data). For example, in an alarm analysis scenario (analytics ID = alarm incident analysis), the analysis report contains alarm source information. If the MDAS consumer (e.g., fault supervision management service (fault supervision MnS)) finds that the actual alarm source differs from the alarm source provided in the analysis report, it provides feedback to the MDAS producer, and the corrected data in the feedback is the actual alarm source information.

[0063] Step 6: Model retraining.

[0064] After receiving feedback from the MDAS consumer, the MDAS producer can use the feedback data to retrain the model, improve model performance, and thus improve the accuracy of data analysis results.

[0065] Figure 2 This is a system architecture diagram of an embodiment of this application. As shown in the figure, the MDAS producer is a device that provides management data analysis services, the MDA is a logical entity that carries management data analysis functions, and the MDAS consumer can be any device that can call MDAS. In actual deployment, there are various possibilities for the devices that assume the roles of MDAS producer and MDAS consumer. In this embodiment of the application, the deployment architecture can be one or more of the following six scenarios:

[0066] 1. MDAS producer stands for Element Management System (EMS), and MDASconsumer stands for Network Management System (NMS).

[0067] 2. The MDAS producer is NMS / EMS, and the MDAS consumer is gNB;

[0068] 3. The MDAS producer is NMS / EMS, and the MDAS consumer is the Network Data Analytics Function (NWDAF).

[0069] 4. The MDAS producer is NMS / EMS, and the MDAS consumer is a human operator;

[0070] 5. The MDAS producer is an NMS / EMS, and the MDAS consumer is a third-party network service optimization tool;

[0071] 6. Both the MDAS producer and MDAS consumer are NMS, or both the MDAS producer and MDASconsumer are EMS.

[0072] like Figure 2 As shown in the diagram, the boxes represent logical functions or logical entities, and the arcs and circles are a common representation of service-oriented architecture, connecting service providers and consumers. The circle side represents the service provider, and the arc side represents the service consumer. Figure 2 As shown in (a), MDA is a logical entity that carries the management data analysis function, consisting of two parts: MDAF and MDAS Producer. MDAS Producer is the service interface provider, and MDAF is the internal processing function of MDA, including data analysis and model training functions. Figure 2 As shown in (b), the MDAS Producer is the service provider, and MDA is equivalent to... Figure 2 MDAF in (a) is the internal processing function of MDAS Producer, which includes data analysis and model training functions.

[0073] Figure 3 This is an example of the data analysis effect feedback method provided in this application, as shown in the figure. The method 300 includes:

[0074] S301, the first type of feedback information sent by the first network device to the second network device.

[0075] Specifically, the first type includes one or more of the following types: model evaluation metrics, operational instruction information, and network performance metrics.

[0076] Model evaluation metrics are used to assess model performance. Specifically, these metrics are indicators that evaluate the quality of a model's performance. Commonly used metrics include precision, accuracy, error rate, recall, mean squared error, root mean square error, root mean square logarithmic error, and mean absolute error. The first network device can determine the model's performance based on the feedback model evaluation metrics and adjust poorly performing models. Model evaluation metrics can be obtained from the predicted values ​​of the model provided by the first network device and the actual values ​​of the model obtained by the second network device.

[0077] As an example rather than a limitation, in some scenarios, the first network device requests the second network device to provide model evaluation metrics. The second network device can derive the true value of the model itself. Furthermore, the second network device can derive the first metric value based on the model's predicted value and the model's true value, and then provide the feedback as the model evaluation metric.

[0078] As an example rather than a limitation, in some scenarios, the first network device requests the second network device to provide model evaluation metrics. The second network device itself cannot obtain the true value of the model. The second network device can request the true value of the model from the fifth network device. Furthermore, the second network device derives the first metric value based on the requested true value of the model and the predicted value given by the first network device, and provides it as the value of the model evaluation metric.

[0079] It should be noted that this first metric value can be calculated based on the model's predicted values ​​and at least one true value. It can be the model's precision, accuracy, error rate, recall, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, etc. In certain scenarios, such as when multiple true values ​​exist, using feedback model evaluation metrics can save some feedback resources.

[0080] The operation instruction information describes the specific actions taken by the second network device after receiving the analysis report, based on the recommendations of the first network device or the predictions of the first network device, including the specific actions taken and the timestamps of the actions taken.

[0081] In some embodiments, the first network device can collect relevant data from the third network device, or the actual values ​​of the model after the second network device executes the recommended actions, based on the operation instructions fed back by the second network device. This data is used to determine the model's performance and then adjust models with poor performance. It should be noted that this relevant data can be network performance metrics corresponding to the execution of the recommended actions.

[0082] As an example, and not a limitation, the first network device can collect the actual value of the model after executing the recommended actions from the third network device based on the operation instructions fed back by the second network device. In the handover optimization scenario, the second network device can be the source base station. The second network device can request a handover optimization analysis report from the first network device and make a handover decision with the assistance of MDAS. The handover optimization analysis report includes the recommended actions and may also include the predicted value after executing the actions, i.e., the predicted QoE, in other words, the QoE that the UE is expected to achieve after handover to a certain target gNB. The source gNB can refer to the prediction results in the analysis report to select the most suitable target gNB for handover. At this time, the first network device requests the second network device to feed back operation instructions. The second network device may return the handover actions taken and the timestamp of the actions taken. Optionally, the first network device can obtain the actual QoE from the third network device (which can be the target base station) based on the operation instructions fed back by the second network device, and then evaluate the model performance based on the feedback information and the actual values ​​of the model.

[0083] As an example, and not a limitation, in an energy-saving optimization scenario, the analysis report provided by the first network device to the second network device includes energy-saving suggestions. Specifically, these suggestions may include: recommended NR cells to enter energy-saving mode, recommended candidate cells with precedence for taking over the traffic of the ES-Cell, and the time period during which ES is or is not allowed. In this scenario, the analysis report provides recommendations. After the second network device executes the action corresponding to the recommendations, it sends operation instructions back to the first network device. Upon receiving these instructions, the first network device can obtain network performance metrics after executing the action from the third network device and then evaluate the model based on the feedback information and network performance metrics.

[0084] The network performance metrics are indicators defined by the first network device that describe network performance, such as standard-defined Key Performance Indicators (KPIs) or custom optimization objective functions. The second network device can execute actions based on the predicted values ​​or recommendations provided by the model from the first network device. After executing these actions, it obtains network performance metrics and sends feedback information to the first network device. This feedback information can be data corresponding to the network performance metrics. The first network device determines whether the expected network performance has been achieved based on the feedback network performance metrics, thus determining the model's performance and adjusting poorly performing models.

[0085] In some possible implementations, the network performance metrics can be those obtained after executing the recommended action. It should be understood that this recommended action can correspond to operational instruction information. For example, a first network device can request operational instruction information from a second network device. The operational instruction information returned by the second network device contains the action to be performed according to the recommended action. The first network device can then obtain the network performance metrics obtained after executing the recommended action from a third network device based on the operational instruction information, and use this information to evaluate the model. Alternatively, the first network device can request the second network device to directly return the network performance metrics. After executing the recommended action, the second network device can obtain the network performance metrics from a fourth network device and return them as feedback to the first network device. Specific examples can be found in the relevant descriptions within the operational instruction information.

[0086] It should be noted that the first type of feedback information can be configured through the `feedbackInfo` signaling. The selectable values ​​for `feedbackInfo` differ depending on the scenario. Specifically, the value of `feedbackInfo` indicates the first type of feedback information sent by the second network device to the first network device. Furthermore, this value can determine the specific content of the feedback information required for that type. As an example, not a limitation, in a handover optimization scenario, `feedbackInfo = actionTakenindication`, indicating that the second network device needs to provide feedback on the final determined target gNB identifier and the handover time point. In an energy-saving scenario, `feedbackInfo = actionTaken indication`, indicating that the second network device needs to provide feedback on the final energy-saving action taken, i.e., the cell identifier for enabling / disabling energy-saving features and the time point. In another energy-saving scenario, `feedbackInfo = Energy Efficiency (EE)`, indicating that the second network device needs to provide feedback on the energy efficiency KPI value after taking the specified energy-saving action. One possible definition of the EE KPI is: EE = Total Network Throughput / Total Network Energy Consumption.

[0087] It should be noted that the first type of feedback information can be sent as a separate message before the analysis report notification, or it can be included in the analysis report notification.

[0088] In some embodiments, it can be assumed that the first network device is aware of the second type of feedback information supported by the second network device, that is, the first network device is aware of the type of feedback information that the second network device can feed back. Further, the first network device determines the first type of feedback information based on the second type, that is, the first network device instructs the second network device to feed back which type of information based on the feedback capability of the second network device.

[0089] In some embodiments, the first network device is unaware of the second type of feedback information supported by the second network device. The second network device can send a first indication message to the first network device, indicating the second type of feedback information supported by the second network device, i.e., the second network device reports its feedback capability. Further, the first network device determines the first type of feedback information based on the first indication message. Optionally, the first indication message can be sent by the second network device to the first network device before the first network device sends the first type of feedback information, indicating the second network device's reporting capability in advance. Optionally, the first indication message can be sent by the first network device to the second network device after the first network device sends the first type of feedback information. Specifically, upon receiving the first type of feedback information and discovering that it cannot report the feedback information corresponding to the first type, the second network device further sends the first indication message to the first network device to report its feedback capability.

[0090] It should be noted that the first indication information indicates the type of feedback information that the second network device can feed back for a specific analysis type. Optionally, the first indication information includes an analysis identifier, and further includes the type of feedback information that can be fed back corresponding to the analysis identifier.

[0091] It should be noted that before step S301, the first network device sends first configuration information to the second network device. See the relevant description in method 400 for details.

[0092] It should be noted that after the first network device sends the first configuration information to the second network device, the second network device sends an analysis report request message to the first network device. This analysis report request message requests an analysis report of a specific analysis type. Optionally, after receiving the analysis report request message from the second network device, the first network device further sends a response message to the second network device, indicating that the first network device has received the analysis report request message. As an example and not a limitation, this response message may include an analysis identifier, which corresponds to the analysis identifier in the aforementioned analysis report request message. It should also be noted that after receiving the analysis report request message, the first network device further performs data analysis based on the analysis report request message, prepares an analysis report, and sends an analysis report notification to the second terminal device. This analysis report notification may include, but is not limited to: an analysis identifier (analytics ID), the timestamp of analytics, the valid period of the analysis, the confidence level of the analysis results, and the specific analysis results. Optionally, the analysis results include the model's output, such as a predicted value or a recommendation.

[0093] It should be noted that the first type of feedback information can be sent either within the analysis report or as a separate message before sending the analysis report. Sending the first type of feedback information separately before the analysis report notification, or even sending it separately from the second network device to the second network device before the second network device sends the analysis report request message to the first network device, without including it in the analysis report notification, means that for the same analysis report type, only the first type of feedback information needs to be sent once, eliminating the need to repeatedly include the same first type of feedback information in each analysis report notification, thus saving some transmission overhead.

[0094] It should be noted that the first type of feedback information may be determined by the first network device in combination with at least one of the following factors: analysis type, which is the type requested in the aforementioned analysis report request message.

[0095] The second type of feedback information that the second network device is able to provide, as indicated in the first instruction information sent by the default or second network device.

[0096] As an example and not a limitation, in energy-saving scenarios, the value of feedbackInfo can be actionTakenindication or EE KPI. However, if the first indication information sent by the second network device indicates that the type of feedback information that the second network device can provide only includes actionTaken indication, then in this scenario, feedbackInfo can only be configured as actionTaken indication.

[0097] S302, the first type of feedback information received by the second network device.

[0098] S303, the second network device sends feedback information to the first network device according to the first type of feedback information.

[0099] Optionally, the feedback information can be determined by the second network device itself based on the first type of feedback information. For example, the second network device can output the model's true value, and the first type indicates that the average difference between the model's predicted output value and the model's true value should be used as the model evaluation index for feedback. In this case, the second network device can obtain the average value based on the first type for feedback.

[0100] Alternatively, the feedback information can be obtained by the second network device from the fourth network device. For example, if the first type of feedback information requested by the first network device is a network performance indicator, the second network device can send the first type of feedback information to the fourth network device and obtain the network performance indicator from the fourth network device. It can also be determined by the second network device after obtaining the model's actual value from the fifth network device, based on the model's actual value and the model's predicted value in the analysis report. Specifically, the second network device sends the first type of feedback information to the fifth network device and further receives the model's actual value from the fifth network device. Specific examples can be found in the relevant description in step S301.

[0101] S304, the first network device receives feedback information sent by the second network device.

[0102] S305, the first network device determines the model's performance based on the feedback information.

[0103] Optionally, after receiving the feedback information sent by the second network device, the first network device sends a first request message to the third network device based on the content of the feedback information. This first request message requests the third network device to obtain the model's true value or other data. It should be noted that this other data can be the network performance metrics corresponding to the operation instruction information described in the relevant example in step S301. It should also be noted that the model whose performance the first network device needs to determine is the model trained by the first network device. Upon receiving the first request message, the third network device further sends the model's true value or other data to the first network device based on the first request message. This third network device can be a third-party device capable of providing data; this is not a limitation but an example. The third network device can be an application function (AF), a network data analytics function (NWDAF), a base station, etc. For example, in the handover optimization scenario, in step S301, feedbackInfo = actionTaken indication. The feedback information from the second network device is the finally determined target base station identifier and the handover time. The first network device sends a first request to the third network device as needed, further requesting the terminal device to switch to the actual target base station and obtain the Quality of Experience (QoE). In this scenario, the third network device is a third-party device that can provide QoE information. The third network device can be the target base station or AF, etc.

[0104] In some embodiments, the first network device determines model performance based on feedback information, or based on feedback information and the model's actual values, or based on feedback information and network performance indicators. Furthermore, it adjusts the data analysis strategy based on the model's performance. It should be noted that the model's actual values ​​or network performance indicators are obtained by the first network device from a third-party device after receiving the feedback information.

[0105] As an example and not a limitation, the first network device may adjust its data analysis strategy in at least one of the following ways:

[0106] Use feedback information to retrain the model.

[0107] If other models are available, use those models for data analysis, for example, models with higher accuracy.

[0108] Collect new training data, train a new model from scratch, and optionally, reprocess the data or perform feature selection.

[0109] Optionally, if retraining the model or retraining a new model takes a long time, it may cause the model to be temporarily unusable. In one possible implementation, the first network device sends second configuration information to the second network device, which includes an analysis identifier. For example, the second configuration information indicates that the analysis type corresponding to the analysis identifier is temporarily unavailable.

[0110] This embodiment provides a flexible feedback method for data analysis results. The second network device can provide the first network device with the type of feedback information required by the first network device based on the first network device's configuration, improving feedback flexibility and saving transmission resources in certain scenarios. Correspondingly, the first network device can judge the model's performance based on the feedback information. Furthermore, if the model performance is poor, the data analysis strategy can be adjusted to ensure the effectiveness of the data analysis.

[0111] Figure 4 This is a schematic diagram of yet another example of the data analysis effect feedback method provided in the embodiments of this application. For example... Figure 4 As shown, the method 400 includes:

[0112] S401, the first network device sends the first configuration information to the second network device.

[0113] Specifically, the first configuration information includes an analytics ID, which indicates the types of analytics that the first network device can provide. Optionally, if the first network device can provide multiple types of analytics, a list of analytics IDs may be included. The analytics type value under each analytics ID uniquely identifies a type of analytics. As an example, and not a limitation, this value could be: handover optimization, coverage issue analysis, MDA-assisted energy saving, etc.

[0114] S402, the second network device sends a first indication message to the first network device, the first indication message being used to indicate a second type of feedback information supported by the second network device.

[0115] It should be understood that step S402 is optional, and the second type can be the default for the first network device.

[0116] Specifically, the first indication information may indicate a second type of feedback information that supports a specific analysis type. This first indication information may include an analytics ID and a second type of supported feedback information corresponding to that analytics ID. The second type of supported feedback information can be represented by `availableFeedbackInfo`. The types of feedback information supported by the second network device include, but are not limited to:

[0117] Model evaluation metrics: Metrics that can be used to evaluate model performance; see Method 300 for a detailed description.

[0118] Operation instruction information: Instructions for the actions to be taken by the second network device after receiving the analysis report. For details, please refer to method 300.

[0119] Network performance metrics: Metrics that can be used to describe network performance; see Method 300 for details.

[0120] S403, the first type of feedback information sent by the first network device to the second network device.

[0121] Specifically, the message content may include an analytics ID and the first type of feedback information corresponding to the analytics type of that ID: feedbackInfo. Optional values ​​for feedbackInfo include, but are not limited to: model evaluation metrics, operational instructions, network performance metrics, etc.

[0122] It should be noted that the first type of feedback information sent by the first network device to the second network device can be sent as a separate message in step S403, or it can be included in the analysis report notification message in step S405. In some embodiments, the first type of feedback information sent by the first network device to the second network device is sent in advance as a separate message, thus decoupling this message from the analysis report notification. For the same analysis type, only the first type of feedback information needs to be configured once, without having to carry this information repeatedly in each analysis report notification, which can save some transmission resource overhead.

[0123] It should be noted that if step S402 does not precede step S403, and the first network device does not have a default second type, the first network device may have misconfigured the first type. The first type of the feedback information sent by the first network device to the second network device may be a type of feedback information that the second network device does not support. Optionally, after the first network device configures an unsupported first type, the second network device sends a feedback modification notification to the first network device, indicating the second type of feedback information that the second network device supports. The first network device re-determines the first type of feedback information based on the feedback modification notification and resends the message to the second network device. It should be noted that the first type of feedback information message sent initially and the resent first type of feedback information message may be the same message. It should be noted that the feedback modification notification may include the unsupported feedback information type (unavaliableInfo) and the reason for not supporting it (reason), and may also include the feedback information type that it can support (avaliableInfo). The values ​​of unavaliableInfo and availableInfo include, but are not limited to, model evaluation metrics, operation instruction information, network performance metrics, etc. It should be noted that the feedback modification notification and the aforementioned first instruction information can be a single message.

[0124] S404, the second network device sends an analysis report request to the first network device.

[0125] Specifically, the second network device can request an analysis report of a certain type of analysis. The content of the message includes the analytics ID, and optionally, the message may also include one or more of the following: analysis area, reporting method, reporting period, etc.

[0126] Optionally, the first network device receives the analytics report request and sends a response message to the second network device. The response message may include an analytics ID to indicate that the first network device has received the request.

[0127] S405, the first network device sends an analysis report notification to the second network device.

[0128] It should be noted that the first type of feedback message in step S403 can be included in the analysis report notification, rather than being sent as a separate message.

[0129] Specifically, after receiving the analysis report request in step S404, the first network device performs data analysis according to the request of the second network device, prepares the analysis report, and further sends an analysis report notification to the second network device. The analysis report notification includes the analysis identifier (analytics ID), the timestamp of analytics, the valid period of the analysis, the confidence level of the analysis results, and the specific analysis results. When the first type of feedback information is included in the analysis report notification, the message also includes the first type of feedback information (feedbackInfo) and its corresponding value.

[0130] S406, the second network device determines the feedback information based on the first type of feedback information.

[0131] Optionally, the second network device itself can derive the true value of the model, process the data, and determine the feedback information. For specific examples, please refer to the relevant description in method 300.

[0132] Optionally, the second network device may send a first type of feedback information to the fourth network device and receive feedback information from the fourth network device. For specific examples, please refer to the relevant description in method 300.

[0133] Optionally, the second network device may send a request to the fifth network device to obtain the true value of the model from the fifth network device, and determine the feedback information based on the true value and the first type of feedback information. See the relevant description in method 300 for a specific example.

[0134] S407, the second network device sends feedback information to the first network device.

[0135] S408, the first network device sends a request message to the third network device.

[0136] It should be noted that this step is optional.

[0137] Specifically, after receiving feedback information, the first network device needs more data to evaluate model performance. It can send a request to the third network device to obtain more additional data or the model's actual values. As an example, and not a limitation, in a handover scenario, if the feedback information returned by the second network device corresponds to the first type of operation instruction information, then the feedback information from the second network device may include the executed action and its timestamp. After receiving the feedback information, the first network device can obtain the UE's actual QoE from the third network device (target base station) to evaluate model performance. As an example, and not a limitation, in an energy-saving scenario, if the feedback information returned by the second network device corresponds to the first type of operation instruction information, then the feedback information from the second network device may include the energy-saving actions taken. After receiving the feedback information, the first network device can obtain other data from the third network device, such as network performance indicators after taking energy-saving actions, to evaluate the model.

[0138] Accordingly, the first network device receives the actual value or other performance data.

[0139] S409, the performance evaluation model for the first network device.

[0140] Specifically, the first network device evaluates the model's performance based on feedback information, or the first network device evaluates the model's performance based on other data or the model's true value and feedback information received in step S408.

[0141] Optionally, the first network device can adjust the poorly performing model, and the adjustment methods include, but are not limited to, the following:

[0142] Retrain the model;

[0143] Call other models that can be used for data analysis; for example, you can choose a model with higher accuracy.

[0144] Collect new data and train a new model from scratch. Optionally, you can re-preprocess the data, perform feature selection, etc.

[0145] S410, the first network device sends the second configuration information to the second network device.

[0146] Specifically, when model retraining or retraining takes too long, rendering the model temporarily unusable, the first network device can send second configuration information to the second network device, indicating that a specific type of analytics is temporarily unavailable. This information may include the analytics ID. It should be noted that this step is optional.

[0147] This embodiment provides a flexible feedback method for data analysis results. The second network device can provide feedback information of the type required by the first network device based on the configuration of the first network device, which improves the flexibility of feedback. In some scenarios, it can also save transmission resources. The first network device can judge the performance of the model based on the feedback information, and further, if the model performance is poor, it can adjust the data analysis strategy to ensure the data analysis effect.

[0148] like Figure 5 This is another example of the data analysis effect feedback method provided in this application. As shown in the figure, the method 500 includes:

[0149] S501, the first network device sends a first request message to the second network device, the first request message instructing the second network device to return the true value of the model to the first network device.

[0150] It should be noted that this step is optional, and the second network device can also send the model's actual values ​​to the first network device by default.

[0151] It should be noted that this first request information can be included in the analysis report notification or sent as a separate message. The steps preceding this step and the specific configuration method for this configuration information are similar to those in method 300, and can be found in the relevant descriptions within method 300.

[0152] S502, the second network device receives the first request information.

[0153] In this embodiment of the method, the second network device itself cannot output the true value of the model, so it requests the true value of the model from the third network device. As an example and not a limitation, in the handover optimization scenario, the second network device is the source base station. The source base station itself cannot obtain the QoE information after the UE handover. Therefore, the second network device requests the third network device, i.e. the target base station, to obtain the QoE after the UE handover and sends it to the first network device as feedback information.

[0154] It should be noted that in this embodiment of the method, an interface has been added so that the second network device can obtain the true value of the model from the third network device.

[0155] S503, the second network device obtains the true value of the model from other network devices.

[0156] S504, the second network device sends feedback information to the first network device.

[0157] The second network device sends the model's true value as feedback information to the first network device.

[0158] S505: The first network device receives the actual value of the model and evaluates the model's performance based on the actual value.

[0159] In this embodiment of the method, by adding a new network interface, when the second network device cannot obtain the model's true value on its own, it can obtain the model's true value from other network devices, and then feed it back to the first network device as feedback information. This allows the first network device to determine the model's performance based on the model's true value, and further train the poorly performing model to ensure the effectiveness of data analysis.

[0160] like Figure 6 This is another example of the data analysis effect feedback method provided in this application. As shown in the figure, the method 600 includes:

[0161] S601, the first network device sends the first configuration information to the second network device.

[0162] This step is similar to step S401 in method 400, and you can refer to the relevant description in step S401 for details.

[0163] S602, the second network device sends a first indication message to the first network device, the first indication message being used to indicate that the second network device supports the true value of the feedback model.

[0164] It should be noted that this step is optional; the second network device can either return the actual value of the model or use the default value of the first network device.

[0165] S603, the first network device sends a first request message to the second network device.

[0166] Specifically, the first request information requests the second network device to return the actual value of the model. It should be noted that this step is optional, and the second network device may default to returning the actual value of the model to the first network device.

[0167] It should be noted that the first request information sent by the first network device to the second network device can be sent as a separate message in step S602, or it can be included in the analysis report notification message in step S604. In some embodiments, the first request information sent by the first network device to the second network device is sent in advance as a separate message, thus decoupling this information from the analysis report notification. For the same analysis type, it only needs to be sent once, without having to carry this information repeatedly in each analysis report notification, which can save some transmission resource overhead.

[0168] For details regarding the specific network signaling and content of this message, please refer to the relevant description in step S403.

[0169] S604, the second network device sends an analysis report request to the first network device.

[0170] This step can be found in the relevant description in step S404.

[0171] S605, the first network device sends an analysis report notification to the second network device.

[0172] It should be noted that the first request information in step S602 can be included in the analysis report notification, rather than being sent as a separate message.

[0173] For a detailed description of this step, please refer to the relevant description of step S405.

[0174] S606, the second network device obtains the true value of the model from other network devices.

[0175] Specifically, based on the first request information, the second network device obtains the actual value of the model from other network devices through the newly added network interface.

[0176] S607, the second network device sends feedback information to the first network device.

[0177] S608, the first network device obtains other performance data based on the model's actual values.

[0178] It should be noted that the first network device may not be able to evaluate the model performance based solely on the model's actual values. The first network device may need to obtain other performance data and then evaluate the model based on both the actual values ​​and the performance data.

[0179] It should be noted that this step is optional.

[0180] S609, the performance evaluation model for the first network device.

[0181] This step is similar to step S409 in method 400, and the relevant description of S409 can be found in the description.

[0182] S610, the first network device sends second configuration information to the second network device. This step is similar to step S410 in method 400, and can be found in the relevant description of S410.

[0183] Figure 7 This is a schematic diagram of a communication device 700 according to an embodiment of this application. Each unit in the communication device 700 can be implemented by software.

[0184] In some embodiments, the communication device 700 may be the first network device in the method embodiment 300 above, or it may be a chip for implementing the functions of the first network device in the method embodiment above. It should be understood that the communication device 700 may correspond to the steps of the first network device in the method 300 of this application embodiment. The communication device 700 includes:

[0185] The transceiver unit 710 is used to send feedback information of the first type to the second network device.

[0186] The processing unit 720 is used to determine the performance of the model based on the feedback information.

[0187] Specifically, the first type includes one or more of the following types: model evaluation metrics, operational instruction information, and network performance metrics.

[0188] The model evaluation metrics are used to assess the model's performance. Specifically, model evaluation metrics are indicators that can be used to evaluate the quality of a model's performance. Commonly used evaluation metrics include: precision, accuracy, error rate, recall, mean squared error, root mean square error, root mean square logarithmic error, and mean absolute error. The processing unit 720 can determine the model's performance based on the feedback model evaluation metrics and adjust poorly performing models. The model evaluation metrics can be obtained from the model's predicted values ​​provided by the processing unit 720 and the model's true values ​​obtained from the second network device.

[0189] As an example rather than a limitation, in some scenarios, the processing unit 720 requires the second network device to provide feedback on the model evaluation index. The second network device itself can obtain the true value of the model. Furthermore, the second network device can derive the first index value based on the model prediction value and the model's true value and provide feedback as the value of the model evaluation index.

[0190] As an example rather than a limitation, in some scenarios, the processing unit 720 requests the second network device to provide model evaluation metrics. If the second network device itself cannot obtain the true value of the model, it can request the fifth network device to obtain the true value of the model. Furthermore, the second network device uses the true value of the model obtained from the request and the predicted value given by the processing unit 720 to derive the first metric value as the value of the model evaluation metric and provides feedback.

[0191] It should be noted that this first metric value can be calculated based on the model's predicted values ​​and at least one true value. It can be the model's precision, accuracy, error rate, recall, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, etc. In certain scenarios, such as when multiple true values ​​exist, using feedback model evaluation metrics can save some feedback resources.

[0192] The operation instruction information describes the specific actions taken by the second network device after receiving the analysis report, based on the recommendations of the processing unit 720 or the predictions of the processing unit 720, including the specific actions taken and the timestamps of the actions taken.

[0193] In some embodiments, the processing unit 720 can collect relevant data from the third network device or the actual value of the model after the second network device executes the recommended action, based on the operation instruction information fed back by the second network device, to determine the model's performance and then adjust models with poor performance. It should be noted that the relevant data can be the network performance indicators corresponding to the execution of the recommended action.

[0194] As an example and not a limitation, processing unit 720 can collect the actual value of the model after executing the recommended actions from the third network device based on the operation instruction information fed back by the second network device. In the handover optimization scenario, the second network device can be the source base station. The second network device can request a handover optimization analysis report from processing unit 720 and make a handover decision with the assistance of MDAS. The handover optimization analysis report includes the recommended actions and may also include the predicted value after executing the actions, i.e., the predicted QoE. In other words, it is the QoE that the UE is expected to achieve after handover to a certain target gNB. The source gNB can refer to the prediction results in the analysis report to select the most suitable target gNB to perform the handover. At this time, processing unit 720 requests the second network device to feed back operation instruction information. The second network device may return the handover action taken and the timestamp of the action taken. Optionally, processing unit 720 can obtain the actual QoE from the third network device (which can be the target base station) based on the operation instruction information fed back by the second network device, and then evaluate the model performance based on the feedback information and the actual value of the model.

[0195] As an example, and not a limitation, in an energy-saving optimization scenario, the analysis report provided by the processing unit 720 to the second network device includes energy-saving suggestions. Specifically, the suggestions may include: recommended NR cells to enter energy-saving mode, recommended candidate cells with precedence for taking over the traffic of the ES-Cell, and a time period during which ES is or is not allowed. In this scenario, the analysis report provides recommended suggestions. After the second network device executes the action corresponding to the recommended suggestions, it feeds back operation instruction information to the transceiver unit 710. The transceiver unit 710 receives the operation instruction information and can obtain the network performance indicators after executing the action from the third network device. Then, it evaluates the model based on the feedback information and the network performance indicators.

[0196] The network performance metrics are indicators defined by the processing unit 720 that describe network performance, such as standard-defined Key Performance Indicators (KPIs) or custom optimization objective functions. The second network device can execute actions based on the model's predictions or recommendations provided by the processing unit 720. After executing these actions, it obtains network performance metrics and sends feedback information to the transceiver unit 710. This feedback information can be data corresponding to the network performance metrics. The processing unit 720 determines whether the expected network performance has been achieved based on the feedback network performance metrics, assesses the model's performance, and adjusts models with poor performance.

[0197] In some possible implementations, the network performance metrics can be those obtained after executing the recommended action. It should be understood that the recommended action can correspond to the operation instruction information. For example, processing unit 720 can request the second network device to provide operation instruction information. The operation instruction information provided by the second network device would contain the action to be performed according to the recommended action. Then, processing unit 720 can obtain the network performance metrics obtained after executing the recommended action from the third network device based on the operation instruction information and perform model evaluation. Alternatively, processing unit 720 can request the second network device to directly provide the network performance metrics. After the second network device executes the recommended action, it can obtain the network performance metrics from the fourth network device and provide them as feedback information to transceiver unit 710. Specific examples can be found in the relevant descriptions within the operation instruction information.

[0198] It should be noted that the first type of feedback information can be configured through the feedbackInfo signaling. The selectable values ​​for feedbackInfo differ depending on the scenario. Specifically, the value of feedbackInfo indicates the first type of feedback information sent by the second network device to the transceiver unit 710. Furthermore, this value can determine the specific content of the feedback information required for that type. As an example, not a limitation, in a handover optimization scenario, feedbackInfo = actionTakenindication, indicating that the second network device needs to provide feedback on the final determined target gNB identifier and the handover time point. In an energy-saving scenario, feedbackInfo = actionTaken indication, indicating that the second network device needs to provide feedback on the final energy-saving action taken, i.e., the cell identifier for enabling / disabling energy-saving features and the time point. In another energy-saving scenario, feedbackInfo = Energy Efficiency (EE), indicating that the second network device needs to provide feedback on the energy efficiency KPI value after taking the specified energy-saving action. One possible definition of the EE KPI is: EE = Total Network Throughput / Total Network Energy Consumption.

[0199] It should be noted that the first type of feedback information can be sent as a separate message before the analysis report notification, or it can be included in the analysis report notification.

[0200] In some embodiments, the processing unit 720 may be known by default to the second type of feedback information supported by the second network device, that is, the processing unit 720 knows the type of feedback information that the second network device can feed back. Further, the processing unit 720 determines the first type of feedback information based on the second type, that is, the processing unit 720 instructs the second network device to feed back which type of information based on the feedback capability of the second network device.

[0201] In some embodiments, the processing unit 720 is unaware of the second type of feedback information supported by the second network device. The second network device can send a first indication message to the transceiver unit 710, indicating the second type of feedback information supported by the second network device, i.e., the second network device reports its feedback capability. Further, the processing unit 720 determines the first type of feedback information based on the first indication message. Optionally, the first indication message can be sent by the second network device to the transceiver unit 710 before the transceiver unit 710 sends the first type of feedback information, indicating the reporting capability of the second network device in advance. Optionally, the first indication message can be sent by the transceiver unit 710 after the transceiver unit 710 sends the first type of feedback information to the second network device. Specifically, upon receiving the first type of feedback information and discovering that it cannot report the feedback information corresponding to the first type, the second network device further sends the first indication message to the transceiver unit 710 to report its feedback capability.

[0202] It should be noted that the first indication information indicates the type of feedback information that the second network device can feed back for a specific analysis type. Optionally, the first indication information includes an analysis identifier, and further includes the type of feedback information that can be fed back corresponding to the analysis identifier.

[0203] It should be noted that, prior to this step, the transceiver unit 710 sends the first configuration information to the second network device.

[0204] It should be noted that after the transceiver unit 710 sends the first configuration information to the second network device, the second network device sends an analysis report request message to the transceiver unit 710. This analysis report request message is used to request an analysis report of a specific analysis type. Optionally, after receiving the analysis report request message from the second network device, the transceiver unit 710 further sends a response message to the second network device. This response message indicates that the transceiver unit 710 has received the analysis report request message. As an example and not a limitation, this response message may include an analysis identifier, which corresponds to the analysis identifier in the aforementioned analysis report request message. It should be noted that after receiving the aforementioned analysis report request message, the transceiver unit 710 further performs data analysis based on the analysis report request message, prepares an analysis report, and sends an analysis report notification to the second terminal device. This analysis report notification may include, but is not limited to: an analysis identifier (analytics ID), the timestamp of analytics, the valid period of the analysis, the confidence level of the analysis results, and the specific analysis results. Optionally, the analysis results may include the model's output, such as a predicted value or a recommendation.

[0205] It should be noted that the first type of feedback information can be sent in the analysis report or as a separate message before sending the analysis report. The first type of feedback information is sent separately before the analysis report notification. Furthermore, it is sent separately by the transceiver unit 710 to the second network device before the second network device sends the analysis report request message to the transceiver unit 710, and is not included in the analysis report notification. For the same analysis report type, only the first type of feedback information needs to be sent once, eliminating the need to repeatedly carry the same first type of feedback information in each analysis report notification, thus saving some transmission overhead.

[0206] It should be noted that the first type of feedback information may be determined by the processing unit 720 in combination with at least one of the following factors:

[0207] Analysis type, which is the type requested in the analysis report request message mentioned above.

[0208] The second type of feedback information that the second network device is able to provide, as indicated in the first instruction information sent by the default or second network device.

[0209] As an example and not a limitation, in energy-saving scenarios, the value of feedbackInfo can be actionTakenindication or EE KPI. However, if the first indication information sent by the second network device indicates that the type of feedback information that the second network device can provide only includes actionTaken indication, then in this scenario, feedbackInfo can only be configured as actionTaken indication.

[0210] The transceiver unit 710 is also used to receive feedback information sent by the second network device.

[0211] The processing unit 720 is also used to determine the performance of the model based on the feedback information.

[0212] Optionally, after receiving the feedback information sent by the second network device, the transceiver unit 710 sends a first request message to the third network device according to the content of the feedback information. This first request message requests the third network device to obtain the model's true value or other data. It should be noted that this other data can be the network performance indicators corresponding to the operation instruction information described in the relevant example in step S301. It should also be noted that the model whose performance needs to be determined by the processing unit 720 is the model trained by the aforementioned processing unit 720. Upon receiving the first request message, the third network device further sends the model's true value or other data to the transceiver unit 710 according to the first request message. This third network device can be a third-party device capable of providing data; this is not a limitation but an example. The third network device can be: an application function (AF), a network data analytics function (NWDAF), a base station, etc. For example, in the handover optimization scenario, in step S301, feedbackInfo = actionTaken indication. The feedback information from the second network device is the finally determined target base station identifier and the handover time. The processing unit 720 sends a first request to the third network device as needed, further requesting the terminal device to switch to the actual target base station and obtain the Quality of Experience (QoE). In this scenario, the third network device is a third-party device that can provide QoE information. The third network device can be the target base station or AF, etc.

[0213] In some embodiments, the processing unit 720 determines model performance based on feedback information, or based on feedback information and the model's actual value, or based on feedback information and network performance indicators. Furthermore, it adjusts the data analysis strategy based on the model's performance. It should be noted that the model's actual value or network performance indicator is obtained by the transceiver unit 710 from a third-party device after receiving the feedback information.

[0214] As an example and not a limitation, the processing unit 720 may adjust the data analysis strategy in at least one of the following aspects:

[0215] Use feedback information to retrain the model.

[0216] If other models are available, use those models for data analysis, for example, models with higher accuracy.

[0217] Collect new training data, train a new model from scratch, and optionally, reprocess the data or perform feature selection.

[0218] Optionally, if the model retraining or retraining a new model takes a long time, the model may be temporarily unusable. In one possible implementation, the transceiver unit 710 sends second configuration information to the second network device. The second configuration information includes an analysis identifier, which indicates that the analysis type corresponding to the analysis identifier is temporarily unavailable.

[0219] In some embodiments, the communication device 700 may be the first network device in the method embodiment 400 above, or it may be a chip for implementing the functions of the first network device in the method embodiment above. It should be understood that the communication device 700 may correspond to the steps of the first network device in the method 400 of this application embodiment. The communication device 700 includes:

[0220] Transceiver unit 710: Used to send first configuration information to the second network device.

[0221] Processing unit 720: Used to determine the performance of the model based on feedback information.

[0222] Specifically, the first configuration information includes an analytics ID, which indicates the types of analytics that the second network device processing unit 720 can provide. Optionally, if the processing unit 720 can provide multiple types of analytics, it may include a list of analytics IDs. The value of the analytics type under each analytics ID uniquely identifies a type of analytics. As an example and not a limitation, this value could be: handover optimization, coverage issue analysis, MDA-assisted energy saving, etc.

[0223] The transceiver unit 710 is also configured to receive first indication information, which indicates the type of feedback information supported by the second network device.

[0224] It should be understood that this step is optional, and the second type can be the default for processing unit 720.

[0225] Specifically, the first indication information may indicate a second type of feedback information that supports a specific analysis type. This first indication information may include an analytics ID and a second type of supported feedback information corresponding to that analytics ID. The second type of supported feedback information can be represented by `availableFeedbackInfo`. The types of feedback information supported by the second network device include, but are not limited to:

[0226] Model evaluation metrics: Metrics that can be used to evaluate model performance; for a detailed description, please refer to the device corresponding to method 300.

[0227] Operation instruction information: Instructions for the actions to be taken by the second network device after receiving the analysis report. For details, please refer to the device corresponding to method 300.

[0228] Network performance metrics: Metrics that can be used to describe network performance; for details, please refer to the device corresponding to method 300.

[0229] The transceiver unit 710 is also used to send a first type of feedback information to a second network device.

[0230] Specifically, the message content may include an analytics ID and the first type of feedback information corresponding to the analytics type of that ID: feedbackInfo. Optional values ​​for feedbackInfo include, but are not limited to: model evaluation metrics, operational instructions, network performance metrics, etc.

[0231] It should be noted that the first type of feedback information sent by the transceiver unit 710 to the second network device can be sent as a separate message or included in the analysis report notification message. In some embodiments, the first type of feedback information sent by the transceiver unit 710 to the second network device is sent in advance as a separate message, thus decoupling this message from the analysis report notification. For the same analysis type, only the first type of feedback information needs to be configured once, without having to repeatedly carry this information in each analysis report notification, which can save some transmission resource overhead.

[0232] It should be noted that if the processing unit 720 does not have a default second type and the transceiver unit 710 has not received the first indication information, the first network device may have misconfigured the first type. The first type of the feedback information sent by the transceiver unit 710 to the second network device may be a type of feedback information that the second network device does not support. Optionally, after the processing unit 720 configures an unsupported first type, the second network device sends a feedback modification notification to the transceiver unit 710 to indicate the second type of feedback information that the second network device supports. The processing unit 720 re-determines the first type of feedback information based on the feedback modification notification and resends the message to the second network device. It should be noted that the first type of feedback information message sent initially and the first type of feedback information message sent resent may be the same message. It should also be noted that the feedback modification notification may include the unsupported feedback information type (unavaliableInfo) and the reason for unsupportion (reason), and may also include the feedback information type that it can support (avaliableInfo). The values ​​of unavaliableInfo and availableInfo include, but are not limited to, model evaluation metrics, operation indication information, network performance metrics, etc. It should be noted that the feedback modification notification and the aforementioned first instruction information can be a single message.

[0233] The transceiver unit 710 is also used to receive analysis report requests sent by the second network device.

[0234] Specifically, the second network device can request an analysis report of a certain type of analysis. The content of the message includes the analytics ID, and optionally, the message may also include one or more of the following: analysis area, reporting method, reporting period, etc.

[0235] Optionally, the transceiver unit 710 receives the analysis report request and sends a response message to the second network device. The response message may include an analytics ID to indicate that the request has been received.

[0236] The transceiver unit 710 is also used to send an analysis report notification to the second network device.

[0237] It should be noted that the first type of feedback message can be included in the analysis report notification, rather than being sent as a separate message.

[0238] Specifically, after receiving the analysis report request, the transceiver unit 710 performs data analysis according to the request of the second network device, prepares the analysis report, and further sends an analysis report notification to the second network device. The analysis report notification includes the analysis identifier (analytics ID), the timestamp of analytics, the valid period of the analysis, the confidence level of the analysis results, and the specific analysis results (analyticsresults). When the first type of feedback information is included in the analysis report notification, the message also includes the first type of feedback information (feedbackInfo) and its corresponding value.

[0239] The transceiver unit 710 is also used to receive feedback information.

[0240] The transceiver unit 710 is also used to send request information to a third network device.

[0241] Specifically, after receiving feedback information, the transceiver unit 710 needs more data to evaluate model performance. It can send a request to the third network device to obtain more additional data or the model's actual values. As an example, and not a limitation, in a handover scenario, if the feedback information returned by the second network device corresponds to the first type of operation instruction information, then the feedback information from the second network device may include the executed action and the timestamp of the executed action. After receiving the feedback information, the transceiver unit 710 can obtain the UE's actual QoE from the third network device (target base station) to evaluate model performance. As an example, and not a limitation, in an energy-saving scenario, if the feedback information returned by the second network device corresponds to the first type of operation instruction information, then the feedback information from the second network device may include the energy-saving actions taken. After receiving the feedback information, the transceiver unit 710 can obtain other data from the third network device, such as network performance indicators after taking energy-saving actions, to evaluate the model.

[0242] Correspondingly, the transceiver unit 710 receives the actual value or other performance data.

[0243] The processing unit 720 is used to evaluate the performance of the model.

[0244] Specifically, the processing unit 720 evaluates the model's performance based on feedback information, or the processing unit 720 evaluates the model's performance based on other received data or the model's true value and feedback information.

[0245] Optionally, the processing unit 720 can adjust the performance of poorly performing models, and the adjustment methods include, but are not limited to, the following:

[0246] Retrain the model;

[0247] Call other models that can be used for data analysis; for example, you can choose a model with higher accuracy.

[0248] Collect new data and train a new model from scratch. Optionally, you can re-preprocess the data, perform feature selection, etc.

[0249] The transceiver unit 710 is also used to send second configuration information.

[0250] Specifically, when the model takes too long to retrain or retrain, making the model temporarily unusable, the processing unit 710 can send second configuration information to the second network device, indicating that a certain type of analysis is temporarily unavailable. This information may include the analytics ID.

[0251] In some embodiments, the communication device 700 may be the first network device in the method embodiment 500 above, or it may be a chip for implementing the functions of the first network device in the method embodiment above. It should be understood that the communication device 700 may correspond to the steps of the first network device in the method 500 of this application embodiment. The communication device 700 includes:

[0252] Transceiver unit 710: Used to send a first request message to the second network device.

[0253] Processing unit 720: Used to evaluate the performance of the model based on the model's true values.

[0254] The first request information instructs the second network device to provide feedback on the true value of the model to the transceiver unit 710.

[0255] It should be noted that this step is optional, and the second network device can also send the actual value of the model to the transceiver unit 710 by default.

[0256] It should be noted that the first request information may be included in the analysis report notification or sent as a separate message. The steps preceding this step and the specific configuration of this configuration information are similar to those of the corresponding apparatus for method 300; please refer to the relevant description of the apparatus corresponding to method 300.

[0257] The transceiver unit 710 is also used to receive feedback information sent by the second network device.

[0258] This feedback information represents the model's true value.

[0259] In some embodiments, the communication device 700 may be the first network device in the method embodiment 600 above, or it may be a chip for implementing the functions of the first network device in the method embodiment above. It should be understood that the communication device 700 may correspond to the steps of the first network device in the method 600 of this application embodiment. The communication device 700 includes:

[0260] Transceiver unit 710: Used to send first configuration information to the second network device.

[0261] Processing unit 720: Used to determine the performance of the model based on feedback information.

[0262] The transceiver unit 710 is also used to receive first indication information sent by the second network device, which indicates that the second network device supports the true value of the feedback model. It should be noted that the second network device can either provide the true value of the feedback model or use the default value provided by the first network device.

[0263] The transceiver unit 710 is also used to send a first request message to a second network device.

[0264] Specifically, the first request information requests the second network device to return the actual value of the model. It should be noted that this step is optional, and the second network device can default to returning the actual value of the model to the transceiver unit 710.

[0265] It should be noted that the first request information sent by the transceiver unit 710 to the second network device can be sent as a separate message or included in the analysis report notification message. In some embodiments, the first request information sent by the transceiver unit 710 to the second network device is sent in advance as a separate message, thus decoupling this information from the analysis report notification. For the same analysis type, it only needs to be sent once, without having to carry this information repeatedly in each analysis report notification, which can save some transmission resource overhead.

[0266] For details regarding the specific network signaling and content of this message, please refer to the relevant description in the device corresponding to method 400.

[0267] The transceiver unit 710 is also used to receive analysis report notifications sent by the second network device.

[0268] The transceiver unit 710 is also used to receive feedback information.

[0269] The processing unit 720 is also used to obtain other performance data based on the model's true values.

[0270] It should be noted that the processing unit 720 may not be able to evaluate the model performance based solely on the model's actual values. The first network device may acquire other performance data as needed and further evaluate the model based on the actual values ​​and performance data.

[0271] The processing unit 720 is also used to determine the second configuration information.

[0272] The transceiver unit 710 is also used to send second configuration information to the second network device.

[0273] In some embodiments, the communication device 700 may be the second network device in the method embodiment 300 above, or it may be a chip for implementing the functions of the second network device in the method embodiment above. It should be understood that the communication device 700 may correspond to the steps of the second network device in the method 300 of this application embodiment. The communication device 700 includes:

[0274] Transceiver unit 710: Used to receive feedback information of the first type from the first network device.

[0275] Processing unit 720: used to determine feedback information based on the first type of feedback information.

[0276] Specifically, the first type includes one or more of the following types: model evaluation metrics, operational instruction information, and network performance metrics.

[0277] Model evaluation metrics are used to assess model performance. Specifically, these metrics are indicators that evaluate the quality of a model's performance. Commonly used metrics include: precision, accuracy, error rate, recall, mean squared error, root mean square error, root mean square logarithmic error, and mean absolute error. The first network device can determine the model's performance based on the feedback model evaluation metrics and adjust poorly performing models. Model evaluation metrics can be obtained from the predicted values ​​of the model provided by the first network device and the actual values ​​of the model obtained by the processing unit 720.

[0278] As an example rather than a limitation, in some scenarios, the first network device requires the processing unit 720 to provide feedback on the model evaluation index. The processing unit 720 can obtain the true value of the model itself. Furthermore, the processing unit 720 can derive the first index value based on the model prediction value and the model's true value and provide feedback as the value of the model evaluation index.

[0279] As an example and not a limitation, in some scenarios, the first network device requests the processing unit 720 to provide a model evaluation index. The processing unit 720 itself cannot obtain the true value of the model. The transceiver unit 710 can request the fifth network device to obtain the true value of the model. Furthermore, the processing unit 720 derives the first index value based on the requested true value of the model and the prediction value given by the first network device, and provides it as the value of the model evaluation index.

[0280] It should be noted that this first metric value can be calculated based on the model's predicted values ​​and at least one true value. It can be the model's precision, accuracy, error rate, recall, mean squared error, root mean square error, root mean square logarithmic error, mean absolute error, etc. In certain scenarios, such as when multiple true values ​​exist, using feedback model evaluation metrics can save some feedback resources.

[0281] The operation instruction information describes the specific actions taken by the transceiver unit 710 after receiving the analysis report, based on the recommendations of the first network device or the predictions of the first network device, including the specific actions taken and the timestamps of the actions taken.

[0282] In some embodiments, the first network device can collect relevant data from the third network device or the actual value of the model after the processing unit 720 executes the recommended action, based on the operation instruction information fed back by the transceiver unit 710. This data is used to determine the model's performance and then adjust models with poor performance. It should be noted that this relevant data can be the network performance indicators corresponding to the execution of the recommended action.

[0283] As an example and not a limitation, the first network device can collect the actual value of the model after executing the recommended actions from the third network device based on the operation instruction information fed back by the transceiver unit 710. In the handover optimization scenario, the device can be the source base station. The transceiver unit 710 can request a handover optimization analysis report from the first network device and make a handover decision with the assistance of MDAS. The handover optimization analysis report includes the recommended actions and may also include the predicted value after executing the actions, i.e., the predicted QoE. In other words, it is the QoE that the UE is expected to achieve after handover to a certain target gNB. The source gNB can refer to the prediction results in the analysis report to select the most suitable target gNB to perform the handover. At this time, the first network device requests the transceiver unit 710 to feed back operation instruction information. The transceiver unit 710 may return the handover action taken and the timestamp of the action taken. Optionally, the first network device can obtain the actual QoE from the third network device (which can be the target base station) based on the operation instruction information fed back by the transceiver unit 710, and then evaluate the model performance based on the feedback information and the actual value of the model.

[0284] As an example, and not a limitation, in an energy-saving optimization scenario, the analysis report provided by the first network device to the processing unit 720 includes energy-saving suggestions. Specifically, the suggestions may include: recommended NR cells to enter energy-saving mode, recommended candidate cells with precedence for taking over the traffic of the ES-Cell, and a time period during which ES is or is not allowed. In this scenario, the analysis report provides recommended suggestions. After the processing unit 720 executes the action corresponding to the recommended suggestions, it feeds back operation instruction information to the first network device. Upon receiving the operation instruction information, the first network device can obtain the network performance indicators after executing the action from the third network device, and then evaluate the model based on the feedback information and the network performance indicators.

[0285] The network performance metrics are indicators defined by the first network device that describe network performance, such as standard-defined Key Performance Indicators (KPIs) or custom optimization objective functions. The processing unit 720 can execute actions based on the predicted values ​​or recommendations provided by the model from the first network device. After executing these actions, it obtains some network performance metrics and further sends feedback information to the first network device. This feedback information can be the data corresponding to the network performance metrics. The first network device determines whether the expected network performance has been achieved based on the feedback network performance metrics, determines the model's performance, and adjusts models with poor performance.

[0286] In some possible implementations, the network performance metrics can be those obtained after executing the recommended action. It should be understood that the recommended action can correspond to the operation instruction information. For example, the first network device can request the transceiver unit 710 to provide operation instruction information. The operation instruction information provided by the transceiver unit 710 contains the action to be performed according to the recommended action. Then, the first network device can obtain the network performance metrics obtained after executing the recommended action from the third network device based on the operation instruction information for model evaluation. Alternatively, the first network device can request the transceiver unit 710 to directly provide the network performance metrics. After the processing unit 720 executes the recommended action, it obtains the network performance metrics from the fourth network device and provides them back to the first network device as feedback information. Specific examples can be found in the relevant descriptions within the operation instruction information.

[0287] It should be noted that the first type of feedback information can be configured through the feedbackInfo signaling. The selectable values ​​for feedbackInfo differ depending on the scenario. Specifically, the value of feedbackInfo indicates the first type of feedback information sent by the transceiver unit 710 to the first network device. Furthermore, this value can determine the specific content of the feedback information required for that type. As an example, not a limitation, in a handover optimization scenario, feedbackInfo = actionTakenindication, indicating that the transceiver unit 710 needs to report the final determined target gNB identifier and the handover time point. In an energy-saving scenario, feedbackInfo = actionTaken indication, indicating that the transceiver unit 710 needs to report the final energy-saving action taken, i.e., the cell identifier for enabling / disabling energy-saving features and the time point. In another energy-saving scenario, feedbackInfo = Energy Efficiency (EE), indicating that the transceiver unit 710 needs to report the energy efficiency KPI value after taking the specified energy-saving action. One possible definition of the EE KPI is: EE = Total Network Throughput / Total Network Energy Consumption.

[0288] It should be noted that the first type of feedback information can be sent as a separate message before the analysis report notification, or it can be included in the analysis report notification.

[0289] In some embodiments, the first network device may be known by default to the second type of feedback information supported by the device, that is, the first network device knows the type of feedback information that the processing unit 720 can feed back. Further, the first network device determines the first type of feedback information based on the second type, that is, the first network device instructs the second network device to feed back which type of information based on the feedback capability of the processing unit 720.

[0290] In some embodiments, the first network device is unaware of the second type of feedback information supported by the processing unit 720. The transceiver unit 710 can send a first indication message to the first network device, indicating the second type of feedback information supported by the processing unit 720, i.e., the transceiver unit 710 reports its feedback capability. Further, the first network device determines the first type of feedback information based on the first indication message. Optionally, the first indication message can be sent by the transceiver unit 710 to the first network device before the first network device sends the first type of feedback information, indicating the reporting capability of the transceiver unit 710 in advance. Optionally, the first indication message can be sent by the first network device after the first network device sends the first type of feedback information to the transceiver unit 710. Specifically, upon receiving the first type of feedback information and discovering that it cannot report the feedback information corresponding to the first type, the transceiver unit 710 further sends the first indication message to the first network device to report its feedback capability.

[0291] It should be noted that the first indication information indicates the type of feedback information that the processing unit 720 can provide for a specific analysis type. Optionally, the first indication information includes an analysis identifier, and further includes the type of feedback information that can be provided corresponding to the analysis identifier.

[0292] It should be noted that, prior to this step, the first network device sends the first configuration information to the transceiver unit 710.

[0293] It should be noted that after the first network device sends the first configuration information to the transceiver unit 710, the transceiver unit 710 sends an analysis report request message to the first network device. This analysis report request message is used to request an analysis report of a specific analysis type. Optionally, after receiving the analysis report request message from the transceiver unit 710, the first network device further sends a response message to the transceiver unit 710, indicating that the first network device has received the analysis report request message. As an example and not a limitation, the response message may include an analysis identifier, which corresponds to the analysis identifier in the aforementioned analysis report request message. It should be noted that after receiving the aforementioned analysis report request message, the first network device further performs data analysis based on the analysis report request message, prepares an analysis report, and sends an analysis report notification to the transceiver unit 710. This analysis report notification may include, but is not limited to: an analysis identifier (analytics ID), the timestamp of analytics, the valid period of the analysis, the confidence level of the analysis results, and the specific analysis results. Optionally, the analysis results may include the model's output, such as a predicted value or a recommendation.

[0294] It should be noted that the first type of feedback information can be sent in the analysis report or as a separate message before sending the analysis report. If the first type of feedback information is sent separately before the analysis report notification, or even more specifically, if the first network device sends the analysis report request message separately to the transceiver unit 710 before the transceiver unit 710 sends it, and it is not included in the analysis report notification, then for the same analysis report type, only the first type of feedback information needs to be sent once, eliminating the need to repeatedly include the same first type of feedback information in each analysis report notification, thus saving some transmission overhead.

[0295] It should be noted that the first type of feedback information may be determined by the first network device in combination with at least one of the following factors:

[0296] Analysis type, which is the type requested in the analysis report request message mentioned above.

[0297] The second type of feedback information that the processing unit 720 can provide, as indicated in the first instruction information sent by the default or transceiver unit 710.

[0298] As an example and not a limitation, in energy-saving scenarios, the value of feedbackInfo can be actionTakenindication or EE KPI. However, if the first indication information sent by the transceiver unit 710 indicates that the type of feedback information that the processing unit 720 can provide only includes actionTaken indication, then in this scenario, feedbackInfo can only be configured as actionTaken indication.

[0299] The transceiver unit 710 is also used to receive the first type of feedback information.

[0300] The processing unit 720 is also configured to send feedback information to the first network device according to the first type of feedback information.

[0301] Optionally, the feedback information can be determined by the processing unit 720 itself based on the first type of feedback information. For example, the processing unit 720 can output the true value of the model, and the first type indicates that the average difference between the predicted value output by the model and the true value of the model should be used as the model evaluation index for feedback. In this case, the processing unit 720 can obtain the average value based on the first type for feedback.

[0302] Alternatively, the feedback information can be obtained by the transceiver unit 710 from the fourth network device. For example, if the first type of feedback information requested by the first network device is a network performance indicator, then the transceiver unit 710 can send the first type to the fourth network device and obtain the network performance indicator from the fourth network device. Alternatively, the feedback information can be determined by the transceiver unit 710 after obtaining the model's actual value from the fifth network device, based on the model's actual value and the model's predicted value in the analysis report. Specifically, the transceiver unit 710 sends the first type of feedback information to the fifth network device and further receives the model's actual value from the fifth network device.

[0303] The transceiver unit 710 is also used to send feedback information to the first network device.

[0304] In some embodiments, the communication device 700 may be the second network device in the method embodiment 400 above, or it may be a chip for implementing the functions of the second network device in the method embodiment above. It should be understood that the communication device 700 may correspond to the steps of the second network device in the method 400 of this application embodiment. The communication device 700 includes:

[0305] Transceiver unit 710: used to receive first configuration information from the first network device.

[0306] Processing unit 720: Used to send first instruction information to the first network device.

[0307] Specifically, the first configuration information includes an analytics ID, which indicates the types of analytics that the processing unit 720, the first network device, can provide. Optionally, if the first network device can provide multiple types of analytics, a list of analytics IDs may be included. The value of the analytics type under each analytics ID uniquely identifies a type of analytics. As an example and not a limitation, this value could be: handover optimization, coverage issue analysis, MDA-assisted energy saving, etc.

[0308] The first indication information is used to indicate a second type of feedback information supported by the second network device. This second type can be the default for the first network device.

[0309] Specifically, the first indication information can indicate a second type of feedback information that supports a specific analysis type. This first indication information may include an analytics ID and a second type of supported feedback information corresponding to that analytics ID. The second type of supported feedback information can be represented by `availableFeedbackInfo`. The types of feedback information supported by the processing unit 720 include, but are not limited to:

[0310] Model evaluation metrics: Metrics that can be used to evaluate model performance; for a detailed description, please refer to the device corresponding to method 300.

[0311] Operation instruction information: Instructions for the actions to be taken by the transceiver unit 710 after receiving the analysis report. For details, please refer to the device corresponding to method 300.

[0312] Network performance metrics: Metrics that can be used to describe network performance; for details, please refer to the device corresponding to method 300.

[0313] The transceiver unit is also used to receive a first type of feedback information sent by a first network device.

[0314] Specifically, the message content may include an analytics ID and the first type of feedback information corresponding to the analytics type of that ID: feedbackInfo. Optional values ​​for feedbackInfo include, but are not limited to: model evaluation metrics, operational instructions, network performance metrics, etc.

[0315] It should be noted that the first type of feedback information sent by the first network device to the transceiver unit 710 can be sent as a separate message or included in the analysis report notification message. In some embodiments, the first type of feedback information sent by the first network device to the transceiver unit 710 is sent in advance as a separate message, thus decoupling this message from the analysis report notification. For the same analysis type, only the first type of feedback information needs to be configured once, without having to repeatedly carry this information in each analysis report notification, which can save some transmission resource overhead.

[0316] It should be noted that if the first network device does not have a default second type, it may have misconfigured the first type. The first type of feedback information sent by the first network device to the transceiver unit 710 may be a type of feedback information not supported by the processing unit 720. Optionally, after configuring an unsupported first type on the first network device, the transceiver unit 710 sends a feedback modification notification to the first network device, indicating the second type of feedback information supported by the processing unit 720. The first network device re-determines the first type of feedback information based on the feedback modification notification and resends the message to the transceiver unit 710. It should be noted that the first type of feedback information sent initially and the first type of feedback information sent resent may be the same message. It should also be noted that the feedback modification notification may include the unsupported feedback information type (unavaliableInfo) and the reason for unsupportion (reason), and may also include the feedback information type it can support (avaliableInfo). The values ​​of unavaliableInfo and availableInfo include, but are not limited to, model evaluation metrics, operation instruction information, and network performance metrics. It should be noted that the feedback modification notification and the aforementioned first instruction information can be a single message.

[0317] The transceiver unit 710 is also used to send an analysis report request to the first network device.

[0318] Specifically, the transceiver unit 710 can request an analysis report of a certain type of analysis. The content of the message includes the analytics ID. Optionally, the message may also include one or more of the following: analysis area, reporting method, reporting period, etc.

[0319] Optionally, the first network device receives the analysis report request and sends a response message to the transceiver unit 710. The response message may include an analytics ID to indicate that the first network device has received the request.

[0320] The transceiver unit 710 is also used to receive analysis report notifications sent by the first network device.

[0321] It should be noted that the first type of feedback message can be included in the analysis report notification, rather than being sent as a separate message.

[0322] Specifically, after receiving the analysis report request, the first network device performs data analysis according to the request of the processing unit 720, prepares the analysis report, and further sends an analysis report notification to the transceiver unit 710. The analysis report notification includes the analysis identifier (analytics ID), the timestamp of analytics, the valid period of the analysis, the confidence level of the analysis results, and the specific analysis results. When the first type of feedback information is included in the analysis report notification, the message also includes the first type of feedback information (feedbackInfo) and its corresponding value.

[0323] The processing unit 720 is also used to determine the feedback information based on the first type of the feedback information.

[0324] Optionally, the processing unit 720 itself can derive the true value of the model, process the data, and determine the feedback information. For a specific example, please refer to the apparatus in method 300.

[0325] Optionally, the transceiver unit 710 can send a first type of feedback information to the fourth network device and receive feedback information from the fourth network device. For specific examples, please refer to the apparatus in method 300.

[0326] Optionally, the transceiver unit 710 can send a request message to the fifth network device, obtain the true value of the model from the fifth network device, and determine the feedback information based on the true value and the first type of the feedback information. A specific example can be found in the apparatus of method 300.

[0327] In some embodiments, the communication device 700 may be the second network device in the method embodiment 500 above, or it may be a chip for implementing the functions of the second network device in the method embodiment above. It should be understood that the communication device 700 may correspond to the steps of the second network device in the method 500 of this application embodiment. The communication device 700 includes:

[0328] Transceiver unit 710: Receives the first request information sent by the first network device.

[0329] Since the processing unit 720 itself cannot output the true value of the model, it requests the true value of the model from the third network device. As an example rather than a limitation, in the handover optimization scenario, this device is the source base station. The source base station itself cannot obtain the QoE information after the UE handover. Therefore, the transceiver unit 710 requests the third network device, i.e. the target base station, to obtain the QoE after the UE handover and sends it to the first network device as feedback information.

[0330] The transceiver unit 710 is also used to receive the actual values ​​of the model from other network devices.

[0331] The transceiver unit 710 is also used to send feedback information to the first network device.

[0332] In some embodiments, the communication device 700 may be the second network device in the method embodiment 600 above, or it may be a chip for implementing the functions of the second network device in the method embodiment above. It should be understood that the communication device 700 may correspond to the steps of the second network device in the method 600 of this application embodiment. The communication device 700 includes:

[0333] Transceiver unit 710: used to receive first configuration information from the first network device.

[0334] The transceiver unit 710 is also used to send a first indication information to the first network device, the first indication information being used to indicate the true value of the feedback model supported by the second network device.

[0335] The transceiver unit 710 is also configured to receive a first request message sent by the first network device, the first request message requesting the transceiver unit 710 to provide the actual value of the model. It should be noted that the transceiver unit 710 may, by default, provide the actual value of the model to the first network device.

[0336] It should be noted that the first request information sent by the first network device to the transceiver unit 710 can be sent as a separate message or included in the analysis report notification message. In some embodiments, the first request information sent by the first network device to the transceiver unit 710 is sent in advance as a separate message, thus decoupling this information from the analysis report notification. For the same analysis type, it only needs to be sent once, without having to carry this information repeatedly in each analysis report notification, which can save some transmission resource overhead.

[0337] The transceiver unit 710 is also used to send an analysis report request to the first network device.

[0338] The transceiver unit 710 is also used to receive analysis report notifications from the first network device.

[0339] The transceiver unit 710 is also used to obtain real values ​​from other network device models.

[0340] The transceiver unit 710 is also used to send feedback information to the first network device.

[0341] It should be noted that this feedback information is the model's actual value.

[0342] Figure 8 This is a schematic diagram of a communication device 800 according to an embodiment of this application. The communication device 800 includes a transceiver 810, a processor 820, and a memory 830. The memory 830 is used to store instructions. The processor 820 is coupled to the memory 830 and is used to execute the instructions stored in the memory to perform the method provided in the embodiment of this application described above.

[0343] Specifically, the transceiver 810 in the communication device 800 can correspond to the transceiver unit 710 in the communication device 700, and the processor 820 in the communication device 800 can correspond to the processing unit 720 in the communication device 700.

[0344] It should be understood that the memory 830 and processor 820 described above can be combined into a single processing device, with the processor 820 executing the program code stored in the memory 830 to achieve the aforementioned functions. In specific implementations, the memory 830 can be integrated into the processor 820 or independent of the processor 820.

[0345] It should be understood that the specific process by which each transceiver processor performs the corresponding steps described above has been explained in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0346] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0347] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0348] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0349] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0350] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0351] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0352] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for transmitting information, characterized in that, include: A first type of feedback information is sent from a first network device to a second network device. The first type includes one or more of the following: model evaluation metrics, operation instruction information, and network performance metrics. The model evaluation metrics are used to indicate the performance of the model. The operation instruction information is used to indicate the operation to be performed based on the analysis results of the model. The network performance metrics are used to indicate the network metrics obtained based on the analysis results of the model. The first type is used to request the feedback information. The first network device receives the feedback information from the second network device; The first network device determines the performance of the model based on the feedback information; The method further includes: The first network device receives the actual value of the model or the network performance index from the third network device; Wherein, the first network device determines the performance of the model based on the feedback information, including: The first network device determines the performance of the model based on the true value of the model or the network performance indicators and the feedback information.

2. The method according to claim 1, characterized in that, The method further includes: The first network device receives first indication information from the second network device, the first indication information indicating a second type of feedback information supported by the second network device; The first network device determines the first type based on the first indication information.

3. A method for transmitting information, characterized in that, include: The second network device receives a first type of feedback information from the first network device. The first type includes one or more of the following: model evaluation metrics, operation instruction information, and network performance metrics. The model evaluation metrics are used to indicate the performance of the model. The operation instruction information is used to indicate the operation to be performed based on the analysis results of the model. The network performance metrics are used to indicate the network metrics obtained based on the analysis results of the model. The first type is used to request the feedback information. The second network device sends the feedback information to the first network device according to the first type of the feedback information; The feedback information is used by the first network device to determine the performance of the model based on the true value of the model received from the third network device or the network performance index and the feedback information.

4. The method according to claim 3, characterized in that, The second network device sends the feedback information to the first network device according to the first type of the feedback information, including: The second network device determines the feedback information based on the first type of the feedback information; The second network device sends the feedback information to the first network device.

5. The method according to claim 3, characterized in that, The second network device sends the feedback information to the first network device according to the first type of the feedback information, including: The second network device receives the feedback information from the fourth network device; The second network device sends the feedback information to the first network device.

6. The method according to claim 3, characterized in that, The second network device determines the feedback information based on the first type of the feedback information, including: The second network device receives the actual value of the model from the fifth network device; The second network device determines the feedback information based on the first type and the true value of the model; The second network device sends the feedback information to the first network device.

7. The method according to any one of claims 3 to 6, characterized in that, The method further includes: The second network device sends a first indication message to the first network device, the first indication message indicating a second type of feedback information supported by the second network device.

8. A communication device, characterized in that, include: A transceiver unit is used to send a first type of feedback information to a second network device. The first type includes one or more of the following: model evaluation metrics, operation instruction information, and network performance metrics. The model evaluation metrics are used to indicate the performance of the model. The operation instruction information is used to indicate the operation to be performed based on the analysis results of the model. The network performance metrics are used to indicate the network metrics obtained based on the analysis results of the model. The first type is used to request the feedback information. The transceiver unit is also configured to receive the feedback information from the second network device; A processing unit is configured to determine the performance of the model based on the feedback information; The transceiver unit is also used for: Receive the actual value of the model or the network performance index from a third network device; The processing unit is used to determine the performance of the model based on the feedback information, including: The processing unit determines the performance of the model based on the model's true value or the network performance indicators and the feedback information.

9. The apparatus according to claim 8, characterized in that, The transceiver unit is also used for: Receive first indication information from the second network device, the first indication information indicating a second type of feedback information supported by the second network device; The processing unit is further configured to determine the first type based on the first indication information.

10. A communication device, characterized in that, include: A transceiver unit is configured to receive a first type of feedback information from a first network device. The first type includes one or more of the following: model evaluation metrics, operation instruction information, and network performance metrics. The model evaluation metrics are used to indicate the performance of the model. The operation instruction information is used to indicate the operation to be performed based on the analysis results of the model. The network performance metrics are used to indicate the network metrics obtained based on the analysis results of the model. The first type is used to request the feedback information. The transceiver unit is also used to send the feedback information to the first network device; The transceiver unit is also configured to send the feedback information to the first network device according to the first type of the feedback information; The feedback information is used by the first network device to determine the performance of the model based on the true value of the model received from the third network device or the network performance index and the feedback information.

11. The apparatus according to claim 10, characterized in that, The transceiver unit sends the feedback information to the first network device according to the first type of the feedback information, including: The processing unit determines the feedback information based on the first type of the feedback information; The transceiver unit sends the feedback information to the first network device.

12. The apparatus according to claim 10, characterized in that, The processing unit sends the feedback information to the first network device according to the first type of the feedback information, and the transceiver unit receives the feedback information from the fourth network device. The transceiver unit sends the feedback information to the first network device.

13. The apparatus according to claim 11, characterized in that, The processing unit determines the feedback information based on the first type of the feedback information, including: The transceiver unit receives the actual value of the model from the fifth network device; The processing unit determines the feedback information based on the first type and the true value of the model; The transceiver unit sends the feedback information to the first network device.

14. The apparatus according to any one of claims 10 to 13, characterized in that, The transceiver unit is also used for: Send a first indication message to the first network device, the first indication message indicating a second type of feedback information supported by the communication device.

15. A communication system, characterized in that, It includes the communication device according to any one of claims 8 or 9 and the communication device according to any one of claims 10-14.

16. A computer-readable storage medium, characterized in that, The computer-readable medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

17. A chip, characterized in that, It includes a processor and a memory, the memory being used to store computer programs, and the processor being used to invoke and run computer media stored in the memory to perform the method as described in any one of claims 1-7.

Images