Model performance monitoring method, device, system, and storage medium

Abstract

The present disclosure relates to a model performance monitoring method, a device, a system, and a storage medium. The method may be executed by a network device. The method comprises: sending first information to a terminal, the first information being used for instructing the terminal to generate second information on the basis of a first AI / ML model; receiving the second information sent by the terminal; and on the basis of the second information, determining whether the first AI / ML model meets a performance requirement, and / or a reason why the performance of the first AI / ML model changes. The present invention can effectively ensure that a model deployed on a terminal side in a bilateral model has good performance, and ensure the communication reliability.

Description

Model performance monitoring method, device, system and storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and particularly relates to a model performance monitoring method, device, system and storage medium. BACKGROUND

[0002] With the development of Artificial Intelligence (AI) and Machine Learning (ML) technologies, it is found that the feedback overhead of a terminal can be reduced or the CSI feedback accuracy can be improved by using AI technologies, for example, a bilateral AI / ML model in which a terminal-side channel state information (CSI) generation part model and a network-side CSI recovery part model are generated can be used to respectively implement compressed feedback and recovery of CSI.

[0003] SUMMARY

[0004] The present disclosure provides a model performance monitoring method, device, system and storage medium.

[0005] According to a first aspect of the present disclosure, a model performance monitoring method is provided, which is performed by a network device, and the method comprises:

[0006] sending first information to a terminal, wherein the first information is used to instruct the terminal to generate second information based on a first AI / ML model;

[0007] receiving the second information sent by the terminal;

[0008] determining whether the first AI / ML model meets a performance requirement and / or a cause of a performance change of the first AI / ML model according to the second information;

[0009] wherein model parameters of the first AI / ML model are same as model parameters of a second AI / ML model, or the first AI / ML model is obtained by updating a second AI / ML model and / or a third AI / ML model by the terminal;

[0010] The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0011] According to a second aspect of the present disclosure, a model performance monitoring method is provided, which is performed by a terminal, and the method comprises:

[0012] receiving the first information sent by the network device;

[0013] generating the second information based on the first AI / ML model according to the first information;

[0014] sending the second information to the network device, the second information being used by the network device to determine whether the first AI / ML model meets a performance requirement, and / or a reason for a performance change of the first AI / ML model;

[0015] wherein model parameters of the first AI / ML model are same as model parameters of a second AI / ML model, or the first AI / ML model is obtained by updating a second AI / ML model and / or a third AI / ML model by the terminal;

[0016] the second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part in a bilateral model, and the third AI / ML model is a second part in the bilateral model.

[0017] According to a third aspect of an embodiment of the present disclosure, a communication device is provided, which includes:

[0018] a transceiver module configured to send first information to a terminal, the first information being used to instruct the terminal to generate second information based on a first AI / ML model;

[0019] the transceiver module is further configured to receive the second information sent by the terminal;

[0020] a processing module configured to determine, according to the second information, whether the first AI / ML model meets a performance requirement, and / or a reason for a performance change of the first AI / ML model;

[0021] wherein model parameters of the first AI / ML model are same as model parameters of a second AI / ML model, or the first AI / ML model is obtained by updating a second AI / ML model and / or a third AI / ML model by the terminal;

[0022] the second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part in a bilateral model, and the third AI / ML model is a second part in the bilateral model.

[0023] According to a fourth aspect of an embodiment of the present disclosure, a communication device is provided, which includes:

[0024] a transceiver configured to receive the first information sent by the network device;

[0025] a processing module configured to generate, according to the first information, second information based on the first AI / ML model;

[0026] the transceiver is configured to send the second information to the network device, the second information being used by the network device to determine whether the first AI / ML model meets a performance requirement and / or a reason for a performance change of the first AI / ML model;

[0027] wherein model parameters of the first AI / ML model are same as model parameters of a second AI / ML model, or the first AI / ML model is obtained by updating a second AI / ML model and / or a third AI / ML model by the terminal;

[0028] the second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0029] According to a fifth aspect of an embodiment of the present disclosure, a communication device is provided, comprising:

[0030] one or more processors;

[0031] The communication device is configured to perform the model performance monitoring method of the first aspect or the second aspect.

[0032] According to a sixth aspect of an embodiment of the present disclosure, a communication system is provided, comprising a network device and a terminal, the network device being configured to send first information to the terminal, the first information being used to instruct the terminal to generate second information based on a first AI / ML model;

[0033] the terminal is configured to receive the first information sent by the network device, and generate the second information based on the first AI / ML model according to the first information;

[0034] the network device is configured to receive the second information sent by the terminal, and determine whether the first AI / ML model meets a performance requirement and / or a reason for a performance change of the first AI / ML model according to the second information;

[0035] The model parameters of the first AI / ML model are the same as the model parameters of the second AI / ML model, or the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model by the terminal.

[0036] The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0037] According to a seventh aspect of the embodiments of the present disclosure, a storage medium is provided, which stores instructions, when the instructions are executed on a communication device, causing the communication device to perform the model performance monitoring method in the first aspect or the second aspect.

[0038] According to an eighth aspect of the embodiments of the present disclosure, a computer program product is provided, which includes a computer program and / or instructions, and when the computer program and / or the instructions are executed by a communication device, the model performance monitoring method in the first aspect or the second aspect is implemented.

[0039] In the above embodiments, the network device can detect the performance of the first AI / ML model used on the terminal side by receiving the second information generated by the terminal based on the first AI / ML model, and accurately determine the cause of the change of the performance of the first AI / ML model, which can effectively ensure that the model deployed on the terminal side in the bilateral model has better performance, and ensure the reliability of the communication process. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following describes the drawings required for the embodiments, and the following drawings are only some embodiments of the present disclosure, and do not specifically limit the protection scope of the present disclosure.

[0041] FIG. 1A is one exemplary schematic diagram of an architecture of a communication system according to an embodiment of the present disclosure.

[0042] FIG. 1B is a schematic diagram illustrating implementation of CSI compression feedback and recovery based on a bilateral AI / ML model according to an embodiment of the present disclosure.

[0043] FIG. 2A is one exemplary interactive schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0044] FIG. 2B is another exemplary interactive schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0045] FIG. 2C is a third example interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0046] FIG. 2D is a fourth example interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0047] FIG. 2E is a fifth example interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0048] FIG. 2F is a sixth example interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0049] FIG. 3A is a first example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0050] FIG. 3B is a second example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0051] FIG. 3C is a third example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0052] FIG. 3D is a fourth example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0053] FIG. 3E is a fifth example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0054] FIG. 3F is a sixth example interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0055] FIG. 3G is a seventh example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0056] FIG. 4A is a first example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0057] FIG. 4B is a second example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0058] FIG. 4C is a third example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0059] FIG. 4D is a fourth example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0060] FIG. 4E is a fifth example flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0061] FIG. 4F is an example flowchart of a sixth example of a model performance monitoring method according to embodiments of the present disclosure.

[0062] FIG. 5 is an example interaction diagram of a model performance monitoring method according to embodiments of the present disclosure.

[0063] FIG. 6 is an example flowchart of a model performance monitoring method according to embodiments of the present disclosure.

[0064] FIG. 7A is an example structural diagram of a terminal according to embodiments of the present disclosure.

[0065] FIG. 7B is an example structural diagram of a network device according to embodiments of the present disclosure.

[0066] FIG. 8A is an example structural diagram of a communication device according to embodiments of the present disclosure.

[0067] FIG. 8B is an example structural diagram of a communication device according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0068] Embodiments of the present disclosure provide a model performance monitoring method, device, system, and storage medium.

[0069] In a first aspect, embodiments of the present disclosure provide a model performance monitoring method, performed by a network device, the method comprising:

[0070] sending first information to a terminal, the first information being used to instruct the terminal to generate second information based on a first AI / ML model;

[0071] receiving the second information sent by the terminal;

[0072] determining, according to the second information, whether the first AI / ML model meets a performance requirement, and / or a cause of a change in performance of the first AI / ML model;

[0073] wherein model parameters of the first AI / ML model are the same as model parameters of a second AI / ML model, or the first AI / ML model is obtained by updating a second AI / ML model and / or a third AI / ML model by the terminal;

[0074] the second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0075] In the above embodiments, the network device can detect the performance of the first AI / ML model used on the terminal side by receiving the second information generated by the terminal based on the first AI / ML model, and accurately determine the cause of the performance change of the first AI / ML model, which can effectively ensure that the model deployed on the terminal side in the bilateral model has better performance and ensure the reliability of the communication process.

[0076] In some embodiments of the first aspect, in some embodiments, the second information includes one or more of the following:

[0077] first quantization information, the first quantization information being obtained by the terminal inputting first channel state information (CSI) into the first AI / ML model, the first information including the first CSI, or the first CSI being predefined;

[0078] second CSI, the second CSI being measured by the terminal;

[0079] second quantization information, the second quantization information being obtained by the terminal inputting the second CSI into the first AI / ML model;

[0080] third CSI, the third CSI being updated by the terminal for the second AI / ML model;

[0081] third quantization information, the third quantization information being obtained by the terminal inputting the third CSI into the first AI / ML model;

[0082] fourth quantization information, the fourth quantization information being obtained by the terminal inputting the third CSI into the second AI / ML model;

[0083] fifth quantization information, the fifth quantization information being obtained by the terminal inputting the first CSI into the second AI / ML model;

[0084] wherein the second AI / ML model has the same quantization method as the first AI / ML model.

[0085] In the above embodiments, the terminal can send different second information to the network device, which can enable the network device to judge the performance of the first AI / ML model in different dimensions, and more accurately determine the cause of the performance change of the first AI / ML model.

[0086] In some embodiments of the first aspect, in some embodiments, the method further includes:

[0087] sending a first signal to the terminal, the first signal being used by the terminal to measure the second CSI.

[0088] In the above embodiments, the network device can send the first signal to the terminal to enable the terminal to accurately measure the actual CSI information.

[0089] In some embodiments of the first aspect, the second information comprises the first quantization information.

[0090] The determining whether the first AI / ML model meets the performance requirement according to the second information comprises:

[0091] inputting the first CSI into the second AI / ML model to obtain sixth quantization information;

[0092] determining whether the first AI / ML model meets the first performance requirement according to the first quantization information and the sixth quantization information.

[0093] In the above embodiments, the terminal can send the first quantization information to the network device, and the network device can input the first CSI into the second AI / ML model to obtain the sixth quantization information, and by comparing the first quantization information with the sixth quantization information, the terminal can accurately determine whether the first AI / ML model meets the performance requirement.

[0094] In some embodiments of the first aspect, the method further comprises:

[0095] sending the sixth quantization information to the terminal, the sixth quantization information being used by the terminal to determine whether the first AI / ML model meets the first performance requirement.

[0096] In the above embodiments, the network device can send the sixth quantization information determined by it to the terminal, so that the terminal can also monitor the performance of its own first AI / ML model.

[0097] In some embodiments of the first aspect, the second information comprises the second quantization information and the second CSI.

[0098] The determining whether the first AI / ML model meets the performance requirement according to the second information comprises:

[0099] inputting the second quantization information into the third AI / ML model to obtain fourth CSI;

[0100] determining whether the first AI / ML model meets the second performance requirement according to the second CSI and the fourth CSI.

[0101] In the above embodiments, the terminal can determine whether the first AI / ML model meets the performance requirement by sending the second quantization information determined based on the measured CSI and the first quantization information to the network device, inputting the second quantization information into the third AI / ML model by the network device to restore the fourth CSI, and comparing the second CSI and the fourth CSI.

[0102] In some embodiments of the first aspect, the method further comprises:

[0103] sending the fourth CSI to the terminal, the fourth CSI being used by the terminal to determine whether the first AI / ML model meets the second performance requirement.

[0104] In the above embodiments, the network device can send the fourth CSI determined by it to the terminal, so that the terminal can also monitor the performance of the first AI / ML model of itself.

[0105] In some embodiments of the first aspect, the sending the first signal to the terminal comprises:

[0106] determining that the first AI / ML model meets the first performance requirement, and sending the first signal to the terminal.

[0107] In the above embodiments, the first signal can be sent to the terminal only when it is determined that the first AI / ML model meets the first performance requirement, which can effectively reduce the resource overhead.

[0108] In some embodiments of the first aspect, the second information comprises the first quantization information and the fifth quantization information.

[0109] The determining whether the first AI / ML model meets the performance requirement according to the second information comprises:

[0110] inputting the first quantization information into the third AI / ML model to obtain a fifth CSI;

[0111] inputting the fifth quantization information into the third AI / ML model to obtain a sixth CSI;

[0112] determining whether the first AI / ML model meets a third performance requirement according to the first CSI, the fifth CSI, and the sixth CSI.

[0113] In the above embodiments, the terminal can send the fifth quantization information and the first quantization information to the network device, and the network device can input the first quantization information and the fifth quantization information into the third AI / ML model to recover the fifth CSI and the sixth CSI, so that the network device can accurately determine whether the first AI / ML model meets the performance requirement based on the first CSI, the fifth CSI, and the sixth CSI.

[0114] In some embodiments of the first aspect, the method further includes:

[0115] sending the fifth CSI and the sixth CSI to the terminal, where the fifth CSI and the sixth CSI are used by the terminal to determine whether the first AI / ML model meets the third performance requirement.

[0116] In the above embodiments, the network device can send the fifth CSI and the sixth CSI determined by the network device to the terminal, so that the terminal can also monitor the performance of the first AI / ML model of the terminal.

[0117] In some embodiments of the first aspect, the second information includes the third CSI, the fifth quantization information, and the third quantization information.

[0118] The determining whether the first AI / ML model meets the performance requirement according to the second information includes:

[0119] inputting the third quantization information into the third AI / ML model to obtain a seventh CSI;

[0120] inputting the fifth quantization information into the third AI / ML model to obtain a sixth CSI;

[0121] determining whether the first AI / ML model meets a fourth performance requirement according to the first CSI, the third CSI, the seventh CSI, and the sixth CSI.

[0122] In the above embodiments, the terminal can send the third CSI, the fifth quantization information, and the third quantization information to the network device, and the network device can input the third quantization information and the fifth quantization information into the third AI / ML model to recover the seventh CSI and the sixth CSI, so that the network device can accurately determine whether the first AI / ML model meets the performance requirement based on the first CSI, the third CSI, the seventh CSI, and the sixth CSI.

[0123] In some embodiments of the first aspect, the method further includes:

[0124] send the seventh CSI and the sixth CSI to the terminal, the seventh CSI and the sixth CSI being used by the terminal to determine whether the first AI / ML model meets a fourth performance requirement.

[0125] In the above embodiments, the network device can send the seventh CSI and the fourth CSI determined by it to the terminal, so that the terminal can also monitor the performance of its own first AI / ML model.

[0126] In some embodiments combined with the first aspect, in some embodiments, the second information includes the second CSI, the third CSI, the second quantization information, and the third quantization information.

[0127] The determining, according to the second information, whether the first AI / ML model meets a performance requirement includes:

[0128] inputting the second quantization information into the third AI / ML model to obtain a fourth CSI;

[0129] inputting the third quantization information into the third AI / ML model to obtain a seventh CSI;

[0130] determining whether the first AI / ML model meets a fifth performance requirement according to the second CSI, the third CSI, the fourth CSI, and the seventh CSI.

[0131] In the above embodiments, the terminal can send the second CSI, the third CSI, the second quantization information, and the third quantization information to the network device, and the network device can input the second quantization information and the third quantization information into the third AI / ML model to restore the fourth CSI and the seventh CSI, so that the network device can accurately determine whether the first AI / ML model meets a performance requirement based on the second CSI, the third CSI, the fourth CSI, and the seventh CSI.

[0132] In some embodiments combined with the first aspect, in some embodiments, the method further includes:

[0133] send the fourth CSI and the seventh CSI to the terminal, the fourth CSI and the seventh CSI being used by the terminal to determine whether the first AI / ML model meets a fifth performance requirement.

[0134] In the above embodiments, the network device can send the seventh CSI and the fourth CSI determined by it to the terminal, so that the terminal can also monitor the performance of its own first AI / ML model.

[0135] In some embodiments of the first aspect, the number of terminals is greater than or equal to one, the second information includes the first quantized information, the first information sent to each of the terminals includes the same first CSI, or the first CSI predefined by each of the terminals is the same.

[0136] The determining, according to the second information, whether the first AI / ML model meets the performance requirement and / or the cause of the change in the performance of the first AI / ML model includes:

[0137] Determining distribution information of the first quantized information according to the first quantized information sent by the plurality of terminals.

[0138] According to the distribution information, determining a first AI / ML model that does not meet a sixth performance requirement from the plurality of first AI / ML models corresponding to the plurality of terminals. In the above embodiment, the terminal can generate the respective first quantized information based on the same first CSI and send it to the network device, so that the network device can perform multi-terminal joint monitoring on the plurality of terminals based on the first quantized information sent by the plurality of terminals, which can effectively ensure the reliability of the AI / ML model used by the terminal.

[0139] In some embodiments of the first aspect, the determining, according to the second information, the cause of the change in the performance of the first AI / ML model includes one or more of the following:

[0140] Determining that the model parameters of the first AI / ML model and the second AI / ML model are the same, and that the first AI / ML model does not meet a first performance requirement, and determining that the change in the performance of the first AI / ML model is caused by model deployment;

[0141] Determining that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model by the terminal, and that the first AI / ML model does not meet a first performance requirement, and determining that the change in the performance of the first AI / ML model is caused by model update;

[0142] Determining that the first AI / ML model does not meet a second performance requirement or a fifth performance requirement, and determining that the change in the performance of the first AI / ML model is caused by channel information change;

[0143] Determining that the first AI / ML model does not meet a third performance requirement or a fourth performance requirement, and determining that the change in the performance of the first AI / ML model is caused by model update;

[0144] Determining that the first AI / ML model does not meet a sixth performance requirement, and determining that the change in the performance of the first AI / ML model is caused by model deployment.

[0145] In the above embodiments, the cause of the change in the performance of the AI / ML model can be accurately determined based on the satisfaction of the performance requirement of the first AI / ML model of the terminal.

[0146] In some embodiments of the first aspect, the method further comprises:

[0147] sending third information to the terminal, the third information comprising the second AI / ML model or the model parameters of the second AI / ML model, and / or the third AI / ML model or the model parameters of the third AI / ML model.

[0148] In some embodiments of the first aspect, the third information comprises a plurality of the second AI / ML models or the model parameters of the plurality of second AI / ML models, and / or a plurality of the third AI / ML models or the model parameters of the plurality of third AI / ML models.

[0149] In the above embodiments, the network device can indicate a plurality of AI / ML models or model parameters through the third information, which can effectively improve the efficiency of AI / ML model configuration.

[0150] In some embodiments of the first aspect, the third information comprises a plurality of the model parameters of the second AI / ML models, and / or a plurality of the model parameters of the third AI / ML models.

[0151] The third information is further used to indicate the second AI / ML model to which the plurality of the model parameters of the second AI / ML models correspond respectively, and / or the third AI / ML model to which the plurality of the model parameters of the third AI / ML models correspond respectively.

[0152] In the above embodiments, when the network device sends the model parameters of a plurality of AI / ML models, the terminal can be ensured to be able to configure the AL / ML model, and the reliability of AI / ML model deployment and use is ensured.

[0153] In some embodiments of the first aspect, the terminal is deployed with a plurality of the first AI / ML models, and the method further comprises:

[0154] receiving fourth information, the fourth information being used to indicate whether the model parameters of each of the plurality of the first AI / ML models are obtained by updating.

[0155] In the above embodiments, the network device can be effectively informed of which AI / ML model is updated by the terminal, and the AI / ML model can be more effectively monitored.

[0156] In a second aspect, the embodiments of the present disclosure provide a model performance monitoring method, executed by a terminal, comprising:

[0157] receiving first information sent by a network device;

[0158] generating second information based on a first AI / ML model according to the first information;

[0159] sending the second information to the network device, the second information being used by the network device to determine whether the first AI / ML model meets a performance requirement, and / or a reason for a performance change of the first AI / ML model;

[0160] wherein model parameters of the first AI / ML model are same as model parameters of a second AI / ML model, or the first AI / ML model is obtained by updating the second AI / ML model and / or a third AI / ML model by the terminal;

[0161] The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0162] In combination with some embodiments of the second aspect, in some embodiments, the second information includes one or more of the following:

[0163] first quantization information, the first quantization information being obtained by inputting first channel state information (CSI) into the first AI / ML model by the terminal, the first information including the first CSI, or the first CSI being predefined;

[0164] second CSI, the second CSI being measured by the terminal;

[0165] second quantization information, the second quantization information being obtained by inputting the second CSI into the first AI / ML model by the terminal;

[0166] third CSI, the third CSI being CSI used for updating the first AI / ML model;

[0167] third quantization information, the third quantization information being obtained by inputting the third CSI into the first AI / ML model by the terminal;

[0168] a fourth quantization information, the fourth quantization information being obtained by the terminal by inputting the third CSI into the second AI / ML model;

[0169] a fifth quantization information, the fifth quantization information being obtained by the terminal by inputting the first CSI into the second AI / ML model;

[0170] wherein the first AI / ML model and the second AI / ML model are quantized by the same method.

[0171] With reference to some embodiments of the second aspect, in some embodiments, the method further comprises:

[0172] receiving the first signal sent by the network device;

[0173] determining the second CSI according to the first signal.

[0174] With reference to some embodiments of the second aspect, in some embodiments, the second information comprises the first quantization information;

[0175] The method further comprises:

[0176] receiving sixth quantization information sent by the network device, the sixth quantization information being obtained by the network device by inputting the first CSI into the second AI / ML model;

[0177] determining whether the first AI / ML model meets the first performance requirement according to the first quantization information and the sixth quantization information.

[0178] With reference to some embodiments of the second aspect, in some embodiments, the second information comprises the second quantization information and the second CSI;

[0179] The method comprises:

[0180] receiving fourth CSI sent by the network device, the fourth CSI being obtained by the network device by inputting the second quantization information into the third AI / ML model;

[0181] determining whether the first AI / ML model meets the second performance requirement according to the second CSI and the fourth CSI.

[0182] With reference to some embodiments of the second aspect, in some embodiments, the second information comprises the first quantization information and the fifth quantization information;

[0183] The method comprises:

[0184] receive a fifth CSI and a sixth CSI sent by the network device, the fifth CSI being obtained by the network device from inputting the first quantized information into the third AI / ML model, and the sixth CSI being obtained by the network device from inputting the fifth quantized information into the third AI / ML model;

[0185] determine whether the first AI / ML model meets a third performance requirement according to the first CSI, the fifth CSI, and the sixth CSI.

[0186] In some embodiments of the second aspect, in some embodiments, the second information includes the third CSI, the fifth quantized information, and the third quantized information.

[0187] The method includes:

[0188] receive a seventh CSI and a sixth CSI sent by the network device, the seventh CSI being obtained by the network device from inputting the third quantized information into the third AI / ML model, and the sixth CSI being obtained by the network device from inputting the fifth quantized information into the third AI / ML model;

[0189] determine whether the first AI / ML model meets a fourth performance requirement according to the first CSI, the third CSI, the seventh CSI, and the sixth CSI.

[0190] In some embodiments of the second aspect, in some embodiments, the second information includes the second CSI, the third CSI, the second quantized information, and the third quantized information.

[0191] The method includes:

[0192] receive a fourth CSI and a seventh CSI sent by the network device, the fourth CSI being obtained by the network device from inputting the second quantized information into the third AI / ML model, and the seventh CSI being obtained by the network device from inputting the third quantized information into the third AI / ML model;

[0193] determine whether the first AI / ML model meets a fifth performance requirement according to the second CSI, the third CSI, the fourth CSI, and the seventh CSI.

[0194] In some embodiments of the second aspect, in some embodiments, the method further includes one or more of the following:

[0195] determining that the first AI / ML model and the second AI / ML model have the same model parameters, and that the first AI / ML model does not meet the first performance requirement, determining that the first AI / ML model causes the performance change due to model deployment;

[0196] determining that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model by the terminal, and that the first AI / ML model does not meet the first performance requirement, determining that the first AI / ML model causes the performance change due to model update;

[0197] determining that the first AI / ML model does not meet the second performance requirement or the fifth performance requirement, determining that the first AI / ML model causes the performance change due to channel information change;

[0198] determining that the first AI / ML model does not meet the third performance requirement or the fourth performance requirement, determining that the first AI / ML model causes the performance change due to model update.

[0199] In some embodiments of the second aspect, the method further comprises:

[0200] receiving third information sent by the network device, the third information comprising the second AI / ML model or model parameters of the second AI / ML model, and / or the third AI / ML model or model parameters of the third AI / ML model.

[0201] In some embodiments of the second aspect, the third information comprises a plurality of the second AI / ML models or model parameters of the plurality of second AI / ML models, and / or a plurality of the third AI / ML models or model parameters of the plurality of third AI / ML models.

[0202] In some embodiments of the second aspect, the third information comprises a plurality of model parameters of the second AI / ML models, and / or a plurality of model parameters of the third AI / ML models.

[0203] The third information is further used to indicate the second AI / ML models corresponding to the plurality of model parameters of the second AI / ML models respectively, and / or the third AI / ML models corresponding to the plurality of model parameters of the third AI / ML models respectively.

[0204] In some embodiments of the second aspect, the terminal is deployed with a plurality of the first AI / ML models, and the method further comprises:

[0205] The fourth information is used to indicate whether the parameter of each of the first AI / ML models is obtained by updating.

[0206] In a third aspect, the embodiments of the present disclosure provide a communication device, comprising:

[0207] The transceiver is configured to send first information to the terminal, the first information being used to indicate that the terminal generates second information based on a first AI / ML model.

[0208] The transceiver is further configured to receive the second information sent by the terminal.

[0209] The processing module is configured to determine whether the first AI / ML model meets a performance requirement and / or a reason for performance change of the first AI / ML model according to the second information.

[0210] The model parameters of the first AI / ML model are the same as the model parameters of a second AI / ML model, or the first AI / ML model is obtained by updating the second AI / ML model and / or a third AI / ML model by the terminal.

[0211] The second AI / ML model and the third AI / ML model are AI / ML models trained by a network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0212] In a fourth aspect, the embodiments of the present disclosure provide a communication device, comprising:

[0213] The transceiver is configured to receive first information sent by a network device.

[0214] The processing module is configured to generate second information based on a first AI / ML model according to the first information.

[0215] The transceiver is configured to send the second information to the network device, the second information being used for the network device to determine whether the first AI / ML model meets a performance requirement and / or a reason for performance change of the first AI / ML model.

[0216] The model parameters of the first AI / ML model are the same as the model parameters of a second AI / ML model, or the first AI / ML model is obtained by updating the second AI / ML model and / or a third AI / ML model by the terminal.

[0217] The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0218] In a fifth aspect, the embodiments of the present disclosure provide a communication device, comprising:

[0219] one or more processors;

[0220] The communication device is configured to perform the communication method in the first aspect or the second aspect.

[0221] In a sixth aspect, the embodiments of the present disclosure provide a communication system, comprising: a terminal, a network device; wherein the terminal is configured to perform the method described in the optional implementation manner of the second aspect, and the network device is configured to perform the method described in the optional implementation manner of the first aspect.

[0222] In a seventh aspect, the embodiments of the present disclosure provide a storage medium, which stores instructions, and when the instructions run on a communication device, the communication device performs the method described in the optional implementation manner of the first aspect and the second aspect.

[0223] In an eighth aspect, the embodiments of the present disclosure provide a computer program product, comprising a computer program and / or instructions, and when the computer program and / or instructions are executed by a communication device, the communication device performs the method described in the optional implementation manner of the first aspect and the second aspect.

[0224] In a ninth aspect, the embodiments of the present disclosure provide a computer program, which, when running on a computer, causes the computer to perform the method described in the optional implementation manner of the first aspect and the second aspect.

[0225] In a tenth aspect, the embodiments of the present disclosure provide a chip or chip system. The chip or chip system comprises processing circuitry configured to perform the method described in the optional implementation manner of the first aspect and the second aspect.

[0226] It can be understood that the terminal, the network device, the communication system, the storage medium, the program product, the computer program, the chip or the chip system are all used to perform the method proposed in the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved are referred to the beneficial effects in the corresponding method, which will not be described here.

[0227] The embodiments of the present disclosure provide a model performance monitoring method, a communication device, a communication system and a storage medium. In some embodiments, the terms of the communication method, the information processing method and the model performance monitoring method can be replaced with each other, the terms of the communication device, the information processing device and the model performance monitoring device can be replaced with each other, and the terms of the information processing system and the communication system can be replaced with each other.

[0228] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing part of the steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation manners in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, part or all of the steps of different embodiments can be combined arbitrarily, an embodiment can be combined with the optional implementation manners of other embodiments.

[0229] In each embodiment of the present disclosure, the terms and / or descriptions between the embodiments are consistent if there is no special description and logical conflict, and can be referred to each other, and the technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationship.

[0230] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and not as a limitation on the present disclosure.

[0231] In the embodiments of the present disclosure, unless otherwise specified, the elements expressed in singular form, such as "one", "a", "the", "above", "said", "preceding", "this" and the like, can represent "one and only one", or "one or more", "at least one" and the like. For example, in the case of using articles such as "a", "an", "the" and the like in English, the noun after the article can be understood as singular expression, or can be understood as plural expression.

[0232] In the embodiments of the present disclosure, "a plurality of" means two or more.

[0233] In some embodiments, the terms "at least one of", "one or more", "a plurality of", "multiple" and the like can be replaced with each other.

[0234] In some embodiments, the description of "at least one of A, B", "A and / or B", "in a case A, in another case B", "in response to a case A, in response to a case B", and the like, can include the following technical solutions according to the case: in some embodiments, A (A is executed regardless of B); in some embodiments, B (B is executed regardless of A); in some embodiments, A and B are selectively executed (A and B are selected from A and B); in some embodiments, A and B (A and B are executed). When there are more branches such as A, B, C, and the like, the above is similar.

[0235] In some embodiments, the description of "A or B" and the like can include the following technical solutions according to the case: in some embodiments, A (A is executed regardless of B); in some embodiments, B (B is executed regardless of A); in some embodiments, A and B are selectively executed (A and B are selected from A and B). When there are more branches such as A, B, C, and the like, the above is similar.

[0236] The prefix words "first", "second", and the like in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute a limitation on the position, order, priority, quantity, or content of the description objects. The description of the description objects should refer to the description in the context of the claims or embodiments, and should not be limited by the prefix words. For example, the description object is "field", and the ordinal words before "field" in "first field" and "second field" do not limit the position or order between "fields". "First" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of "first field" and "second field". For another example, the description object is "level", and the ordinal words before "level" in "first level" and "second level" do not limit the priority between "levels". For another example, the quantity of the description object is not limited by the ordinal words, and can be one or more. For example, "first device", where the quantity of "device" can be one or more. In addition, the objects modified by different prefix words can be the same or different, for example, the description object is "device", and "first device" and "second device" can be the same device or different devices, and their types can be the same or different; for another example, the description object is "information", and "first information" and "second information" can be the same information or different information, and their contents can be the same or different.

[0237] In some embodiments, "including A", "containing A", "for indicating A", "carrying A" can be interpreted as directly carrying A, or indirectly indicating A.

[0238] In some embodiments, the terms "time / frequency", "time / frequency domain", and the like refer to the time domain and / or the frequency domain.

[0239] In some embodiments, the terms “in response to,” “in response to determining,” “in the event that,” “when,” “if,” “upon,” and the like can be replaced with each other.

[0240] In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” “above,” and the like can be replaced with each other, and the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” “below,” and the like can be replaced with each other.

[0241] In some embodiments, an apparatus and the like can be interpreted as an entity, and can also be interpreted as virtual, and the name thereof is not limited to the name described in the embodiments, and the terms “apparatus,” “equipment,” “device,” “circuit,” “network element,” “node,” “function,” “unit,” “section,” “system,” “network,” “chip,” “chip system,” “entity,” “subject,” and the like can be replaced with each other.

[0242] In some embodiments, “network” can be interpreted as an apparatus (for example, an access network device, a core network device, and the like) included in the network.

[0243] In some embodiments, the terms “access network device (AN device),” “radio access network device (RAN device),” “base station (BS),” “radio base station,” “fixed station,” “node,” “access point,” “transmission point (TP),” “reception point (RP),” “transmission / reception point (TRP),” “panel,” “antenna panel,” “antenna array,” “cell,” “macro cell,” “small cell,” “femto cell,” “pico cell,” “sector,” “cell group,” “serving cell,” “carrier,” “component carrier,” “bandwidth part (BWP),” and the like can be used interchangeably.

[0244] In some embodiments, the terms "terminal," "terminal device," "user equipment (UE)," "user terminal," "mobile station (MS)," "mobile terminal (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile terminal," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," and so on can be replaced with each other.

[0245] In some embodiments, the access network device, the core network device, or the network device can be replaced with a terminal. For example, the embodiments of the present disclosure can also be applied to a structure in which communication between the access network device, the core network device, or the network device and the terminal is replaced with communication between a plurality of terminals (e.g., device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, the terminal can also be configured to have all or part of the functions of the access network device. In addition, the terms "uplink," "downlink," and the like can also be replaced with terms corresponding to the inter-terminal communication (e.g., "side"). For example, the uplink channel, the downlink channel, and the like can be replaced with the side channel, and the uplink, the downlink, and the like can be replaced with the sidelink.

[0246] In some embodiments, the terminal can be replaced with the access network device, the core network device, or the network device. In this case, the access network device, the core network device, or the network device can also be configured to have all or part of the functions of the terminal.

[0247] In some embodiments, obtaining data, information, etc. can comply with laws and regulations of the country where the location is.

[0248] In some embodiments, data, information, etc. can be obtained after obtaining consent of the user.

[0249] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0250] FIG. 1A is a schematic diagram of an architecture of a communication system according to an embodiment of the present disclosure. As shown in FIG. 1A, the communication system 100 includes a terminal 101 and a network device 102. In some embodiments, the network device 102 includes at least one of an access network device and a core network device.

[0251] In some embodiments, the terminal 101 includes at least one of a mobile phone, a wearable device, an Internet of Things device, a car with communication function, a smart car, a Pad, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, a wireless terminal device in smart home, etc., but is not limited thereto.

[0252] In some embodiments, the access network device is, for example, a node or device that accesses a terminal to a wireless network, and the access network device can include at least one of an evolved NodeB (eNB) in a 5G communication system, a next generation eNB (ng-eNB), a next generation NodeB (gNB), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an open base station (Open RAN), a cloud base station (Cloud RAN), a base station in other communication systems, an access node in a Wi-Fi system, but is not limited thereto.

[0253] In some embodiments, the technical solutions of the present disclosure can be applied to an Open RAN architecture, at this time, the interfaces between or within the access network devices involved in the embodiments of the present disclosure can become internal interfaces of the Open RAN, and the processes and information interactions between these internal interfaces can be realized through software or programs.

[0254] In some embodiments, the access network device can be composed of a central unit (CU) and a distributed unit (DU), wherein the CU can also be referred to as a control unit (control unit). The CU-DU structure can split the protocol layers of the access network device, and part of the functions of the protocol layers are controlled by the CU, and the remaining part or all of the functions of the protocol layers are distributed in the DU and controlled by the CU, but the present disclosure is not limited thereto.

[0255] In some embodiments, the core network device can be one device including a first network element, a second network element, etc., or can be multiple devices or device groups, respectively including all or part of the first network element, the second network element, etc. The network element can be virtual or physical. The core network includes, for example, at least one of an evolved packet core (EPC), a 5G core network (5GCN), and a next generation core (NGC).

[0256] It can be understood that the communication system described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and does not constitute a limitation on the technical solutions proposed by the embodiments of the present disclosure. Those skilled in the art can know that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions proposed by the embodiments of the present disclosure are also applicable to similar technical problems.

[0257] The following embodiments of the present disclosure can be applied to the communication system 100 shown in FIG. 1A or part of the subject, but are not limited thereto. The subjects shown in FIG. 1A are exemplary, and the communication system can include all or part of the subjects in FIG. 1A, or other subjects other than FIG. 1A. The number and form of each subject is arbitrary, each subject can be physical or virtual, the connection relationship between each subject is exemplary, each subject can not be connected or can be connected, the connection can be in any way, can be direct connection or indirect connection, can be wired connection or wireless connection.

[0258] Embodiments of the present disclosure can be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G new radio (NR), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New radio access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (Bluetooth (registered trademark)), Public Land Mobile Network (PLMN) network, Device-to-Device (D2D) system, Machine to Machine (M2M) system, Internet of Things (IoT) system, Vehicle-to-Everything (V2X), system using other communication methods, next-generation system expanded based thereon, and the like. Further, a plurality of systems can be applied in combination (for example, combination of LTE or LTE-A and 5G, and the like).

[0259] FIG. IB is a schematic diagram illustrating implementation of CSI compression feedback and recovery based on a bilateral AI / ML model according to an embodiment of the present disclosure. As shown in FIG. IB, the UE side compresses the downlink channel information H through a CSI generation partial model (for example, referred to as an encoder) and sends it to the device gNB of the NW side through quantization into a binary bit stream, and the gNB side recovers the recovered downlink channel H' similar to the original downlink information through a CSI recovery partial model (defined as a decoder).

[0260] In some embodiments, the CSI generation partial model can be sent by the gNB to the UE, or can be further trained by the UE based on the model sent by the gNB. Optionally, the second AI / ML model involved in some embodiments described below can be the CSI generation partial model, which can be the first part of the bilateral model. Optionally, the third AI / ML model involved in the embodiments described below can be the CSI recovery partial model, which can be the second part of the bilateral model.

[0261] In some implementations, since the encoder and the decoder described above are respectively deployed on the UE side and the NW side, the training of the model needs bilateral cooperation to complete the training of the model, or the single-end trained model sends part of the model to the opposite end for model inference, or re-trains a new model based on the received transfer model. If the UE and each NW vendor independently perform bilateral model training, the complexity of bilateral model training will be greatly increased.

[0262] In order to alleviate the complexity of bilateral model training cooperation for CSI compression feedback, the following optional implementations for bilateral AI / ML training cooperation are proposed to obtain a bilateral AI / ML model:

[0263] Option 1, standardize the model structure and model parameters of the reference model.

[0264] Option 2, standardize the data set.

[0265] Option 3, standardize the model structure, and exchange the model parameters through the NW side and the UE side.

[0266] Option 4, standardize the format of the data, and exchange the data set through the NW side and the UE side.

[0267] Option 5, standardize the model format, and exchange the reference model through the NW side and the UE side.

[0268] In some embodiments, for different uses and / or transferred contents, the following optional schemes exist:

[0269] Optionally, for the above optional solution three and optional solution five, the UE side can retrain a new encoder for the received encoder parameters or model. Alternatively, the UE side retrain a new encoder for the received decoder parameters or model. Alternatively, the UE side retrain a new encoder for the received encoder and decoder parameters or model.

[0270] Optionally, for the above optional solution three, the UE can directly use the received encoder parameters sent by the NW through the air interface for model inference.

[0271] Optionally, for the above optional solution five, the UE can directly use the received encoder parameters sent by the NW through the air interface for model inference.

[0272] Optionally, for the above optional solution four, the NW side sends the target CSI and CSI feedback information to the UE side. Alternatively, the NW side sends the recovered target CSI and CSI feedback information to the UE side. Optionally, the NW side sends the target CSI, CSI feedback information and recovered target CSI to the UE side.

[0273] Optionally, for the above optional solution one, the UE side and the NW side can respectively train the updated encoder and decoder models of the UE side and the NW side according to the collected data set (different from the data set used to train the standardized model).

[0274] In some embodiments, in order to train the model at the UE side or verify the performance of the model performance, the NW side will also pass the data set or related information for data collection to the UE side and the target performance.

[0275] In some embodiments, for some of the above optional solutions, because of the transmission error of the model or parameters, the deterioration of the retrained encoder model, and other factors, it is possible that the encoder model deployed on the UE cannot meet the performance requirements. Therefore, it is necessary to study and give the corresponding performance monitoring mechanism to identify the cause of performance deterioration, so as to ensure that the deployed bilateral model obtains better performance. Due to the change of the channel environment where the UE is located or the difference of the network parameter configuration, the channel data of the UE side and the current model do not match, which can also cause the model performance to deteriorate. If only the aforementioned NW side or UE side monitoring method is used, the monitoring result may not be reliable. How to reduce the signaling transmission overhead or ensure the reliability of the bilateral model monitoring result is a problem to be solved.

[0276] In some embodiments, for the model performance monitoring on the NW side, the target CSI reported on the UE side can be implemented based on the eType II codebook or a higher-precision eType II codebook.

[0277] In some embodiments, for the model performance monitoring on the NW side, the model performance can be monitored based on the estimated intermediate KPI, for example, based on the decoder output information deployed on the UE side. Alternatively, the model monitoring can also be performed based on the monitoring results in addition to the intermediate KPI. Alternatively, the model monitoring can also be performed based on the precoded reference signal such as the precoded CSI-RS or DMRS sent by the NW side, and the precoding is obtained based on the decoder output information on the NW side. Alternatively, the monitoring is performed based on the decoder output information on the NW side, and the output information of the decoder is indicated to the UE by the NW through the eType II codebook or the high-precision eType II codebook.

[0278] In some embodiments, the encoder deployed on the UE side can be the same as or different from the encoder deployed on the NW side, or it can be a reference model provided by the NW or a proxy model developed by the UE side.

[0279] However, the above-mentioned embodiments for monitoring the model performance on the NW side or the UE side cannot necessarily obtain accurate monitoring results, because this method cannot verify whether the performance degradation is caused by the change of the channel data or the degradation of the updated partial model. In addition, the transmission of the target CSI or the recovered CSI between the UE and the NW side will consume the uplink or downlink transmission resources.

[0280] To this end, in some embodiments, the UE can use the pre-defined target CSI or the target CSI sent by the NW or the target CSI for training a new model as the encoder input to monitor the cause of the change of the model performance deployed by the UE.

[0281] FIG. 2A is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 2, the present disclosure relates to a model performance monitoring method, and the above-mentioned method comprises:

[0282] In step S2101, the network device sends third information to the terminal.

[0283] In some embodiments, the third information comprises the second AI / ML model, or model parameters of the second AI / ML model. Optionally, the second AI / ML model can be the first part of the bilateral model. Optionally, the second AI / ML model can be the CSI generation part model, i.e., the encoder part, in the bilateral model.

[0284] In some embodiments, the third information can comprise the second AI / ML model and / or the third AI / ML model. Optionally, the third AI / ML model can be the second part of the bilateral model. Optionally, the second AI / ML model can be the CSI recovery part model, i.e., the decoder part, in the bilateral model.

[0285] It can be understood that the network device can convert the first part and / or the second part of the bilateral model into a standardized model format and send it to the terminal through the third information, or standardize the model structure and send the model parameters to the terminal through the third information. For example, the terminal can pre-configure the structure of the model, the terminal can determine the model structure corresponding to the model parameters sent by the network device, and then determine the corresponding model.

[0286] In some embodiments, after receiving the third information, the terminal can directly use the second AI / ML model as the encoder for inference, or update the second AI / ML model and / or the third AI / ML model to obtain an updated encoder for inference. For example, the terminal can use the second AI / ML model as the first AI / ML model used by it, or train the first AI / ML model according to the second AI / ML model and / or the third AI / ML model.

[0287] Optionally, the first AI / ML model can be the CSI generation part model, i.e., the encoder part, in the bilateral model. Optionally, the second AI / ML model and the first AI / ML model have the same quantization method. That is, the quantization method of the encoder used by the network device and the encoder used by the terminal can be the same.

[0288] In some embodiments, the first AI / ML model and / or the second AI / ML model can have the ability to quantize data, and the quantization methods of the two are the same, or the first AI / ML model and / or the second AI / ML model can use the same quantization module to quantize the data output by them.

[0289] It is worth noting that the first AI / ML model involved in the present disclosure can refer to a model used by the terminal. The second AI / ML model can be a model sent by the network device to the terminal, i.e., a model used by the network device. That is, the network device can indicate the encoder or the model parameters of the encoder used by it to the terminal.

[0290] In addition, updating the second AI / ML model and / or the third AI / ML model can refer to the process of training an updated AI / ML model by the terminal based on the second AI / ML model and / or the third AI / ML model. For example, the terminal can train an updated encoder for CSI compression and quantization based on the second AI / ML model indicated by the third information, or train an updated encoder based on the third AI / ML model indicated by the third information, or train an updated encoder based on the combination of the second AI / ML model and the third AI / ML model indicated by the third information.

[0291] In some embodiments, the third information includes a plurality of second AI / ML models or model parameters of the plurality of second AI / ML models, and / or a plurality of third AI / ML models or model parameters of the plurality of third AI / ML models. Optionally, the terminal can be deployed with a plurality of AI / ML models for CSI-related inference, i.e., the terminal can be deployed with a plurality of first AI / ML models. Different second AI / ML models or third AI / ML models can correspond to different bilateral models.

[0292] It should be understood that the model structure of each bilateral model can be the same and can include a CSI generation part model (encoder) and a CSI recovery part model (decoder). Optionally, the model parameters of different bilateral models can be different, and different bilateral models can correspond to different use scenarios.

[0293] Optionally, the number of models used by the terminal (i.e., the first AI / ML model) can be greater than or equal to one. For example, the terminal can use different models for inference according to different scenarios, or can use multiple models for inference at the same time. Optionally, different first AI / ML models can be obtained based on different second AI / ML models. For example, the third information can include 5 reference models or parameters of the reference models (such as model parameters of the second AI / ML model and / or the third AI / ML model), and the terminal can determine 5 different first AI / ML models according to the 5 reference models, and use different first AI / ML models for inference in different scenarios, or use multiple different first AI / ML models for inference at the same time.

[0294] In some embodiments, after receiving the third information, the terminal can directly use all or part of the plurality of second AI / ML models as the models used by the terminal, or update all or part of the plurality of second AI / ML models to obtain the models used by the terminal. For example, the third information can include the model parameters of the four second AI / ML models, and the terminal can use the parameters of the first and third second AI / ML models as the model parameters of two first AI / ML models used by the terminal, and update the second and fourth second AI / ML models to obtain the model parameters of the other two first AI / ML models used by the terminal.

[0295] In some embodiments, the third information includes the model parameters of the plurality of second AI / ML models, and / or the model parameters of the plurality of third AI / ML models; and the third information further indicates the second AI / ML models to which the model parameters of the plurality of second AI / ML models correspond respectively, and / or the AI / ML models to which the model parameters of the plurality of third AI / ML models correspond respectively.

[0296] For example, the third information can include model parameter 1, model parameter 2, model parameter 3, and model parameter 4, which are all first parts (i.e., second AI / ML models) of a bilateral model, for example. The terminal can use four different models (i.e., first AI / ML models) to infer the CSI in four scenarios respectively, and the third information can further indicate that model parameter 1 corresponds to the model used by the terminal in scenario 1, model parameter 2 corresponds to the model used by the terminal in scenario 2, model parameter 3 corresponds to the model used by the terminal in scenario 3, and model parameter 2 corresponds to the model used by the terminal in scenario 3. In turn, the terminal can determine the model parameters of the models used in different scenarios according to different model parameters.

[0297] In some embodiments, the number of terminals can be greater than or equal to one. Optionally, the network device sends the third information to one or more terminals. Optionally, the network device can send the same third information to each terminal, or send different third information to different terminals.

[0298] In some embodiments, the third information can be referred to as "model indication information", "model parameter information", "training indication information", etc., and the name of the embodiment of the present disclosure is not limited.

[0299] In some embodiments, the terminal receives the third information. Optionally, the terminal determines the first AI / ML model according to the third information. Optionally, the terminal determines one or more first AI / ML models according to the third information. Optionally, after determining one or more first AI / ML models according to the third information, the terminal performs step S2102.

[0300] Step S2102, the terminal sends fourth information to the network device.

[0301] In some embodiments, the terminal is deployed with a plurality of first AI / ML models, and the fourth information is used to indicate whether the model parameters of each first AI / ML model in the plurality of first AI / ML models are obtained by updating.

[0302] For example, the third information sent by the network device in step S2101 can include four second AI / ML models (or model parameters), and the terminal can determine the corresponding fourth information after receiving the third information and determining the four first AI / ML models used in four different scenarios. The fourth information may, for example, be a bitmap, which can include four bits. The first bit is used to indicate whether the model used in scenario 1 is obtained by updating the second AI / ML model, the second bit is used to indicate whether the model used in scenario 2 is obtained by updating the second AI / ML model, the third bit is used to indicate whether the model used in scenario 3 is obtained by updating the second AI / ML model, and the fourth bit is used to indicate whether the model used in scenario 4 is obtained by updating the second AI / ML model.

[0303] In some embodiments, step S2102 is optional. For example, whether the terminal updates the model parameters of each second AI / ML model (and / or third AI / ML model) can be default or predefined, or can be indicated by the third information. In this case, the terminal can not need to send the fourth information to the network device.

[0304] In some embodiments, the fourth information can be referred to as "update indication information", "update bitmap", etc., and the name of the embodiment of the present disclosure is not limited.

[0305] Step S2103, the network device sends the first information to the terminal.

[0306] In some embodiments, the network device can send the same or different first information to one or more terminals.

[0307] In some embodiments, the first information can include first CSI. Optionally, the network device can send the same first CSI to a plurality of terminals.

[0308] In some embodiments, the first information is used to instruct the terminal to generate the second information based on the first AI / ML model. Optionally, the first information is used to instruct the terminal to generate the second information based on one or more AI / ML models in the plurality of first AI / ML models.

[0309] In some embodiments, the terminal receives first information sent by the network device. Optionally, after receiving the first information sent by the network device, the terminal generates second information. Optionally, the terminal generates the second information according to the first CSI in the first information.

[0310] In some embodiments, the first information can be used to indicate the first AI / ML model to be monitored. For example, the terminal can be configured with multiple first AI / ML models, and the first information can be used to indicate the first AI / ML model to be detected, so that the terminal uses the first AI / ML model to generate corresponding third information.

[0311] In an example, the second information can include first indication information, the first indication information being used to indicate the first AI / ML model corresponding to the second information, and second indication information being used to indicate that the second information is used to indicate the CSI quantization information to be generated. Optionally, when the CSI quantization information to be generated includes the following first quantization information, the second information can further include the first CSI.

[0312] In some embodiments, the second information can include at least one of the following:

[0313] The first quantization information is obtained by inputting the first channel state information (CSI) into the first AI / ML model by the terminal, the first CSI is predefined, or is sent by the network device to the terminal;

[0314] The second quantization information is obtained by inputting the second CSI into the first AI / ML model by the terminal;

[0315] The third quantization information is obtained by inputting the third CSI into the first AI / ML model by the terminal;

[0316] The fourth quantization information is obtained by inputting the third CSI into the second AI / ML model by the terminal;

[0317] The fifth quantization information is obtained by inputting the first CSI into the second AI / ML model by the terminal;

[0318] The second CSI is measured by the terminal;

[0319] The third CSI is the CSI used to update the first AI / ML model.

[0320] In some embodiments, the terminal generates the second information based on the first AI / ML model according to the first information. Optionally, the terminal generates the second information based on the first AI / ML model according to the first information, including one or more of the following:

[0321] The first CSI is input into the first AI / ML model to obtain first quantization information;

[0322] The second CSI is input into the first AI / ML model to obtain second quantization information;

[0323] The third CSI is input into the first AI / ML model to obtain third quantization information;

[0324] The third CSI is input into the second AI / ML model to obtain fourth quantization information;

[0325] The first CSI is input into the second AI / ML model to obtain fifth quantization information;

[0326] The first signal is measured to obtain the second CSI.

[0327] Optionally, the first CSI can be determined by the network device and sent to the terminal through the first information. For example, the first information includes the first CSI.

[0328] In some embodiments, the first information sent by the network device to the plurality of terminals includes the same first CSI. Alternatively, the first CSI predefined by the plurality of terminals is the same. At this time, the plurality of terminals can generate a plurality of corresponding first quantization information based on the respective first AI / ML model. Optionally, the plurality of first quantization information can be used to determine whether each terminal meets the sixth performance requirement.

[0329] It can be understood that when receiving different first information, the terminal can generate and send second information including different contents to the network device. When the contents included in the second information are different, the second information can be used to determine whether the first AI / ML model meets different performance requirements, or to determine different performance change reasons. For example, when monitoring whether the first AI / ML model meets the performance change caused by model deployment, the network device can send corresponding first information, and the second information can include the first quantization information; when monitoring whether the first AI / ML model meets the performance change caused by channel information change, the network device can send corresponding first information, and the second information can include the second quantization information, and so on.

[0330] In the embodiments of the present disclosure, the second information can include the first quantization information and / or the second quantization information and the second CSI. In other embodiments, the second information can include other different information. For example, in the embodiment shown in FIG. 2B, the second information can include the first quantization information and the fifth quantization information, in the embodiment shown in FIG. 2C, the second information can include the third CSI, the fifth quantization information and the third quantization information, and so on.

[0331] In some embodiments, when the terminal determines that the model parameters of the first AI / ML model and the second AI / ML model are the same, i.e., when the terminal does not update the first AI / ML model, it can determine only one or more of the first quantization information, the second quantization information, and the second CSI. When the terminal determines that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model, it can also determine one or more of the third CSI, the third quantization information, the fourth quantization information, and the fifth quantization information.

[0332] In some embodiments, the terminal can generate and send multiple pieces of second information to the network device, each piece of second information including different information. For example, in steps S2104 and S2110 of this embodiment, the terminal can send not only second information including first quantization information to the network device, but also second information including second CSI and second quantization information. As another example, the terminal can send different second information for different first AI / ML models. For instance, for a first AI / ML model used directly without updates, the terminal can send second information including first quantization information and / or second CSI and second quantization information to the network device; for a first AI / ML model obtained after updates, the terminal can send second information including first quantization information and fifth quantization information to the network device, and so on.

[0333] In some embodiments, the first information may be referred to as "monitoring indication information", "model monitoring indication", etc., and the name is not limited in this disclosure.

[0334] In some embodiments, the second information may be referred to as "monitoring feedback information", "CSI quantification information", etc., and the name is not limited in this disclosure.

[0335] In some embodiments of this disclosure, quantization information such as first quantization information may also be referred to as "compressed CSI", "compressed quantized CSI", etc., and this disclosure does not limit this.

[0336] Step S2104: The terminal sends the first quantization information to the network device.

[0337] In some embodiments, the terminal sends second information, including first quantization information, to the network device.

[0338] In some embodiments, after receiving the first information, the terminal generates and sends the second information, which includes the first quantization information, to the network device.

[0339] In some embodiments, the terminal inputs the first CSI into the first AI / ML model to obtain the first quantized information after receiving the first information. Optionally, the first CSI can be predefined, can be obtained through pre-negotiation between the terminal and the network device, or can be sent by the network device to the terminal through the first information.

[0340] In some embodiments, the network device receives the first quantized information sent by the terminal. Optionally, after receiving the first quantized information sent by the terminal, the network device performs step S2105.

[0341] In step S2105, the network device inputs the first CSI into the second AI / ML model to obtain the sixth quantized information.

[0342] In some embodiments, after determining the sixth quantized information, the network device performs step S2106 or step S2107.

[0343] In step S2106, the network device sends the sixth quantized information to the terminal.

[0344] In some embodiments, step S2106 is optional. That is, the network device can not send the sixth quantized information to the terminal. At this time, only the performance of the first AI / ML model used by the terminal is monitored by the network device.

[0345] In step S2107, the network device and the terminal respectively determine whether the first AI / ML model meets the first performance requirement according to the first quantized information and the sixth quantized information.

[0346] In some embodiments, when step S2106 is omitted, only the network device determines whether the first AI / ML model meets the first performance requirement according to the first quantized information and the sixth quantized information.

[0347] In some embodiments, the network device and / or the terminal can compare the similarity of the first quantized information and the sixth quantized information to determine whether the first performance requirement is met. For example, when the similarity is lower than a preset threshold, it is determined that the first AI / ML model does not meet the first performance requirement.

[0348] In some embodiments, the network device and / or the terminal determine that the model parameters of the first AI / ML model and the second AI / ML model are the same, and the first AI / ML model does not meet the first performance requirement, and determine that the performance change of the first AI / ML model is caused by model deployment.

[0349] In some embodiments, the network device and / or the terminal determine that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model by the terminal, and the first AI / ML model does not meet the first performance requirement, and determine that the performance change of the first AI / ML model is caused by model update.

[0350] In some embodiments, the network device determines that the first AI / ML model meets the first performance requirement, and performs step S2108. That is, the network device can only send the first signal to the terminal when it is determined that the first AI / ML model meets the first performance requirement.

[0351] Step S2108, the network device sends the first signal to the terminal.

[0352] In some embodiments, the first signal can be a downlink pilot signal. Alternatively, the first signal can be a channel state information reference signal (CSI-RS).

[0353] In some embodiments, the first signal is used by the terminal to determine the second CSI.

[0354] In some embodiments, the terminal receives the first signal sent by the network device. Alternatively, after receiving the first signal, the terminal performs step S2109.

[0355] Step S2109, the terminal determines the second CSI according to the first signal.

[0356] In some embodiments, the terminal measures the first signal to obtain the second CSI. In some embodiments, step S2108 and step S2109 can be performed when it is determined that the first AI / ML model meets the first performance requirement, or can be performed before or at the same time as step S2107, and the embodiments of the present disclosure are not limited thereto.

[0357] Step S2110, the terminal sends the second quantization information and the second CSI to the network device.

[0358] In some embodiments, step S2110 can be performed at the same time as step S2107, for example, the terminal can send second information including the first quantization information, the second CSI, and the second quantization information. Alternatively, step S2110 can be performed after step S2109, for example, the terminal can first send second information including the first quantization information, and then send the second quantization information and the second CSI to the network device after determining that the first AI / ML model meets the first performance requirement.

[0359] In some embodiments, the network device receives the second quantization information and the second CSI sent by the terminal. Alternatively, after receiving the second quantization information and the second CSI sent by the terminal, the network device performs step S2111.

[0360] Step S2111, the network device inputs the second quantization information into the third AI / ML model to obtain the fourth CSI.

[0361] In some embodiments, the network device performs step S2112 or step S2113 after obtaining the fourth CSI.

[0362] In step S2112, the network device sends the fourth CSI to the terminal.

[0363] In some embodiments, step S2112 is optional. That is, the network device can not send the fourth CSI to the terminal. In this case, only the performance of the first AI / ML model used by the terminal side is monitored by the network device.

[0364] In step S2113, the network device and the terminal respectively determine whether the first AI / ML model meets the second performance requirement according to the second CSI and the fourth CSI.

[0365] In some embodiments, when step S2112 is omitted, only the network device determines whether the first AI / ML model meets the second performance requirement according to the second CSI and the fourth CSI.

[0366] In some embodiments, the network device and / or the terminal can compare the similarity of the second CSI and the fourth CSI to determine whether the second performance requirement is met. For example, when the similarity is lower than a preset threshold, it is determined that the first AI / ML model does not meet the second performance requirement.

[0367] Alternatively, the network device and / or the terminal determine whether the second performance requirement is met according to the normalized mean squared error (NMMSE) or the square generalized cosine similarity (SGCS) of the second CSI and the fourth CSI.

[0368] In some embodiments, when the network device and / or the terminal determines that the first AI / ML model does not meet the second performance requirement, it is determined that the performance change of the first AI / ML model is caused by the change of the channel information.

[0369] In some embodiments, the names of information and the like are not limited to the names described in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "field", "symbol", "codebook", "codeword", "codepoint", "bit", "data", "program", "chip", and the like can be replaced with each other.

[0370] In some embodiments, terms such as "uplink", "uplink", "physical uplink", and the like can be replaced with each other, terms such as "downlink", "downlink", "physical downlink", and the like can be replaced with each other, terms such as "side", "sidelink", "sidelink communication", "sidelink communication", "direct connection", "direct connection link", "direct connection", "direct connection link communication", and the like can be replaced with each other.

[0371] In some embodiments, terms such as "downlink control information (DCI)", "downlink (DL) assignment", "DL DCI", "uplink (UL) grant", "UL DCI", and the like can be replaced with each other.

[0372] In some embodiments, terms such as "synchronization signal (SS)", "synchronization signal block (SSB)", "reference signal (RS)", "pilot", "pilot signal", and the like can be replaced with each other.

[0373] In some embodiments, terms such as "time", "time point", "time", "time position", and the like can be replaced with each other, and terms such as "duration", "period", "time window", "window", "time", and the like can be replaced with each other.

[0374] In some embodiments, “acquire”, “obtain”, “get”, “receive”, “transmit”, “bidirectionally transmit”, “send and / or receive” can be replaced by each other, which can be interpreted as receiving from other subjects, acquiring from protocols, acquiring from higher layers, obtaining by self-processing, implementing autonomously, and the like.

[0375] In some embodiments, the terms “send”, “transmit”, “report”, “issue”, “transmit”, “bidirectionally transmit”, “send and / or receive” can be replaced by each other.

[0376] In some embodiments, the terms “certain”, “preset”, “preset”, “set”, “indicated”, “certain”, “arbitrary”, “first” and the like can be replaced by each other. “Certain A”, “preset A”, “preset A”, “set A”, “indicated A”, “certain A”, “arbitrary A”, “first A” can be interpreted as A specified in advance in protocols and the like, can be interpreted as A obtained by setting, configuring, or indicating, and the like, can be interpreted as certain A, certain A, arbitrary A, or first A, and the like, but are not limited thereto.

[0377] In some embodiments, determination or judgment can be made by a value represented by 1 bit (0 or 1), or by a true or false value (Boolean value) represented by true or false, or by comparison of numerical values (for example, comparison with a predetermined value), but is not limited thereto.

[0378] In some embodiments, “not expecting to receive” can be interpreted as not receiving on time domain resources and / or frequency domain resources, or can be interpreted as not performing subsequent processing on the data and the like after receiving the data and the like; “not expecting to send” can be interpreted as not sending, or can be interpreted as sending but not expecting the receiving party to respond to the content of the sending.

[0379] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S2101-S2113. For example, step S2102 can be implemented as an independent embodiment, step S2104 can be implemented as an independent embodiment, step S2107 can be implemented as an independent embodiment, step S2110 can be implemented as an independent embodiment, step S2113 can be implemented as an independent embodiment, steps S2104 to S2107 can be implemented as independent embodiments, steps S2110 to S2113 can be implemented as independent embodiments, but are not limited thereto.

[0380] In some embodiments, step S2104 and step S2110 can be exchanged in order or performed at the same time, and step S2107 and step S2113 can be exchanged in order or performed at the same time.

[0381] In some embodiments, steps S2101 to S2106 and steps S2108 to S2113 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0382] In some embodiments, steps S2101 to S2112 and step S2113 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0383] In some embodiments, steps S2101 to S2113 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0384] In some embodiments, other optional implementations can be referred to the description before or after the description of Figure 2A.

[0385] Figure 2B is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in Figure 2B, the embodiment of the present disclosure relates to a model performance monitoring method, and the above method comprises:

[0386] Step S2201, the terminal sends first quantization information and fifth quantization information to the network device.

[0387] In some embodiments, step S2201 can be performed by the terminal after receiving the first information.

[0388] In some embodiments, step S2201 can be performed by the terminal after receiving the first information and determining that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model.

[0389] For example, the terminal can only input the first CSI into the second AI / ML model and the first AI / ML model to obtain the fifth quantization information and the first quantization information respectively, and further send the fifth quantization information and the first quantization information to the network device, in the case of determining that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model.

[0390] In some embodiments, the terminal sends second information comprising the first quantization information and the fifth quantization information to the network device.

[0391] Optional implementations of the second information can be referred to the optional implementations involved in step 2103 of Figure 2A, which will not be described here.

[0392] In some embodiments, the network device receives the second information sent by the terminal. Optionally, the network device receives the first quantized information and the fifth quantized information sent by the terminal. Optionally, after receiving the second information sent by the terminal, the network device performs step S2202. Optionally, after receiving the first quantized information and the fifth quantized information sent by the terminal, the network device performs step S22202.

[0393] Step S2202: The network device inputs the first quantized information and the fifth quantized information into the third AI / ML model respectively to obtain the fifth CSI and the sixth CSI.

[0394] In some embodiments, the network device inputs the first quantized information into the third AI / ML model to obtain the fifth CSI.

[0395] In some embodiments, the network device inputs the fifth quantized information into the third AI / ML model to obtain the sixth CSI.

[0396] In some embodiments, after determining the fifth CSI and the sixth CSI, the network device performs step S2203 or step S2204.

[0397] Step S2203: The network device sends the fifth CSI and the sixth CSI to the terminal.

[0398] In some embodiments, step S2203 is optional. That is, the network device can not send the fifth CSI and the sixth CSI to the terminal. At this time, only the performance of the first AI / ML model used by the terminal side is monitored by the network device.

[0399] Step S2204: The network device and the terminal respectively determine whether the first AI / ML model meets the third performance requirement according to the first CSI, the fifth CSI and the sixth CSI.

[0400] In some embodiments, when step S2203 is omitted, only the network device determines whether the first AI / ML model meets the third performance requirement according to the first CSI, the fifth CSI and the sixth CSI.

[0401] In some embodiments, when the network device and / or the terminal determines that the first AI / ML model does not meet the third performance requirement, it is determined that the performance of the first AI / ML model is changed due to model updating.

[0402] In some embodiments, the network device can first determine whether the third AI / ML model can accurately recover the CSI according to the similarity of the first CSI and the sixth CSI, and determine whether the third performance requirement is met according to the fifth CSI (or the first CSI) and the sixth CSI in a case that the third AI / ML model can accurately recover the CSI.

[0403] The model performance monitoring related by the embodiments of the present disclosure can include at least one of steps S2201-S2204.

[0404] In some embodiments, step S2203 is optional, and can be omitted or replaced in different embodiments.

[0405] In some embodiments, steps S2201-S2204 can be combined with one or more steps in FIG. 2A, for example, can be combined with steps S2101-S2103, can be combined with steps S2104-S2107, and can be combined with steps S2110-S2113, but are not limited thereto.

[0406] FIG. 2C is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 2C, the embodiments of the present disclosure relate to a model performance monitoring method, and the method includes:

[0407] In step S2301, the terminal sends the third CSI, the fifth quantization information, and the third quantization information to the network device.

[0408] In some embodiments, step S2301 can be performed by the terminal after receiving the first information.

[0409] In some embodiments, step S2301 can be performed by the terminal after receiving the first information and determining that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model.

[0410] For example, the terminal can input the first CSI into the second AI / ML model to obtain the fifth quantization information, and input the third CSI into the first AI / ML model to obtain the third quantization information, and further send the fifth quantization information and the third quantization information to the network device only in a case that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model.

[0411] In some embodiments, the terminal sends second information including the third CSI, the third quantization information, and the fifth quantization information to the network device.

[0412] The optional implementation of the second information can refer to the optional implementation involved in step 2103 in FIG. 2A, which is not described herein.

[0413] In some embodiments, the network device receives the second information sent by the terminal. Optionally, the network device receives the third CSI, the fifth quantization information and the third quantization information sent by the terminal. Optionally, after receiving the second information sent by the terminal, the network device performs step S2302. Optionally, after receiving the third CSI, the fifth quantization information and the third quantization information sent by the terminal, the network device performs step S2302.

[0414] In step S2302, the network device inputs the fifth quantization information and the third quantization information into the third AI / ML model respectively to obtain the sixth CSI and the seventh CSI.

[0415] In some embodiments, the network device inputs the fifth quantization information into the third AI / ML model to obtain the sixth CSI.

[0416] In some embodiments, the network device inputs the third quantization information into the third AI / ML model to obtain the seventh CSI.

[0417] In some embodiments, after determining the sixth CSI and the seventh CSI, the network device performs step S2303 or step S2304.

[0418] In step S2303, the network device sends the sixth CSI and the seventh CSI to the terminal.

[0419] In some embodiments, step S2303 is optional. That is, the network device can not send the seventh CSI and the sixth CSI to the terminal. At this time, only the performance of the first AI / ML model used by the terminal side is monitored by the network device.

[0420] In step S2304, the network device and the terminal respectively determine whether the first AI / ML model meets the fourth performance requirement according to the first CSI, the third CSI, the seventh CSI and the sixth CSI.

[0421] In some embodiments, when step S2303 is omitted, only the network device determines whether the first AI / ML model meets the fourth performance requirement according to the first CSI, the third CSI, the seventh CSI and the sixth CSI.

[0422] In some embodiments, when the network device and / or the terminal determines that the first AI / ML model does not meet the fourth performance requirement, it is determined that the performance of the first AI / ML model is changed due to model updating.

[0423] In some embodiments, the network device and / or the terminal can determine whether the third AI / ML model can accurately recover the CSI according to the similarity between the first CSI and the sixth CSI, and determine whether the fourth performance requirement is met according to the similarity between the seventh CSI and the third CSI in a case where it is determined that the third AI / ML model can accurately recover the CSI.

[0424] The model performance monitoring related by the embodiments of the present disclosure can include at least one of steps S2301-S2304.

[0425] In some embodiments, step S2303 is optional, and can be omitted or replaced in different embodiments.

[0426] In some embodiments, steps S2301-S2304 can be combined with one or more steps in FIG. 2A, for example, can be combined with steps S2101-S2103, can be combined with steps S2104-S2107, and can be combined with steps S2110-S2113, but are not limited thereto.

[0427] In some embodiments, steps S2301-S2304 can also be combined with one or more of steps S2201-S2204 in FIG. 2B.

[0428] FIG. 2D is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 2D, the embodiments of the present disclosure relate to a model performance monitoring method, and the above method includes:

[0429] In step S2401, the terminal sends the second CSI, the third CSI, the second quantization information, and the third quantization information to the network device.

[0430] In some embodiments, step S2301 can be performed by the terminal after receiving the first information.

[0431] In some embodiments, step S2301 can be performed by the terminal after receiving the first information and determining that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model.

[0432] In some embodiments, the second CSI can be determined based on steps S2108 and S2109 in FIG. 2A. For optional implementations of the second CSI, please refer to the related part in FIG. 2A, which will not be repeated here.

[0433] For example, the terminal can only input the second CSI into the first AI / ML model to obtain the second quantization information and input the third CSI into the first AI / ML model to obtain the third quantization information in a case where it is determined that the first AI / ML model is updated by the second AI / ML model and / or the third AI / ML model, and further transmit the second quantization information and the third quantization information to the network device.

[0434] In some embodiments, the terminal transmits, to the network device, second information including the second CSI, the third CSI, the second quantization information, and the third quantization information.

[0435] Optional implementation of the second information can refer to the optional implementation involved in step 2103 in FIG. 2A, which is not described herein.

[0436] In some embodiments, the network device receives the second information transmitted by the terminal. Optionally, the network device receives the second CSI, the third CSI, the second quantization information, and the third quantization information transmitted by the terminal. Optionally, after receiving the second information transmitted by the terminal, the network device performs step S2402. Optionally, after receiving the second CSI, the third CSI, the second quantization information, and the third quantization information transmitted by the terminal, the network device performs step S2402.

[0437] In step S2402, the network device inputs the second quantization information and the third quantization information into the third AI / ML model respectively to obtain the fourth CSI and the seventh CSI.

[0438] In some embodiments, the network device inputs the second quantization information into the third AI / ML model to obtain the fourth CSI.

[0439] In some embodiments, the network device inputs the third quantization information into the third AI / ML model to obtain the seventh CSI.

[0440] In some embodiments, after determining the fourth CSI and the seventh CSI, the network device performs step S2403 or step S2404.

[0441] In step S2403, the network device transmits the fourth CSI and the seventh CSI to the terminal.

[0442] In some embodiments, step S2403 is optional. That is, the network device can not transmit the seventh CSI and the sixth CSI to the terminal. At this time, only the performance of the first AI / ML model used by the terminal is monitored by the network device.

[0443] In step S2404, the network device and the terminal respectively determine whether the first AI / ML model meets the fifth performance requirement according to the second CSI, the third CSI, the fourth CSI, and the seventh CSI.

[0444] In some embodiments, when step S2303 is omitted, the network device determines whether the first AI / ML model meets the fifth performance requirement according to the second CSI, the third CSI, the fourth CSI, and the seventh CSI.

[0445] In some embodiments, the network device and / or the terminal can determine whether the fifth performance requirement is met according to the similarity of the second CSI and the fourth CSI, and according to the similarity of the seventh CSI and the third CSI. Alternatively, if either of the above two similarities is lower than a preset threshold, it can be determined that the first AI / ML model does not meet the fifth performance requirement.

[0446] In some embodiments, when the network device and / or the terminal determines that the first AI / ML model does not meet the fifth performance requirement, it is determined that the performance of the first AI / ML model is changed due to model updating.

[0447] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S2401-S2404.

[0448] In some embodiments, step S2403 is optional, and in different embodiments, this step can be omitted or replaced.

[0449] In some embodiments, steps S2401-S2404 can be combined with one or more steps in FIG. 2A, for example, can be combined with steps S2101-S2103, can be combined with steps S2104-S2107, and can be combined with steps S2110-S2113, but are not limited thereto.

[0450] In some embodiments, steps S2401-S2404 can also be combined with one or more of steps S2201-S2204 in FIG. 2B, steps S2301-S2304 in FIG. 2C.

[0451] FIG. 2E is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 2E, the embodiments of the present disclosure relate to a model performance monitoring method, and the method includes:

[0452] Step S2501, the network device sends first information to a plurality of terminals.

[0453] In some embodiments, the first information sent to the plurality of terminals includes the same first CSI. Alternatively, the first CSI predefined by the plurality of terminals is the same.

[0454] In some embodiments, after receiving the first information sent by the network device, the plurality of terminals respectively determine the first quantized information according to the first CSI.

[0455] Some optional implementation manners of step S2501 can refer to step S2103 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C and FIG. 2D, which will not be repeated here.

[0456] Step S2502, the network device receives a plurality of first quantized information sent by a plurality of terminals.

[0457] Some optional implementation manners of step S2502 can refer to step S2104 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C and FIG. 2D, which will not be repeated here.

[0458] Step S2503, the network device determines distribution information of the first quantized information according to the first quantized information sent by the plurality of terminals.

[0459] In some embodiments, the distribution information can be used to indicate the difference between each first quantized information and the mean value of the first quantized information. Alternatively, the distribution information can be used to indicate the standard deviation of each first quantized information. Alternatively, the distribution information can be used to indicate the Z-score of each first quantized information. Alternatively, the distribution information can be used to indicate the distance between each first quantized information and the clustering center, wherein the clustering center can be calculated based on the K-menas algorithm.

[0460] Step S2504, the network device determines a first AI / ML model corresponding to the plurality of terminals from the plurality of first AI / ML models that does not meet the sixth performance requirement according to the distribution information.

[0461] In some embodiments, the network device can determine that the first AI / ML model corresponding to the N first quantized information with the largest difference from the mean value in the first quantized information sent by each terminal does not meet the sixth performance requirement.

[0462] In some embodiments, the network device can determine that the first AI / ML model corresponding to the N first quantized information with the farthest distance from the clustering center in the first quantized information sent by each terminal does not meet the sixth performance requirement.

[0463] In some embodiments, the network device can determine that the first AI / ML model corresponding to the N first quantized information with the largest standard deviation does not meet the sixth performance requirement.

[0464] Alternatively, the value of N can be determined according to the number of terminals, for example, N is five percent of the number of terminals, or the value of N can be a preconfigured value, for example, a value indicated by a higher layer.

[0465] For example, the network device can send the same first information to multiple terminals, receive first quantized information fed back by each terminal, cluster multiple first quantized information, obtain the distance between each first quantized information and the cluster center, and take the first AI / ML model corresponding to the N first quantized information with the largest distance as the first AI / ML model that does not meet the sixth performance requirement, so as to implement joint monitoring of the model of the multiple terminals.

[0466] In some embodiments, the network device determines that the first AI / ML model does not meet the sixth performance requirement, and determines that the first AI / ML model causes performance change due to model deployment.

[0467] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S2401-S2404.

[0468] In some embodiments, steps S2501-S2504 can be combined with one or more steps in FIG. 2A, for example, can be combined with steps S2101-S2103, can be combined with steps S2104-S2107, and can be combined with steps S2110-S2113, but are not limited thereto.

[0469] In some embodiments, steps S2501-S2504 can also be combined with one or more of steps S2201-S2204 in FIG. 2B, steps S2301-S2304 in FIG. 2C, and steps S2401-S2404 in FIG. 2D.

[0470] FIG. 2F is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 2F, the present disclosure relates to a model performance monitoring method, and the above method includes:

[0471] In step S2601, the network device sends third information to the terminal.

[0472] The optional implementation of step S2601 can refer to the optional implementation of step 2101 in FIG. 2A, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, and FIG. 2E, which will not be repeated here.

[0473] In step S2602, the terminal sends fourth information to the network device.

[0474] The optional implementation of step S2602 can refer to the optional implementation of step 2102 in FIG. 2A, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, and FIG. 2E, which will not be repeated here.

[0475] At step S2603, the network device sends the first information to the terminal.

[0476] The optional implementation of step S2602 can refer to the optional implementation of step 2103 in FIG. 2A, step S2501 in FIG. 2E, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, and FIG. 2E, which are not described herein again.

[0477] At step S2604, the terminal sends the second information to the network device.

[0478] The optional implementation of step S2604 can refer to step 2104 in FIG. 2A, step S2110, step S2201 in FIG. 2B, step S2301 in FIG. 2C, step S2401 in FIG. 2D, the optional implementation of step S2502 in FIG. 2E, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, and FIG. 2D, which are not described herein again.

[0479] At step S2605, the network device determines whether the first AI / ML model meets the performance requirement and / or the reason for the change in the performance of the first AI / ML model according to the second information.

[0480] The optional implementation of step S2605 can refer to step 2107 in FIG. 2A, step S2113, step S2204 in FIG. 2B, step S2304 in FIG. 2C, step S2404 in FIG. 2D, the optional implementation of step S2503 to step S2504 in FIG. 2E, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, and FIG. 2D, which are not described herein again.

[0481] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S2601 to S2605. For example, step S2602 can be implemented as an independent embodiment, step S2604 can be implemented as an independent embodiment, step S2605 can be implemented as an independent embodiment, steps S2603 to S2604 can be implemented as an independent embodiment, steps S2604 to S2605 can be implemented as an independent embodiment, but the present disclosure is not limited thereto.

[0482] In some embodiments, step S2602 is optional, which can be omitted or replaced in different embodiments.

[0483] FIG. 3A is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 3A, the present disclosure relates to a model performance monitoring method (network device side), which includes:

[0484] At step S3101, the third information is sent.

[0485] The optional implementation of step S3101 can refer to the optional implementation of step 2101 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F. Details are not repeated here.

[0486] In step S3102, fourth information is acquired.

[0487] The optional implementation of step S3102 can refer to the optional implementation of step 2102 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F. Details are not repeated here.

[0488] In step S3103, the first information is sent.

[0489] The optional implementation of step S3103 can refer to the optional implementation of step 2103 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F. Details are not repeated here.

[0490] In step S3104, first quantization information is acquired.

[0491] The optional implementation of step S3104 can refer to the optional implementation of step 2104 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F. Details are not repeated here.

[0492] In step S3105, the first CSI is input into the second AI / ML model to obtain sixth quantization information.

[0493] The optional implementation of step S3105 can refer to the optional implementation of step 2105 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F. Details are not repeated here.

[0494] In step S3106, the sixth quantization information is sent.

[0495] The optional implementation of step S3106 can refer to the optional implementation of step 2106 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F. Details are not repeated here.

[0496] In step S3107, it is determined whether the first AI / ML model meets the first performance requirement according to the first quantization information and the sixth quantization information.

[0497] The optional implementation of step S3107 can be referred to the optional implementation of step 2107 in FIG. 2A and other associated parts in embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E and FIG. 2F. Details are not repeated here.

[0498] Step S3108, sending the first signal.

[0499] The optional implementation of step S3108 can be referred to the optional implementation of step 2108 in FIG. 2A and other associated parts in embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E and FIG. 2F. Details are not repeated here.

[0500] Step S3109, obtaining the second quantization information and the second CSI.

[0501] The optional implementation of step S3109 can be referred to the optional implementation of step 2110 in FIG. 2A and other associated parts in embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E and FIG. 2F. Details are not repeated here.

[0502] Step S3110, inputting the second quantization information into the third AI / ML model to obtain the fourth CSI.

[0503] The optional implementation of step S3110 can be referred to the optional implementation of step 2111 in FIG. 2A and other associated parts in embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E and FIG. 2F. Details are not repeated here.

[0504] Step S3111, sending the fourth CSI.

[0505] The optional implementation of step S3111 can be referred to the optional implementation of step 2112 in FIG. 2A and other associated parts in embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E and FIG. 2F. Details are not repeated here.

[0506] Step S3112, determining whether the first AI / ML model meets the second performance requirement according to the second CSI and the fourth CSI.

[0507] The optional implementation of step S3112 can be referred to the optional implementation of step 2113 in FIG. 2A and other associated parts in embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E and FIG. 2F. Details are not repeated here.

[0508] The model performance monitoring related by the embodiments of the present disclosure can include at least one of steps S3101-S3112. For example, step S3102 can be implemented as an independent embodiment, step S3104 can be implemented as an independent embodiment, step S3107 can be implemented as an independent embodiment, step S3110 can be implemented as an independent embodiment, step S3112 can be implemented as an independent embodiment, steps S3104-S3107 can be implemented as independent embodiments, steps S3109-S3112 can be implemented as independent embodiments, but are not limited thereto.

[0509] In some embodiments, steps S3104 and S3109 can be exchanged in order or executed simultaneously, and steps S3107 and S3112 can be exchanged in order or executed simultaneously.

[0510] In some embodiments, steps S3101-S3106 and steps S3108-S3112 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0511] In some embodiments, steps S3101-S3111 and step S3112 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0512] FIG. 3B is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 3B, the embodiments of the present disclosure relate to a model performance monitoring method (network device side), which includes:

[0513] Step S3201, obtaining first quantization information and fifth quantization information.

[0514] The optional implementation of step S3201 can refer to the optional implementation of step 2201 in FIG. 2B and other associated parts in the embodiments related by FIGS. 2A, 2B, 2C, 2D, 2E, 2F, and 3A, which will not be repeated here.

[0515] Step S3202, inputting the first quantization information and the fifth quantization information into a third AI / ML model respectively to obtain a fifth CSI and a sixth CSI.

[0516] The optional implementation of step S3202 can refer to the optional implementation of step 2202 in FIG. 2B and other associated parts in the embodiments related by FIGS. 2A, 2B, 2C, 2D, 2E, 2F, and 3A, which will not be repeated here.

[0517] Step S3203, transmitting the fifth CSI and the sixth CSI.

[0518] The optional implementation of step S3203 can refer to the optional implementation of step 2203 in FIG. 2B, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, and 3A, which are not described here again.

[0519] In step S3204, it is determined whether the first AI / ML model meets the third performance requirement according to the first CSI, the fifth CSI, and the sixth CSI.

[0520] The optional implementation of step S3204 can refer to the optional implementation of step 2204 in FIG. 2B, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, and 3A, which are not described here again.

[0521] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S3201-S3204.

[0522] In some embodiments, step S3203 is optional, and in different embodiments, this step can be omitted or replaced.

[0523] In some embodiments, steps S3201-S3204 can be combined with one or more steps in FIG. 2A, for example, can be combined with steps S3101-S3103, can be combined with steps S3104-S3107, and can be combined with steps S3109-S3112, but are not limited thereto.

[0524] FIG. 3C is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 3C, the embodiments of the present disclosure relate to a model performance monitoring method (network device side), and the above method includes:

[0525] In step S3301, the third CSI, the fifth quantization information, and the third quantization information are obtained.

[0526] The optional implementation of step S3301 can refer to the optional implementation of step 2301 in FIG. 2C, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, and 3B, which are not described here again.

[0527] In step S3302, the fifth quantization information and the third quantization information are respectively input into the third AI / ML model to obtain the sixth CSI and the seventh CSI.

[0528] The optional implementation of step S3302 can refer to the optional implementation of step 2302 in FIG. 2C, and other associated parts in the embodiments of FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, and 3B, which are not described here again.

[0529] Step S3303, sending the sixth CSI and the seventh CSI.

[0530] The optional implementation of step S3303 can refer to the optional implementation of step 2303 in FIG. 2C, and other associated parts in the embodiments of FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, and 3B, which are not described here again.

[0531] Step S3304, determining whether the first AI / ML model meets the fourth performance requirement according to the first CSI, the third CSI, the seventh CSI, and the sixth CSI.

[0532] The optional implementation of step S3304 can refer to the optional implementation of step 2304 in FIG. 2C, and other associated parts in the embodiments of FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, and 3B, which are not described here again.

[0533] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S3301-S3304.

[0534] In some embodiments, step S3303 is optional, and in different embodiments, this step can be omitted or replaced.

[0535] In some embodiments, steps S3301-S3304 can be combined with one or more steps in FIG. 2A, for example, can be combined with steps S3101-S3103, can be combined with steps S3104-S3107, and can be combined with steps S3109-S3112, but are not limited thereto.

[0536] In some embodiments, steps S3301-S3304 can also be combined with steps S3201-S3204 in FIG. 2B.

[0537] FIG. 3D is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 3D, the present disclosure relates to a model performance monitoring method (network device side), and the above method includes:

[0538] Step S3401, obtaining a second CSI, a third CSI, second quantization information, and third quantization information.

[0539] The optional implementation of step S3401 can refer to the optional implementation of step 2401 in FIG. 2D, and other associated parts in the embodiments of FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, which are not described here.

[0540] In step S3402, the second quantization information and the third quantization information are respectively input into a third AI / ML model to obtain a fourth CSI and a seventh CSI.

[0541] The optional implementation of step S3402 can refer to the optional implementation of step 2402 in FIG. 2D, and other associated parts in the embodiments of FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, which are not described here.

[0542] In step S3403, the fourth CSI and the seventh CSI are sent.

[0543] The optional implementation of step S3403 can refer to the optional implementation of step 2403 in FIG. 2D, and other associated parts in the embodiments of FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, which are not described here.

[0544] In step S3404, whether the first AI / ML model meets a fifth performance requirement is determined according to the second CSI, the third CSI, the fourth CSI, and the seventh CSI.

[0545] The optional implementation of step S3404 can refer to the optional implementation of step 2404 in FIG. 2D, and other associated parts in the embodiments of FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, which are not described here.

[0546] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S3401-S3404.

[0547] In some embodiments, step S3403 is optional, and in different embodiments, this step can be omitted or replaced.

[0548] In some embodiments, steps S3401-S3404 can be combined with one or more steps in FIG. 2A, for example, can be combined with steps S3101-S3103, can be combined with steps S3104-S3107, and can be combined with steps S3109-S3112, but are not limited thereto.

[0549] In some embodiments, steps S3301 to S3304 can also be combined with one or more of steps S3201 to S3204 in FIG. 3B, steps S3301 to S3304 in FIG. 3C.

[0550] FIG. 3E is a flow diagram of a model performance monitoring method, according to an embodiment of the present disclosure. As shown in FIG. 3E, the embodiments of the present disclosure relate to a model performance monitoring method (network device side), which comprises the following steps:

[0551] In step S3501, the first information is sent to the plurality of terminals.

[0552] The optional implementation of step S3501 can refer to the optional implementation of step 2401 in FIG. 2E, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, which will not be repeated here.

[0553] In step S3502, the plurality of first quantization information sent by the plurality of terminals is received.

[0554] The optional implementation of step S3502 can refer to the optional implementation of step 2402 in FIG. 2E, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, which will not be repeated here.

[0555] In step S3503, the distribution information of the first quantization information is determined according to the first quantization information sent by each terminal.

[0556] The optional implementation of step S3503 can refer to the optional implementation of step 2403 in FIG. 2E, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, which will not be repeated here.

[0557] In step S3504, the first AI / ML model that does not meet the sixth performance requirement among the plurality of first AI / ML models corresponding to the plurality of terminals is determined according to the distribution information.

[0558] The optional implementation of step S3504 can refer to the optional implementation of step 2404 in FIG. 2E, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, which will not be repeated here.

[0559] In some embodiments, steps S3501 to S3504 can be combined with one or more steps in FIG. 3A, for example, can be combined with steps S3101 to S3103, can be combined with steps S3104 to S3107, and can be combined with steps S3109 to S3112, but are not limited thereto.

[0560] In some embodiments, steps S3501 to S3304 can also be combined with one or more of steps S3201 to S3204 in FIG. 3B, steps S3301 to S3304 in FIG. 3C, and steps S3401 to S3404 in FIG. 3D.

[0561] FIG. 3F is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 3F, the present embodiment relates to a model performance monitoring method (network device side), which comprises:

[0562] Step S3601: transmitting third information.

[0563] The optional implementation of step S3601 can refer to the optional implementation of step S3101 in FIG. 3A, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, 3E, which will not be repeated here.

[0564] Step S3602: obtaining fourth information.

[0565] The optional implementation of step S3602 can refer to the optional implementation of step 3102 in FIG. 3A, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, 3E, which will not be repeated here.

[0566] Step S3603: transmitting first information.

[0567] The optional implementation of step S3603 can refer to the optional implementation of step 3103 in FIG. 3A, step S3501 in FIG. 3E, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, 3E, which will not be repeated here.

[0568] Step S3604: obtaining second information.

[0569] The optional implementation of step S3604 can refer to the optional implementation of step S3104 in FIG. 3A, step S3109 in FIG. 3A, step S3201 in FIG. 3B, step S3301 in FIG. 3C, step S3401 in FIG. 3D, step S3502 in FIG. 3E, and other associated parts in the embodiments described above with reference to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, and 3E, and will not be described here again.

[0570] In step S3605, it is determined, according to the second information, whether the first AI / ML model meets the performance requirement, and / or the reason for the change in the performance of the first AI / ML model.

[0571] The optional implementation of step S3605 can refer to step S3107 and step S3112 in FIG. 3A, step S3204 in FIG. 3B, step S3304 in FIG. 3C, step S3404 in FIG. 3D, the optional implementation of step S3503 and step S3504 in FIG. 3E, and other associated parts in the embodiments described above with reference to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, and 3E, and will not be described here again.

[0572] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S3601 to S3605. For example, step S3602 can be implemented as an independent embodiment, step S3604 can be implemented as an independent embodiment, step S3605 can be implemented as an independent embodiment, steps S3603 to S3604 can be implemented as an independent embodiment, steps S3604 to S3605 can be implemented as an independent embodiment, but the present disclosure is not limited thereto.

[0573] In some embodiments, step S3602 is optional and can be omitted or replaced in different embodiments.

[0574] FIG. 3G is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 3G, the present disclosure relates to a model performance monitoring method (network device side), and the above method includes the following steps:

[0575] In step S3701, the first information is sent.

[0576] The optional implementation of step S3701 can refer to step S3102 in FIG. 3A, step S3501 in FIG. 3E, step S3603 in FIG. 3F, the optional implementation of FIG. 3E, and other associated parts in the embodiments described above with reference to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 3A, 3B, 3C, 3D, 3E, and 3F, and will not be described here again.

[0577] obtaining the second information.

[0578] The optional implementation of step S3702 can refer to the optional implementation of step S3104 in FIG. 3A, step S3109 in FIG. 3A, step S3201 in FIG. 3B, step S3301 in FIG. 3C, step S3401 in FIG. 3D, step S3502 in FIG. 3E, step S3604 in FIG. 3F, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 3A, FIG. 3B, FIG. 3C, FIG. 3D, FIG. 3E, FIG. 3F, which are not described herein again.

[0579] determining, according to the second information, whether the first AI / ML model meets the performance requirement, and / or a reason for the change in performance of the first AI / ML model.

[0580] The optional implementation of step S3703 can refer to step S3107, step S3112 in FIG. 3A, step S3204 in FIG. 3B, step S3304 in FIG. 3C, step S3404 in FIG. 3D, step S3503 and step S3504 in FIG. 3E, and the optional implementation of step S3605 in FIG. 3F, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 3A, FIG. 3B, FIG. 3C, FIG. 3D, FIG. 3E, FIG. 3F, which are not described herein again.

[0581] In some embodiments, the first information is sent to the terminal, and the first information is used to instruct the terminal to generate the second information based on the first AI / ML model;

[0582] receiving the second information sent by the terminal;

[0583] determining, according to the second information, whether the first AI / ML model meets the performance requirement, and / or a reason for the change in performance of the first AI / ML model.

[0584] The model parameters of the first AI / ML model are the same as the model parameters of the second AI / ML model, or the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model by the terminal;

[0585] The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0586] In some embodiments, the second information includes one or more of the following:

[0587] first quantization information, the first quantization information being obtained by inputting the first channel state information (CSI) into the first AI / ML model by the terminal, the first information comprising the first CSI, or the first CSI being predefined;

[0588] second CSI, the second CSI being measured by the terminal;

[0589] second quantization information, the second quantization information being obtained by inputting the second CSI into the first AI / ML model by the terminal;

[0590] third CSI, the third CSI being updated by the terminal for use in the second AI / ML model;

[0591] third quantization information, the third quantization information being obtained by inputting the third CSI into the first AI / ML model by the terminal;

[0592] fourth quantization information, the fourth quantization information being obtained by inputting the third CSI into the second AI / ML model by the terminal;

[0593] fifth quantization information, the fifth quantization information being obtained by inputting the first CSI into the second AI / ML model by the terminal;

[0594] wherein the second AI / ML model has the same quantization method as the first AI / ML model.

[0595] In some embodiments, the method comprises:

[0596] sending a first signal to the terminal, the first signal being used for the terminal to measure the second CSI.

[0597] In some embodiments, the second information comprises the first quantization information;

[0598] determining whether the first AI / ML model meets the performance requirement according to the second information, comprising:

[0599] inputting the first CSI into the second AI / ML model to obtain sixth quantization information;

[0600] judging whether the first AI / ML model meets the first performance requirement according to the first quantization information and the sixth quantization information.

[0601] In some embodiments, the method comprises:

[0602] sending the sixth quantization information to the terminal, the sixth quantization information being used for the terminal to judge whether the first AI / ML model meets the first performance requirement.

[0603] In some embodiments, the second information comprises the second quantization information and the second CSI;

[0604] According to the second information, it is determined whether the first AI / ML model meets the performance requirement, including:

[0605] The second quantization information is input into the third AI / ML model to obtain fourth CSI;

[0606] According to the second CSI and the fourth CSI, it is determined whether the first AI / ML model meets the second performance requirement.

[0607] In some embodiments, the method comprises:

[0608] The fourth CSI is sent to the terminal, and the fourth CSI is used for the terminal to determine whether the first AI / ML model meets the second performance requirement.

[0609] In some embodiments, the first signal is sent to the terminal, including:

[0610] It is determined that the first AI / ML model meets the first performance requirement, and the first signal is sent to the terminal.

[0611] In some embodiments, the second information comprises the first quantization information and the fifth quantization information;

[0612] According to the second information, it is determined whether the first AI / ML model meets the performance requirement, including:

[0613] The first quantization information is input into the third AI / ML model to obtain fifth CSI;

[0614] The fifth quantization information is input into the third AI / ML model to obtain sixth CSI;

[0615] According to the first CSI, the fifth CSI and the sixth CSI, it is determined whether the first AI / ML model meets the third performance requirement.

[0616] In some embodiments, the method comprises:

[0617] The fifth CSI and the sixth CSI are sent to the terminal, and the fifth CSI and the sixth CSI are used for the terminal to determine whether the first AI / ML model meets the third performance requirement.

[0618] In some embodiments, the second information comprises the third CSI, the fifth quantization information and the third quantization information;

[0619] According to the second information, it is determined whether the first AI / ML model meets the performance requirement, including:

[0620] The third quantization information is input into the third AI / ML model to obtain seventh CSI;

[0621] The fifth quantization information is input into the third AI / ML model to obtain sixth CSI;

[0622] determine whether the first AI / ML model meets the fourth performance requirement according to the first CSI, the third CSI, the seventh CSI and the sixth CSI.

[0623] In some embodiments, the method comprises:

[0624] sending the seventh CSI and the sixth CSI to the terminal, the seventh CSI and the sixth CSI being used by the terminal to determine whether the first AI / ML model meets the fourth performance requirement.

[0625] In some embodiments, the second information comprises the second CSI, the third CSI, the second quantization information and the third quantization information.

[0626] determining whether the first AI / ML model meets the performance requirement according to the second information, comprising:

[0627] inputting the second quantization information into the third AI / ML model to obtain the fourth CSI;

[0628] inputting the third quantization information into the third AI / ML model to obtain the seventh CSI;

[0629] determining whether the first AI / ML model meets the fifth performance requirement according to the second CSI, the third CSI, the fourth CSI and the seventh CSI.

[0630] In some embodiments, the method comprises:

[0631] sending the fourth CSI and the seventh CSI to the terminal, the fourth CSI and the seventh CSI being used by the terminal to determine whether the first AI / ML model meets the fifth performance requirement.

[0632] In some embodiments, the number of terminals is greater than or equal to one, the second information comprises the first quantization information, the first information sent to each terminal comprises the same first CSI, or the first CSI predefined by each terminal is the same.

[0633] determining whether the first AI / ML model meets the performance requirement and / or the reason for the change in the performance of the first AI / ML model according to the second information, comprising:

[0634] determining distribution information of the first quantization information according to the first quantization information sent by each terminal;

[0635] determining, according to the distribution information, a first AI / ML model that does not meet the sixth performance requirement from among a plurality of first AI / ML models corresponding to the plurality of terminals.

[0636] In some embodiments, determining the reason for the change in the performance of the first AI / ML model according to the second information comprises one or more of:

[0637] determining that the model parameters of the first AI / ML model and the second AI / ML model are the same, and that the first AI / ML model does not meet the first performance requirement, determining that the performance change of the first AI / ML model is caused by model deployment;

[0638] determining that the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model by the terminal, and that the first AI / ML model does not meet the first performance requirement, determining that the performance change of the first AI / ML model is caused by model update;

[0639] determining that the first AI / ML model does not meet the second performance requirement or the fifth performance requirement, determining that the performance change of the first AI / ML model is caused by channel information change;

[0640] determining that the first AI / ML model does not meet the third performance requirement or the fourth performance requirement, determining that the performance change of the first AI / ML model is caused by model update;

[0641] determining that the first AI / ML model does not meet the sixth performance requirement, determining that the performance change of the first AI / ML model is caused by model deployment.

[0642] In some embodiments, the method comprises:

[0643] sending third information to the terminal, the third information comprising the second AI / ML model or the model parameters of the second AI / ML model, and / or the third AI / ML model or the model parameters of the third AI / ML model.

[0644] In some embodiments, the third information comprises a plurality of second AI / ML models or the model parameters of the plurality of second AI / ML models, and / or a plurality of third AI / ML models or the model parameters of the plurality of third AI / ML models.

[0645] In some embodiments, the third information comprises the model parameters of a plurality of second AI / ML models, and / or the model parameters of a plurality of third AI / ML models.

[0646] The third information is further used to indicate the second AI / ML models to which the model parameters of the plurality of second AI / ML models correspond respectively, and / or the third AI / ML models to which the model parameters of the plurality of third AI / ML models correspond respectively.

[0647] In some embodiments, the terminal is deployed with a plurality of first AI / ML models, and the method comprises:

[0648] receiving fourth information, the fourth information being used to indicate whether the parameters of each first AI / ML model in the plurality of first AI / ML models are obtained by update.

[0649] FIG. 4A is a flow diagram illustrating a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 4A, the present embodiment relates to a model performance monitoring method (terminal side), which comprises the following steps:

[0650] In step S4101, the third information is acquired.

[0651] The optional implementation of step S4101 can refer to the optional implementation of step 2101 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F, which will not be repeated here.

[0652] In step S4102, the fourth information is sent.

[0653] The optional implementation of step S4102 can refer to the optional implementation of step 2102 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F, which will not be repeated here.

[0654] In step S4103, the first information is acquired.

[0655] The optional implementation of step S4103 can refer to the optional implementation of step 2103 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F, which will not be repeated here.

[0656] In step S4104, the first quantization information is sent.

[0657] The optional implementation of step S4104 can refer to the optional implementation of step 2104 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F, which will not be repeated here.

[0658] In step S4105, the sixth quantization information is acquired.

[0659] The optional implementation of step S4105 can refer to the optional implementation of step 2106 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F, which will not be repeated here.

[0660] In step S4106, it is determined whether the first AI / ML model meets the first performance requirement according to the first quantization information and the sixth quantization information.

[0661] The optional implementation of step S4106 can refer to the optional implementation of step 2107 in FIG. 2A and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, and 2F, which will not be repeated here.

[0662] Step S4107: obtaining the first signal.

[0663] Optional implementation of step S4107 can refer to optional implementation of step 2108 in FIG. 2A, and other associated parts in the embodiments involved in FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, and FIG. 2F, which are not described here again.

[0664] Step S4108: determining the second CSI according to the first signal.

[0665] Optional implementation of step S4108 can refer to optional implementation of step 2109 in FIG. 2A, and other associated parts in the embodiments involved in FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, and FIG. 2F, which are not described here again.

[0666] Step S4109: sending the second quantization information and the second CSI.

[0667] Optional implementation of step S4109 can refer to optional implementation of step 2110 in FIG. 2A, and other associated parts in the embodiments involved in FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, and FIG. 2F, which are not described here again.

[0668] Step S4110: obtaining the fourth CSI.

[0669] Optional implementation of step S4110 can refer to optional implementation of step 2112 in FIG. 2A, and other associated parts in the embodiments involved in FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, and FIG. 2F, which are not described here again.

[0670] Step S4111: determining whether the first AI / ML model meets the second performance requirement according to the second CSI and the fourth CSI.

[0671] Optional implementation of step S4111 can refer to optional implementation of step 2113 in FIG. 2A, and other associated parts in the embodiments involved in FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, and FIG. 2F, which are not described here again.

[0672] The model performance monitoring involved in the embodiments of the present disclosure can include at least one of steps S4101-S4111. For example, step S4102 can be implemented as an independent embodiment, step S4104 can be implemented as an independent embodiment, step S4109 can be implemented as an independent embodiment, step S4106 can be implemented as an independent embodiment, step S4111 can be implemented as an independent embodiment, steps S4104-S4106 can be implemented as an independent embodiment, steps S4109-S4111 can be implemented as an independent embodiment, but not limited thereto.

[0673] In some embodiments, steps S4104 and S4109 can be exchanged in order or performed simultaneously, and steps S4106 and S4111 can be exchanged in order or performed simultaneously.

[0674] In some embodiments, steps S4101 to S4103 and steps S4105 to S4111 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0675] In some embodiments, steps S4101 to S4108 and steps S4110 to S4111 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0676] FIG. 4B is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 4B, the embodiment of the present disclosure relates to a model performance monitoring method (terminal side), which includes the following steps:

[0677] In step S4201, first quantization information and fifth quantization information are transmitted.

[0678] Optional implementation of step S4201 can refer to optional implementation of step 2201 in FIG. 2B, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, and 4A, which will not be repeated here.

[0679] In step S4202, the fifth CSI and the sixth CSI are obtained.

[0680] Optional implementation of step S4202 can refer to optional implementation of step 2203 in FIG. 2B, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, and 4A, which will not be repeated here.

[0681] In step S4203, it is determined whether the first AI / ML model meets the third performance requirement according to the first CSI, the fifth CSI, and the sixth CSI.

[0682] Optional implementation of step S4203 can refer to optional implementation of step 2204 in FIG. 2B, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, and 4A, which will not be repeated here.

[0683] The model performance monitoring related to the embodiments of the present disclosure can include at least one of steps S4201 to S4203.

[0684] In some embodiments, steps S4202 and S4203 are optional, and one or more of them can be omitted or replaced in different embodiments.

[0685] In some embodiments, steps S4201 to S4203 can be combined with one or more steps in FIG. 4A, for example, can be combined with steps S4101 to S4103, can be combined with steps S4104 to S4106, and can be combined with steps S4109 to S4111, but are not limited thereto.

[0686] FIG. 4C is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 4C, the embodiment of the present disclosure relates to a model performance monitoring method (terminal side), which comprises:

[0687] Step S4301: sending third CSI, fifth quantization information, and third quantization information.

[0688] The optional implementation of step S4301 can refer to the optional implementation of step 2301 in FIG. 2C and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 4A, and 4B, which will not be repeated here.

[0689] Step S4302: obtaining sixth CSI and seventh CSI.

[0690] The optional implementation of step S4302 can refer to the optional implementation of step 2303 in FIG. 2C and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 4A, and 4B, which will not be repeated here.

[0691] Step S4303: determining whether the first AI / ML model meets the fourth performance requirement according to the first CSI, the third CSI, the seventh CSI, and the sixth CSI.

[0692] The optional implementation of step S4303 can refer to the optional implementation of step 2304 in FIG. 2C and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 4A, and 4B, which will not be repeated here.

[0693] The model performance monitoring related to the embodiments of the present disclosure can comprise at least one of steps S4301 to S4303.

[0694] In some embodiments, steps S4302 and S4303 are optional, and one or more of them can be omitted or replaced in different embodiments.

[0695] In some embodiments, steps S4301 to S4303 can be combined with one or more steps in FIG. 4A, for example, can be combined with steps S4101 to S4103, can be combined with steps S4104 to S4106, and can be combined with steps S4109 to S4111, but are not limited thereto.

[0696] FIG. 4D is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 4D, the present embodiment relates to a model performance monitoring method (terminal side), which comprises the following steps:

[0697] In step S4401, the second CSI, the third CSI, the second quantization information, and the third quantization information are transmitted.

[0698] The optional implementation of step S4401 can refer to the optional implementation of step 2401 in FIG. 2D, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 4A, FIG. 4B, and FIG. 4C, which will not be repeated here.

[0699] In step S4402, the fourth CSI and the seventh CSI are obtained.

[0700] The optional implementation of step S4402 can refer to the optional implementation of step 2403 in FIG. 2D, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 4A, FIG. 4B, and FIG. 4C, which will not be repeated here.

[0701] In step S4403, it is determined whether the first AI / ML model meets the fifth performance requirement according to the second CSI, the third CSI, the fourth CSI, and the seventh CSI.

[0702] The optional implementation of step S4403 can refer to the optional implementation of step 2404 in FIG. 2D, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 4A, FIG. 4B, and FIG. 4C, which will not be repeated here.

[0703] The model performance monitoring related to the embodiments of the present disclosure can comprise at least one of steps S4401 to S4403.

[0704] In some embodiments, steps S4402 and S4403 are optional, and one or more of them can be omitted or replaced in different embodiments.

[0705] In some embodiments, steps S4401 to S4403 can be combined with one or more steps in FIG. 4A, for example, can be combined with steps S4101 to S4103, can be combined with steps S4104 to S4106, and can be combined with steps S4109 to S4111, but are not limited thereto.

[0706] FIG. 4E is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 4E, the present embodiment relates to a model performance monitoring method (terminal side), which comprises:

[0707] Step S4501, obtaining third information.

[0708] The optional implementation of step S4501 can refer to the optional implementation of step S4101 in FIG. 4A, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 4A, FIG. 4B, FIG. 4C, FIG. 4D, which will not be repeated here.

[0709] Step S4502, sending fourth information.

[0710] The optional implementation of step S4502 can refer to the optional implementation of step S4102 in FIG. 4A, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 4A, FIG. 4B, FIG. 4C, FIG. 4D, which will not be repeated here.

[0711] Step S4503, obtaining first information.

[0712] The optional implementation of step S4503 can refer to the optional implementation of step S4103 in FIG. 4A, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 4A, FIG. 4B, FIG. 4C, FIG. 4D, which will not be repeated here.

[0713] Step S4504, sending second information.

[0714] The optional implementation of step S4504 can refer to the optional implementation of step S4104, step S4109 in FIG. 4A, step S4201 in FIG. 4B, step S4301 in FIG. 4C, step S4401 in FIG. 4D, and other associated parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 4A, FIG. 4B, FIG. 4C, FIG. 4D, which will not be repeated here.

[0715] The model performance monitoring related by the embodiments of the present disclosure can include at least one of steps S4501-S4504. For example, step S4502 can be implemented as an independent embodiment, step S4504 can be implemented as an independent embodiment, steps S4503-S4504 can be implemented as independent embodiments, but are not limited thereto.

[0716] In some embodiments, step S4502 is optional, and can be omitted or replaced in different embodiments.

[0717] FIG. 4F is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 4F, the embodiments of the present disclosure relate to a model performance monitoring method (terminal side), and the above method includes:

[0718] Step S4601: obtaining first information.

[0719] The optional implementation of step S4601 can refer to the optional implementation of step S4103 in FIG. 4A, step S4503 in FIG. 4E, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 4A, 4B, 4C, 4D, and 4E, which will not be repeated here.

[0720] Step S4602: sending second information.

[0721] The optional implementation of step S4602 can refer to step S4104, step S4109 in FIG. 4A, step S4201 in FIG. 4B, step S4301 in FIG. 4C, step S4401 in FIG. 4D, the optional implementation of step S4504 in FIG. 4E, and other associated parts in the embodiments related to FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 4A, 4B, 4C, 4D, and 4E, which will not be repeated here.

[0722] In some embodiments, the terminal receives the first information sent by the network device;

[0723] According to the first information, the second information is generated based on the first AI / ML model;

[0724] The second information is sent to the network device, and the second information is used by the network device to determine whether the first AI / ML model meets the performance requirement, and / or the reason for the change of the performance of the first AI / ML model;

[0725] The model parameters of the first AI / ML model are the same as the model parameters of the second AI / ML model, or the first AI / ML model is obtained by updating the second AI / ML model and / or the third AI / ML model by the terminal;

[0726] The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device, the second AI / ML model and the first AI / ML model are a first part of a bilateral model, and the third AI / ML model is a second part of the bilateral model.

[0727] In some embodiments, the second information includes one or more of the following:

[0728] The first quantization information is obtained by inputting, by the terminal, first channel state information (CSI) into the first AI / ML model, the first information includes the first CSI, or the first CSI is predefined.

[0729] The second CSI is obtained by measuring, by the terminal,

[0730] The second quantization information is obtained by inputting, by the terminal, the second CSI into the first AI / ML model.

[0731] The third CSI is obtained by updating the CSI used by the first AI / ML model.

[0732] The third quantization information is obtained by inputting, by the terminal, the third CSI into the first AI / ML model.

[0733] The fourth quantization information is obtained by inputting, by the terminal, the third CSI into the second AI / ML model.

[0734] The fifth quantization information is obtained by inputting, by the terminal, the first CSI into the second AI / ML model.

[0735] The first AI / ML model and the second AI / ML model have the same quantization method.

[0736] In some embodiments, the method includes:

[0737] receiving a first signal sent by the network device;

[0738] determining the second CSI according to the first signal.

[0739] In some embodiments, the second information includes the first quantization information.

[0740] The method includes:

[0741] receiving sixth quantization information sent by the network device, the sixth quantization information being obtained by inputting, by the network device, the first CSI into the second AI / ML model;

[0742] determining whether the first AI / ML model meets the first performance requirement according to the first quantization information and the sixth quantization information.

[0743] In some embodiments, the second information includes second quantization information and the second CSI;

[0744] The method includes:

[0745] receiving fourth CSI sent by the network device, the fourth CSI being obtained by the network device inputting the second quantization information into the third AI / ML model;

[0746] According to the second CSI and the fourth CSI, it is judged whether the first AI / ML model meets the second performance requirement.

[0747] In some embodiments, the second information includes first quantization information and fifth quantization information;

[0748] The method includes:

[0749] receiving fifth CSI and sixth CSI sent by the network device, the fifth CSI being obtained by the network device inputting the first quantization information into the third AI / ML model, and the sixth CSI being obtained by the network device inputting the fifth quantization information into the third AI / ML model;

[0750] According to the first CSI, the fifth CSI and the sixth CSI, it is judged whether the first AI / ML model meets the third performance requirement.

[0751] In some embodiments, the second information includes third CSI, fifth quantization information and third quantization information;

[0752] The method includes:

[0753] receiving seventh CSI and sixth CSI sent by the network device, the seventh CSI being obtained by the network device inputting the third quantization information into the third AI / ML model, and the sixth CSI being obtained by the network device inputting the fifth quantization information into the third AI / ML model;

[0754] According to the first CSI, the third CSI, the seventh CSI and the sixth CSI, it is judged whether the first AI / ML model meets the fourth performance requirement.

[0755] In some embodiments, the second information includes second CSI, third CSI, second quantization information and third quantization information;

[0756] The method includes:

[0757] receiving fourth CSI and seventh CSI sent by the network device, the fourth CSI being obtained by the network device inputting the second quantization information into the third AI / ML model, and the seventh CSI being obtained by the network device inputting the third quantization information into the third AI / ML model;

[0758] The first AI / ML model is determined to meet the fifth performance requirement according to the second CSI, the third CSI, the fourth CSI, and the seventh CSI.

[0759] In some embodiments, the method further includes one or more of:

[0760] The model parameters of the first AI / ML model are determined to be the same as those of the second AI / ML model, and the first AI / ML model is determined to not meet the first performance requirement, and it is determined that the performance change of the first AI / ML model is caused by model deployment;

[0761] The first AI / ML model is determined to be obtained by updating the second AI / ML model and / or the third AI / ML model by the terminal, and the first AI / ML model is determined to not meet the first performance requirement, and it is determined that the performance change of the first AI / ML model is caused by model update;

[0762] The first AI / ML model is determined to not meet the second performance requirement or the fifth performance requirement, and it is determined that the performance change of the first AI / ML model is caused by channel information change;

[0763] The first AI / ML model is determined to not meet the third performance requirement or the fourth performance requirement, and it is determined that the performance change of the first AI / ML model is caused by model update.

[0764] In some embodiments, the method includes:

[0765] The third information sent by the network device is received, and the third information includes the second AI / ML model or the model parameters of the second AI / ML model, and / or the third AI / ML model or the model parameters of the third AI / ML model.

[0766] In some embodiments, the third information includes a plurality of second AI / ML models or a plurality of model parameters of the second AI / ML models, and / or a plurality of third AI / ML models or a plurality of model parameters of the third AI / ML models.

[0767] In some embodiments, the third information includes a plurality of model parameters of the second AI / ML models, and / or a plurality of model parameters of the third AI / ML models.

[0768] The third information is further used to indicate the second AI / ML models corresponding to the plurality of model parameters of the second AI / ML models, and / or the third AI / ML models corresponding to the plurality of model parameters of the third AI / ML models.

[0769] In some embodiments, the terminal is deployed with a plurality of first AI / ML models, and the method includes:

[0770] The fourth information is used to indicate whether the parameters of each first AI / ML model in the plurality of first AI / ML models are obtained by updating.

[0771] FIG. 5 is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 5, the embodiment of the present disclosure relates to a model performance monitoring method, and the method comprises:

[0772] In step S5101, the network device sends first information to the terminal.

[0773] The optional implementation of step S5101 can refer to the optional implementation of step S2102 in FIG. 2A, step S3102 in FIG. 3A, step S3501 in FIG. 3E, step S3602 in FIG. 3F, step S3602 in FIG. 3G, step S4102 in FIG. 4A, step S4502 in FIG. 4E, step S4602 in FIG. 4F, and other related parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 3A, FIG. 3B, FIG. 3C, FIG. 3D, FIG. 3E, FIG. 3F, FIG. 3G, FIG. 4A, FIG. 4B, FIG. 4C, FIG. 4D, FIG. 4E, FIG. 4F, which will not be repeated here.

[0774] In step S5102, the terminal sends second information to the network device.

[0775] The optional implementation of step S5102 can refer to step 2104, step S2110 in FIG. 2A, step S2201 in FIG. 2B, step S2301 in FIG. 2C, step S2401 in FIG. 2D, step S2604 in FIG. 2F, step S3104 in FIG. 3A, step S3109 in FIG. 3A, step S3201 in FIG. 3B, step S3301 in FIG. 3C, step S3401 in FIG. 3D, step S3502 in FIG. 3E, step S3604 in FIG. 3F, step S3703 in FIG. 3G, step S4104 in FIG. 4A, step S4109 in FIG. 4A, step S4201 in FIG. 4B, step S4301 in FIG. 4C, step S4401 in FIG. 4D, step S4504 in FIG. 4E, step S4603 in FIG. 4F, and other related parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 3A, FIG. 3B, FIG. 3C, FIG. 3D, FIG. 3E, FIG. 3F, FIG. 3G, FIG. 4A, FIG. 4B, FIG. 4C, FIG. 4D, FIG. 4E, FIG. 4F, which will not be repeated here.

[0776] In step S5103, the network device determines whether the first AI / ML model meets the performance requirement and / or the reason for the change in the performance of the first AI / ML model according to the second information.

[0777] The optional implementation of step S5102 can refer to step S2107, step S2113 in FIG. 2A, step S2204 in FIG. 2B, step S2304 in FIG. 2C, step S2404 in FIG. 2D, step S2605 in FIG. 2F, step S3107, S3112 in FIG. 3A, step S3204 in FIG. 3B, step S3304 in FIG. 3C, step S3404 in FIG. 3D, step S3503 and step S3504 in FIG. 3E, step S3605 in FIG. 3F, step S3703 in FIG. 3G, and other related parts in the embodiments related to FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG. 2F, FIG. 3A, FIG. 3B, FIG. 3C, FIG. 3D, FIG. 3E, FIG. 3F, FIG. 3G, FIG. 4A, FIG. 4B, FIG. 4C, FIG. 4D, FIG. 4E, FIG. 4F, which are not described herein.

[0778] In some embodiments, the above method can include the method described in the above embodiments related to the network device side and the terminal side, which are not described herein.

[0779] FIG. 6 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. As shown in FIG. 6, the present disclosure relates to a model performance monitoring method, and the above method includes:

[0780] In step S6101, the UE takes the pre-defined target CSI, or the target CSI sent by the NW, or the target CSI for training a new model as the encoder input, to obtain information for monitoring the model performance deployed by the UE.

[0781] In some embodiments, the model deployed by the UE can be the first AI / ML model described in the above embodiments. Alternatively, the first AI / ML model can be an updated model, or a model without updating. For example, the UE can directly use the received model parameters or model for inference. Alternatively, the UE can retrain a new model based on the received model parameters or model for inference.

[0782] For the scenario where the UE directly uses the received model parameters or model for inference, the following embodiments can be used:

[0783] In some embodiments, the NW sends the target CSI to the UE, or the CSI compression information output by the encoder on the network side after quantization, wherein the target CSI is the input information of the encoder on the network side, and the quantization of the CSI compression information is the same as that of the CSI feedback. The UE takes the received target CSI as the input information of the encoder on the UE side, and reports the CSI feedback information.

[0784] Optionally, the NW side determines whether the model deployed at the UE side causes performance change due to the deployed model according to the CSI feedback quantization information reported by the UE and the quantized CSI compression information output by the network side encoder.

[0785] Optionally, the UE side determines whether the model deployed at the UE side causes performance change due to the deployed model according to the CSI feedback quantization information output by the UE side encoder and the CSI compression quantization information sent by the network side.

[0786] In some embodiments, the NW instructs the UE to use a predefined target CSI as the input of the UE side encoder.

[0787] Optionally, the NW side determines whether the model deployed at the UE side causes performance change due to the deployed model according to the CSI feedback quantization information reported by the UE and the quantized CSI compression information output by the network side encoder.

[0788] In some embodiments, the NW sends the target CSI to N UEs, N is greater than 1, and the UEs use the received target CSI as the input of the encoder. Or the NW instructs the N UEs to use the predefined target CSI as the input of the UE side encoder.

[0789] Optionally, the NW side determines which UE side model causes performance change due to the deployed model according to different CSI feedback quantization information reported by different UEs.

[0790] In some embodiments, the NW side instructs the UE to use the predefined target CSI or the target CSI sent by the NW to the UE and the target CSI measured by the UE in the actual scene as the input of the UE side encoder, and quantizes the corresponding CSI feedback information and reports it to the NW. Or the NW sends the recovered target CSI output by the decoder to the UE side.

[0791] Optionally, the NW side determines whether the model deployed at the UE side meets the performance requirements according to the two different CSI feedback quantization information reported by the UE and the reported target CSI, to verify whether the performance change is caused by the change of the channel information measured in the field.

[0792] Optionally, the UE determines whether the model performance change is caused by the change of the channel information measured in the field according to the received recovered target CSI and the target CSI input by the encoder.

[0793] For the scenario that the UE re-trains a new model on the basis of the received model parameters or model for inference, the following embodiments can be used:

[0794] In some embodiments, the UE indicates that the model is updated by sending an indication information to the NW. Optionally, the indication information can be A bits of information indicating whether the model is updated, or a new model ID or function ID, or an ID corresponding to the training data set associated with the updated model implicitly indicating whether the model is updated.

[0795] In some embodiments, the NW side indicates the UE to use the pre-defined target CSI or the target CSI sent by the NW to the UE as the input of the encoder on the UE side.

[0796] Optionally, the NW side determines whether the model on the UE side causes performance changes due to the update of the model according to the CSI feedback quantization information reported by the UE and the compressed CSI quantization information output by the encoder on the NW side.

[0797] In some embodiments, the NW side indicates the UE to use the pre-defined target CSI or the target CSI sent by the NW to the UE and the target CSI trained by the new model as the input of the encoder on the UE side. And the corresponding CSI feedback quantization information and the target CSI trained by the new model are reported to the NW.

[0798] Optionally, the NW side determines whether the model on the UE side causes performance changes due to the update of the model according to the two different CSI feedback quantization information reported by the UE and the target CSI.

[0799] In some embodiments, the NW side indicates the UE to use the target CSI trained by the new model and the target CSI measured on site as the input of the encoder on the UE side, and reports the two different CSI feedback quantization information and the target CSI trained by the new model and the target CSI measured on site to the NW.

[0800] Optionally, the NW side determines whether the performance changes due to the change of the channel information measured on site according to the two CSI feedback quantization information reported by the UE and the target CSI trained by the new model and the target CSI measured on site.

[0801] In the above embodiments, it should be understood that the performance of the AI / ML model can be determined by the combination of the schemes involved in one or more of the above embodiments to determine the cause of the performance change.

[0802] Based on some of the embodiments described above, the present disclosure also provides the following examples:

[0803] In an example, assuming the NW side has trained the encoder and decoder models based on the collected training dataset, if the encoder model structure has been standardized and the NW has passed the parameters of the encoder to the UE, the UE can directly use the received encoder parameters for inference.

[0804] For the NW side to verify the performance of the encoder deployed on the UE side: the NW sends the network-side target CSI to the UE, or predefines the target CSI as the input of the encoder through negotiation between the NW and the UE, and then the UE sends the quantized output information C1 of the encoder to the NW. On the NW side, the target CSI sent to the UE or the predefined target CSI is taken as the input of the NW-side encoder, and the output information of the NW-side encoder is obtained by using the same quantization method as the UE side to obtain C2. The NW determines whether the model on the UE side is deteriorated or whether the encoder deployed on the UE can meet the performance requirements by comparing the relationship between C1 and C2.

[0805] For the UE side to verify the performance of the encoder deployed on the UE side: in addition to sending the target CSI, the NW side also sends the output quantization information C2 of the NW-side encoder to the UE, the UE takes the received target CSI as the input of the UE-side encoder and obtains C1, and the UE determines whether the model on the UE side is deteriorated or whether the encoder deployed on the UE can meet the performance requirements by comparing the relationship between C1 and C2.

[0806] For the NW side to jointly monitor the performance of multiple UEs: the NW can send the same target CSI to N=4 UEs or the four UEs use the same predefined target CSI, and then the four UEs take the same target CSI as the input of the UE-side encoder and report the compressed CSI quantization information to the NW, and the NW determines whether the encoders of which UEs can meet the performance requirements according to the reported compressed CSI information.

[0807] For further verification of the performance of the encoder deployed at the UE side by the NW side whether it changes due to the change of channel information: the NW instructs the UE to use the predefined target CSI as the input of the encoder at the UE side, and the UE reports the output CSI quantization information C3 of the encoder to the NW. The NW uses the predefined target CSI as the input of the encoder at the NW side, and the output CSI quantization information of the encoder at the NW side is C4. The encoder at the UE and the encoder at the NW use the same quantization method. The NW first determines whether the encoder deployed at the UE meets the performance requirement according to C3 and C4. If the performance requirement can be met, the NW sends CSI-RS to the UE for measuring the target CSI on site, and instructs the UE to report the measured target CSI and the compressed CSI output by the encoder to the NW together. The NW inputs the received compressed CSI into the decoder to obtain the recovered target CSI, and then determines whether the encoder deployed at the UE meets the performance requirement by calculating the NMMSE or SGCS between the recovered target CSI and the target CSI delivered by the UE.

[0808] In another example, the UE can be preconfigured with the model structure of 4 encoders, assuming that the NW delivers the model parameters of 4 encoders to the UE, the UE can determine the 4 encoders according to the model parameters. Among them, in some embodiments, the NW can also transmit indication information to the UE to indicate the model structure corresponding to the parameters of the 4 encoders respectively. For example, when the NW delivers the model parameters of the encoder to the UE, it also indicates the model structure identification corresponding to each model parameter, and the UE can determine the model corresponding to each model parameter according to the model structure identification corresponding to each model parameter.

[0809] Further, the UE can send the bitmap indication information A=4 bits to the NW to indicate which encoder model the UE has retrained and updated, as shown in Table 1.

[0810] Table 1

[0811] Referring to Table 1, the NW can then know that the UE has updated the first encoder and the third encoder, and the following takes the first encoder as an example.

[0812] Similar to the way of verifying the performance of the encoder deployed at the UE side in the previous example, the NW can instruct the UE to use a predefined target CSI as the input of the first encoder and report the compressed CSI quantization information C5 output by the encoder to the NW. The NW uses the same predefined target CSI as the input of the NW-side encoder and obtains the compressed CSI quantization information C6 output by the encoder, and determines whether the updated encoder meets the performance requirement according to the correlation of C5 and C6 or the SGCS values corresponding to the two.

[0813] For verifying whether the performance of the encoder deployed at the UE side changes due to model updating, the NW can instruct the UE to use a predefined target CSI as the input of the encoder before updating and the input of the encoder after updating respectively, and report the compressed information C7 and C8 output by the two encoders to the NW, and the NW takes C7 and C8 as the input of the decoder and obtains two recovered target CSIs. The NW judges the performance of the updated model according to the SGCS values calculated based on the two recovered target CSIs and the predefined target CSI respectively.

[0814] Alternatively, the UE can also take the predefined target CSI and the target CSI used for updating the model as the input of the encoder before updating and the encoder after updating respectively, and report C7 and C9 output by the encoders and the target CSI used for training the updated model to the NW. The NW takes C7 and C9 as the input of the decoder and obtains two recovered target CSIs, and then calculates the SGCS values according to the recovered target CSIs and the reported target CSI and the predefined target CSI to determine the performance of the updated model.

[0815] It can be understood that for the updated encoder, the monitoring way of whether its performance changes due to the change of channel information can be the same as the way involved in the previous example. In addition, C1 to C9 involved in the above design are only exemplary, and C1 to C9 can be values determined for multiple target CSIs or average values for multiple times. The target CSI can be full channel information or feature vector information of full channel, or channel information in the angle and / or delay domain after DFT inverse transformation, etc.

[0816] In the embodiments of the present disclosure, part or all of the steps, the optional implementation manners thereof, can be combined with part or all of the steps in other embodiments, or can be combined with the optional implementation manners of other embodiments.

[0817] The embodiments of the present disclosure further provide a device for implementing any of the above methods, for example, a device comprising units or modules for implementing the steps performed by the terminal in any of the above methods. For another example, another device is further provided, comprising units or modules for implementing the steps performed by the network equipment (such as an access network device, a core network function node, a core network device, etc.) in any of the above methods.

[0818] It should be understood that the division of each unit or module in the above device is only a logical function division, and all or part of the units or modules can be integrated into one physical entity or physically separated in actual implementation. In addition, the units or modules in the device can be implemented in the form of processor invoking software: for example, the device comprises a processor connected with a memory, the memory stores instructions, and the processor invokes the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit or module of the device, wherein the processor is a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the device or a memory outside the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuit, and the functions of part or all of the units or modules can be implemented by the design of the hardware circuit, and the hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the units or modules are implemented by the design of the logical relationship of the elements in the circuit; for another example, in another implementation, the hardware circuit is a programmable logic device (PLD), and taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to implement the functions of part or all of the units or modules. All units or modules of the above device can be implemented in the form of processor invoking software, or all units or modules can be implemented in the form of hardware circuit, or part of the units or modules are implemented in the form of processor invoking software, and the remaining part is implemented in the form of hardware circuit.

[0819] In embodiments of the present disclosure, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), or the like. In another implementation, the processor can implement certain functions through a logical relationship of hardware circuit. The logical relationship of the hardware circuit is fixed or reconfigurable. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the configuration of the hardware circuit. It can be understood that the processor loads instructions to implement the functions of part or all of the units or modules described above. In addition, the hardware circuit can also be designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), or the like.

[0820] FIG. 7A is a structural schematic diagram of a terminal according to an embodiment of the present disclosure. As shown in FIG. 7A, the terminal 7100 can include at least one of a transceiver module 7101, a processing module 7102, and the like. Optionally, the transceiver module 7101 is configured to perform at least one of the communication steps, such as sending and / or receiving, performed by the terminal in any of the methods described above. Details are not described herein again. Optionally, the processing module 7102 is configured to perform at least one of the other steps performed by the terminal in any of the methods described above. Details are not described herein again.

[0821] FIG. 7B is a structural schematic diagram of a network device according to an embodiment of the present disclosure. As shown in FIG. 7B, the network device 7200 can include at least one of a transceiver module 7201, a processing module 7202, and the like. Optionally, the transceiver module 7201 is configured to perform at least one of the communication steps, such as sending and / or receiving, performed by the network device in any of the methods described above. Details are not described herein again. Optionally, the processing module 7202 is configured to perform at least one of the other steps performed by the network device in any of the methods described above. Details are not described herein again.

[0822] In some embodiments, the transceiving module can include a transmitting module and / or a receiving module, which can be separate or integrated together. Alternatively, the transceiving module can be mutually replaced with a transceiver.

[0823] In some embodiments, the processing module can be one module or include multiple sub-modules. Alternatively, the multiple sub-modules perform all or part of the steps required to be performed by the processing module, respectively. Alternatively, the processing module can be mutually replaced with a processor.

[0824] FIG. 8A is a structural schematic diagram of a communication device 8100 according to the embodiments of the present disclosure. The communication device 8100 can be a network device (such as an access network device, a core network device, etc.), a terminal (such as a user equipment, etc.), a chip, a chip system, or a processor supporting the network device to implement any of the above methods, or a chip, a chip system, or a processor supporting the terminal to implement any of the above methods. The communication device 8100 can be used to implement the methods described in the above method embodiments, and details can be referred to the descriptions in the above method embodiments.

[0825] As shown in FIG. 8A, the communication device 8100 includes one or more processors 8101. The processor 8101 can be a general-purpose processor or a special-purpose processor, for example, a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, and the central processing unit can be used to control the communication device (such as a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute programs, and process data of the programs. Alternatively, the communication device 8100 is configured to perform any of the above methods. Alternatively, the one or more processors 8101 are configured to invoke instructions to cause the communication device 8100 to perform any of the above methods.

[0826] In some embodiments, the communication device 8100 further includes one or more transceivers 8102. When the communication device 8100 includes the one or more transceivers 8102, the transceiver 8102 performs at least one of the communication steps such as transmitting and / or receiving in the above methods, and the processor 8101 performs at least one of the other steps. In alternative embodiments, the transceiver can include a receiver and / or a transmitter, which can be separate or integrated together. Alternatively, the terms of transceiver, transceiving unit, transceiver, transceiving circuit, interface circuit, interface, etc. can be mutually replaced, and the terms of transmitter, transmitting unit, transmitter, transmitting circuit, etc. can be mutually replaced, and the terms of receiver, receiving unit, receiver, receiving circuit, etc. can be mutually replaced.

[0827] In some embodiments, the communication device 8100 further includes one or more memories 8103 for storing data. Alternatively, all or part of the memories 8103 can be external to the communication device 8100. In optional embodiments, the communication device 8100 can include one or more interface circuits 8104. Optionally, the interface circuit 8104 is connected to the memory 8103, and the interface circuit 8104 can be used to receive data from the memory 8103 or other devices, and can be used to send data to the memory 8103 or other devices. For example, the interface circuit 8104 can read data stored in the memory 8103 and send the data to the processor 8101.

[0828] The communication device 8100 described in the above embodiments can be a network device or a terminal, but the scope of the communication device 8100 described in the present disclosure is not limited thereto, and the structure of the communication device 8100 can not be limited by Figure 8A. The communication device can be a standalone device or can be part of a larger device. For example, the communication device can be: 1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally include a storage component for storing data, programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, a smart terminal device, a cellular phone, a wireless device, a handset, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) other devices, etc.

[0829] Figure 8B is a structural schematic diagram of a chip 8200 according to an embodiment of the present disclosure. For the case where the communication device 8100 is a chip or a chip system, the structural schematic diagram of the chip 8200 shown in Figure 8B can be referred to, but is not limited thereto.

[0830] The chip 8200 includes one or more processors 8201. The chip 8200 is configured to execute any of the above methods.

[0831] In some embodiments, the chip 8200 further includes one or more interface circuits 8202. Optionally, the terms interface circuit, interface, transceiver pin, etc. can be replaced by each other. In some embodiments, the chip 8200 further includes one or more memories 8203 for storing data. Optionally, all or part of the memories 8203 can be external to the chip 8200. Optionally, the interface circuit 8202 is connected to the memory 8203, and the interface circuit 8202 can be used to receive data from the memory 8203 or other devices, and the interface circuit 8202 can be used to send data to the memory 8203 or other devices. For example, the interface circuit 8202 can read data stored in the memory 8203 and send the data to the processor 8201.

[0832] In some embodiments, the interface circuit 8202 performs at least one of the communication steps such as transmitting and / or receiving in the above-described methods. The interface circuit 8202 performing the communication steps such as transmitting and / or receiving in the above-described methods refers to, for example, the interface circuit 8202 performing data interaction between the processor 8201, the chip 8200, the memory 8203, or a transceiver device. In some embodiments, the processor 8201 performs at least one of the other steps.

[0833] The modules and / or devices described in each of the embodiments of the virtual device, the physical device, the chip, etc. can be combined or separated according to the situation. Optionally, part or all of the steps can also be performed by a plurality of modules and / or devices in cooperation, which is not limited here.

[0834] The disclosure further proposes a storage medium having instructions stored thereon, which, when executed on the communication device 8100, causes the communication device 8100 to perform any of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer-readable storage medium, but is not limited to this, and it can also be a storage medium readable by other devices. Optionally, the storage medium can be a non-transitory storage medium, but is not limited to this, and it can also be a transitory storage medium.

[0835] The disclosure further proposes a program product which, when executed by the communication device 8100, causes the communication device 8100 to perform any of the above methods. Optionally, the program product is a computer program product.

[0836] The disclosure further proposes a computer program which, when executed on a computer, causes the computer to perform any of the above methods.

Claims

1. A method for monitoring model performance, characterized in that, Performed by a network device, the method includes: Send first information to the terminal, the first information being used to instruct the terminal to generate second information based on a first AI / ML model; Receive the second information sent by the terminal; Based on the second information, determine whether the first AI / ML model meets the performance requirements, and / or the reason for the change in the performance of the first AI / ML model; Wherein, the model parameters of the first AI / ML model are the same as those of the second AI / ML model, or the first AI / ML model is obtained by the terminal updating the second AI / ML model and / or the third AI / ML model; The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device. The second AI / ML model and the first AI / ML model are the first part of the bilateral model, and the third AI / ML model is the second part of the bilateral model.

2. The method according to claim 1, characterized in that, The second information includes one or more of the following: The first quantization information is obtained by the terminal inputting the first channel state information (CSI) into the first AI / ML model. The first information includes the first CSI, or the first CSI is predefined. The second CSI is measured by the terminal. The second quantification information is obtained by the terminal inputting the second CSI into the first AI / ML model; The third CSI is the CSI used by the terminal to update and obtain the second AI / ML model; The third quantification information is obtained by the terminal inputting the third CSI into the first AI / ML model; The fourth quantization information is obtained by the terminal inputting the third CSI into the second AI / ML model; The fifth quantization information is obtained by the terminal inputting the first CSI into the second AI / ML model; The second AI / ML model uses the same quantization method as the first AI / ML model.

3. The method according to claim 2, characterized in that, The method further includes: A first signal is sent to the terminal, the first signal being used by the terminal to measure the second CSI.

4. The method according to any one of claims 2-3, characterized in that, The second information includes the first quantification information; The step of determining whether the first AI / ML model meets the performance requirements based on the second information includes: The first CSI is input into the second AI / ML model to obtain the sixth quantization information; Based on the first quantization information and the sixth quantization information, determine whether the first AI / ML model meets the first performance requirement.

5. The method according to claim 4, characterized in that, The method further includes: The sixth quantization information is sent to the terminal, and the sixth quantization information is used by the terminal to determine whether the first AI / ML model meets the first performance requirement.

6. The method according to any one of claims 2-5, characterized in that, The second information includes the second quantization information and the second CSI; The step of determining whether the first AI / ML model meets the performance requirements based on the second information includes: The second quantization information is input into the third AI / ML model to obtain the fourth CSI; Based on the second CSI and the fourth CSI, determine whether the first AI / ML model meets the second performance requirement.

7. The method according to claim 6, characterized in that, The method further includes: The fourth CSI is sent to the terminal, and the fourth CSI is used by the terminal to determine whether the first AI / ML model meets the second performance requirement.

8. The method according to claim 6 or 7, characterized in that, Sending a first signal to the terminal includes: Once it is determined that the first AI / ML model meets the first performance requirement, the first signal is sent to the terminal.

9. The method according to any one of claims 2-8, characterized in that, The second information includes the first quantization information and the fifth quantization information; The step of determining whether the first AI / ML model meets the performance requirements based on the second information includes: The first quantization information is input into the third AI / ML model to obtain the fifth CSI; The fifth quantization information is input into the third AI / ML model to obtain the sixth CSI; Based on the first CSI, the fifth CSI, and the sixth CSI, determine whether the first AI / ML model meets the third performance requirement.

10. The method according to claim 9, characterized in that, The method further includes: The fifth CSI and the sixth CSI are sent to the terminal. The fifth CSI and the sixth CSI are used by the terminal to determine whether the first AI / ML model meets the third performance requirement.

11. The method according to any one of claims 2-10, characterized in that, The second information includes the third CSI, the fifth quantization information, and the third quantization information; The step of determining whether the first AI / ML model meets the performance requirements based on the second information includes: The third quantization information is input into the third AI / ML model to obtain the seventh CSI; The fifth quantization information is input into the third AI / ML model to obtain the sixth CSI; Based on the first CSI, the third CSI, the seventh CSI, and the sixth CSI, determine whether the first AI / ML model meets the fourth performance requirement.

12. The method according to claim 11, characterized in that, The method further includes: The seventh CSI and the sixth CSI are sent to the terminal, and the seventh CSI and the sixth CSI are used by the terminal to determine whether the first AI / ML model meets the fourth performance requirement.

13. The method according to any one of claims 2-12, characterized in that, The second information includes the second CSI, the third CSI, the second quantization information, and the third quantization information; The step of determining whether the first AI / ML model meets the performance requirements based on the second information includes: The second quantization information is input into the third AI / ML model to obtain the fourth CSI; The third quantization information is input into the third AI / ML model to obtain the seventh CSI; Based on the second CSI, the third CSI, the fourth CSI, and the seventh CSI, determine whether the first AI / ML model meets the fifth performance requirement.

14. The method according to claim 13, characterized in that, The method includes: The fourth CSI and the seventh CSI are sent to the terminal. The fourth CSI and the seventh CSI are used by the terminal to determine whether the first AI / ML model meets the fifth performance requirement.

15. The method according to any one of claims 2-14, characterized in that, The number of terminals is greater than or equal to one, and the first information sent to each of the terminals includes the same first CSI, or the first CSI predefined by each of the terminals is the same.

16. The method according to claim 15, characterized in that, The second information includes the first quantification information; The step of determining whether the first AI / ML model meets the performance requirements based on the second information, and / or the reasons for the change in the performance of the first AI / ML model, includes: Based on the first quantization information sent by the multiple terminals, determine the distribution information of the first quantization information; Based on the distribution information, determine the first AI / ML model that does not meet the sixth performance requirement among the multiple first AI / ML models corresponding to the multiple terminals.

17. The method according to claims 2-16, characterized in that, Based on the second information, determine the cause of the performance change in the first AI / ML model, including one or more of the following: If the model parameters of the first AI / ML model and the second AI / ML model are the same, and the first AI / ML model does not meet the first performance requirement, then the performance of the first AI / ML model is determined to have changed due to model deployment. If it is determined that the first AI / ML model is obtained by the terminal updating the second AI / ML model and / or the third AI / ML model, and the first AI / ML model does not meet the first performance requirement, then it is determined that the performance of the first AI / ML model has changed due to the model update. It is determined that the first AI / ML model does not meet the second or fifth performance requirement, and that the first AI / ML model is due to... Changes in channel information cause performance changes; Determine that the first AI / ML model does not meet the third or fourth performance requirements, and determine that the performance of the first AI / ML model has changed due to model updates; It is determined that the first AI / ML model does not meet the sixth performance requirement, and that the performance of the first AI / ML model is affected by the model deployment.

18. The method according to any one of claims 1-17, characterized in that, The method further includes: Send third information to the terminal, the third information including the second AI / ML model or the model parameters of the second AI / ML model, and / or the third AI / ML model or the model parameters of the third AI / ML model.

19. The method according to any one of claims 1-18, characterized in that, The terminal is deployed with multiple of the first AI / ML models, and the method includes: Receive fourth information, which is used to indicate whether the parameters of each of the multiple first AI / ML models are obtained through updating.

20. A method for monitoring model performance, characterized in that, The method, executed by a terminal, includes: Receive the first transmission information sent by the network device; Based on the first information, generate the second information using the first AI / ML model; The second information is sent to the network device, and the second information is used by the network device to determine whether the first AI / ML model meets the performance requirements, and / or the reason for the change in the performance of the first AI / ML model; Wherein, the model parameters of the first AI / ML model are the same as those of the second AI / ML model, or the first AI / ML model is obtained by the terminal updating the second AI / ML model and / or the third AI / ML model; The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device. The second AI / ML model and the first AI / ML model are the first part of the bilateral model, and the third AI / ML model is the second part of the bilateral model.

21. The method according to claim 20, characterized in that, The second information includes one or more of the following: The first quantization information is obtained by the terminal inputting the first channel state information (CSI) into the first AI / ML model. The first information includes the first CSI, or the first CSI is predefined. The second CSI is the measurement obtained by the terminal. The second quantification information is obtained by the terminal inputting the second CSI into the first AI / ML model; The third CSI is the CSI used to update the first AI / ML model; The third quantification information is obtained by the terminal inputting the third CSI into the first AI / ML model; The fourth quantization information is obtained by the terminal inputting the third CSI into the second AI / ML model; The fifth quantization information is obtained by the terminal inputting the first CSI into the second AI / ML model; The quantization methods used for the first AI / ML model and the second AI / ML model are the same.

22. The method according to claim 21, characterized in that, The method further includes: Receive the first signal sent by the network device; The second CSI is determined based on the first signal.

23. The method according to any one of claims 21-22, characterized in that, The second information includes the first quantification information; The method further includes: The network device receives the sixth quantization information, which is obtained by the network device by inputting the first CSI into the second AI / ML model. Based on the first quantization information and the sixth quantization information, determine whether the first AI / ML model meets the first performance requirement.

24. The method according to any one of claims 21-23, characterized in that, The second information includes the second quantization information and the second CSI; The method further includes: Receive the fourth CSI sent by the network device, wherein the fourth CSI is obtained by the network device by inputting the second quantization information into the third AI / ML model; Based on the second CSI and the fourth CSI, determine whether the first AI / ML model meets the second performance requirement.

25. The method according to any one of claims 21-24, characterized in that, The second information includes the first quantization information and the fifth quantization information; The method further includes: The network device receives a fifth CSI and a sixth CSI, wherein the fifth CSI is obtained by the network device by inputting the first quantization information into the third AI / ML model, and the sixth CSI is obtained by the network device by inputting the fifth quantization information into the third AI / ML model. Based on the first CSI, the fifth CSI, and the sixth CSI, determine whether the first AI / ML model meets the third performance requirement.

26. The method according to any one of claims 20-25, characterized in that, The second information includes the third CSI, the fifth quantization information, and the third quantization information; The method further includes: The network device receives the seventh CSI and the sixth CSI, wherein the seventh CSI is obtained by the network device by inputting the third quantization information into the third AI / ML model, and the sixth CSI is obtained by the network device by inputting the fifth quantization information into the third AI / ML model. Based on the first CSI, the third CSI, the seventh CSI, and the sixth CSI, determine whether the first AI / ML model meets the fourth performance requirement.

27. The method according to any one of claims 21-26, characterized in that, The second information includes the second CSI, the third CSI, the second quantization information, and the third quantization information; The method further includes: The network device receives a fourth CSI and a seventh CSI, wherein the fourth CSI is obtained by the network device by inputting the second quantization information into the third AI / ML model, and the seventh CSI is obtained by the network device by inputting the third quantization information into the third AI / ML model. Based on the second CSI, the third CSI, the fourth CSI, and the seventh CSI, determine whether the first AI / ML model meets the fifth performance requirement.

28. The method according to any one of claims 21-27, characterized in that, The method also includes one or more of the following: If the model parameters of the first AI / ML model and the second AI / ML model are the same, and the first AI / ML model does not meet the first performance requirement, then the performance of the first AI / ML model is determined to have changed due to model deployment. If it is determined that the first AI / ML model is obtained by the terminal updating the second AI / ML model and / or the third AI / ML model, and the first AI / ML model does not meet the first performance requirement, then it is determined that the performance of the first AI / ML model has changed due to the model update. Determine that the first AI / ML model does not meet the second or fifth performance requirements, and determine that the performance of the first AI / ML model has changed due to changes in channel information; It is determined that the first AI / ML model does not meet the third or fourth performance requirements, and that the performance of the first AI / ML model has changed due to model updates.

29. The method according to any one of claims 20-28, characterized in that, The method further includes: The system receives third information sent by the network device, the third information including the second AI / ML model or the model parameters of the second AI / ML model, and / or the third AI / ML model or the model parameters of the third AI / ML model.

30. The method according to any one of claims 20-29, characterized in that, The terminal is deployed with multiple of the first AI / ML models, and the method further includes: A fourth message is sent to the network device, the fourth message indicating whether the parameters of each of the multiple first AI / ML models have been updated.

31. A communication device, characterized in that, include: The transceiver module is configured to send first information to the terminal, the first information being used to instruct the terminal to generate second information based on a first AI / ML model; The transceiver module is also configured to receive the second information sent by the terminal; The processing module is configured to determine, based on the second information, whether the first AI / ML model meets the performance requirements, and / or the reason for the change in the performance of the first AI / ML model; Wherein, the model parameters of the first AI / ML model are the same as those of the second AI / ML model, or the first AI / ML model is obtained by the terminal updating the second AI / ML model and / or the third AI / ML model; The second AI / ML model and the third AI / ML model are AI / ML models trained by network devices. The second AI / ML model and the first AI / ML model are the first part of the bilateral model, and the third AI / ML model is the second part of the bilateral model.

32. A communication device, characterized in that, include: The transceiver module is configured to receive the first transmission information sent by the network device; The processing module is configured to generate second information based on the first information and a first AI / ML model. The transceiver module is configured to send the second information to the network device, the second information being used by the network device to determine whether the first AI / ML model meets the performance requirements, and / or the reason for the change in the performance of the first AI / ML model; Wherein, the model parameters of the first AI / ML model are the same as those of the second AI / ML model, or the first AI / ML model is obtained by the terminal updating the second AI / ML model and / or the third AI / ML model; The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device. The second AI / ML model and the first AI / ML model are the first part of the bilateral model, and the third AI / ML model is the second part of the bilateral model.

33. A communication device, characterized in that, include: One or more processors; The communication device is used to execute the model performance monitoring method according to any one of claims 1-19 or any one of claims 20-30.

34. A communication system, characterized in that, The network device includes a network device and a terminal, wherein the network device is configured to send first information to the terminal, the first information being used to instruct the terminal to generate second information based on a first AI / ML model; The terminal is configured to receive first information sent by a network device, and generate second information based on a first AI / ML model according to the first information. The network device is configured to receive the second information sent by the terminal, and based on the second information, determine whether the first AI / ML model meets the performance requirements, and / or the reason for the change in the performance of the first AI / ML model; Wherein, the model parameters of the first AI / ML model are the same as those of the second AI / ML model, or the first AI / ML model is obtained by the terminal updating the second AI / ML model and / or the third AI / ML model; The second AI / ML model and the third AI / ML model are AI / ML models trained by the network device. The second AI / ML model and the first AI / ML model are the first part of the bilateral model, and the third AI / ML model is the second part of the bilateral model.

35. A storage medium storing instructions, characterized in that, When the instruction is executed on the communication device, the communication device performs the model performance monitoring method as claimed in any one of claims 1-19 or any one of claims 20-30.

36. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or the instructions are executed by the communication device, the model performance monitoring method as described in any one of claims 1-19 or any one of claims 20-30 is implemented.

Images