Model performance monitoring method and apparatus, and communication system

Abstract

Embodiments of the present application provide a model performance monitoring method and apparatus, and a communication system. The model performance monitoring apparatus is applied to a terminal device, and comprises a first processing unit, which controls the terminal device to perform the following operations: receiving at least a first part of first information sent by a network device, and / or the terminal device reporting at least a second part of the first information to the network device, wherein the first information includes information about an index and / or information about a threshold used when monitoring the performance of a first model, and the first model is an artificial intelligence model.

Description

Method, device and communication system for monitoring model performance Technical Field

[0001] The embodiments of the present application relate to the field of communication technologies. Background Art

[0002] Massive multiple-input multiple-output (MIMO) technology is one of the key technologies for 5G mobile communications. MIMO can provide higher channel capacity, but achieving these benefits depends on obtaining accurate channel state information.

[0003] In MIMO technology, terminal devices measure spatial channels and provide channel state information (CSI) back to the network. Based on this CSI, the network selects an appropriate precoding matrix for downlink transmission to the terminal, minimizing the probability of bit errors in the terminal's reception.

[0004] The channel state information generation and feedback process can be summarized as follows. The network device sends a channel state information reference signal (CSI-RS) to each terminal device. The terminal device estimates the channel using the received CSI-RS and obtains an estimate of the spatial channel matrix. The terminal device further uses the estimated spatial channel to obtain CSI. In new radio (NR) technology, CSI feedback is implicit. That is, the terminal device feeds back CSI in the form of recommended transmission parameters to the network device. These transmission parameters include the channel state information reference signal resource indicator (CQI), precoding matrix indicator (PMI), CSI-RS resource indicator (CRI), synchronization signal block resource indicator (SSBRI), layer indicator (LI), rank indicator (RI), and physical layer RSRP (L1-RSRP). The base station can directly use the parameters recommended by the terminal device for downlink transmission, or it can choose not to use the recommended parameters.

[0005] When using the traditional codebook method to feed back CSI, if the rank of the spatial channel matrix estimated by the terminal device is greater than 1, the RI fed back by the terminal device to the base station (if reported) may be greater than 1. In this case, the PMI is a multi-rank codebook. In NR Rel-15, two codebooks, type I and type II, are defined. The former is a conventional precision codebook and can be used for single-user multiple input multiple output (SU-MIMO) and multi-user multiple input multiple output (MU-MIMO) transmission. The latter is a high-precision codebook, mainly used in MU-MIMO scenarios. The latter has higher accuracy than the former, but has higher overhead. Both NR codebooks use a parameterized codebook structure and are divided into two levels (W=W1W2), where W1 describes the long-term, wideband characteristics of the channel and contains an oversampled DFT beam (group); W2 describes the short-term, subband characteristics of the channel. For the above two codebooks, the selection method of W1 is the same. Regarding the selection of W2, the type I codebook consists of a weighted column selection vector, which selects a beam for the subband from the oversampled beams in W1. In the high-precision type II codebook, W2 is used to linearly combine the DFT beams in W1.

[0006] To address the excessive overhead of the Type II codebook, Rel-16 defines an enhanced Type II codebook (e-type II codebook). The e-type II codebook still uses a two-level structure: reporting a set of wideband beams and then adding a set of narrowband combining coefficients to each beam. The enhancement to the Rel-16 e-type II codebook leverages frequency domain correlation to reduce reporting overhead. Furthermore, the e-type II CSI allows for a two-fold increase in the frequency domain granularity of PMI reporting.

[0007] It should be noted that the above introduction to the technical background is merely intended to provide a clear and complete description of the technical solutions of this application and facilitate understanding by those skilled in the art. Simply because these solutions are described in the background technology section of this application, it should not be assumed that the above technical solutions are well known to those skilled in the art.

[0008] Summary of the Invention

[0009] There is a certain delay in the generation of CSI after the terminal device measures the channel, and there is also a delay in the use of the CSI after the base station performs scheduling (such as MU-MIMO scheduling) after receiving the CSI. Therefore, the time at which the channel corresponding to the CSI is different from the time at which it is applied, which is called channel aging. In some scenarios, such as when the terminal device moves at a fast speed (for example, greater than or equal to 30km / h) or the surrounding environment changes rapidly, the channel aging problem will be serious. In order to cope with the channel aging problem, in Rel-18, an auto-regression (AR) algorithm is used to predict the channel at more than one moment in the future (relative to the moment when CSI is generated), and an enhanced type II codebook for predicted PMI is defined. In this application, we refer to it as the Rel-18 codebook. For the case of predicting the channel at one moment in the future, the Rel-18 codebook is similar to the Rel-16 codebook. In the case of predicting the channel at more than one time instant in the future, the Rel-18 codebook compresses the channel in the Doppler domain by utilizing the time correlation of the channels at more than one time instant.

[0010] With the development of artificial intelligence / machine learning (AI / ML) technology, applying AI / ML technology to the physical layer of wireless communications to solve the difficulties of traditional methods has become a current technical direction.

[0011] Figure 1 is a schematic diagram of a terminal device performing CSI prediction based on an AI / ML model. The input of the AI / ML model is spatial channel information at one or more time points X. The spatial channel information is obtained by the terminal device through channel estimation of the received channel state information reference signal (CSI-RS) and can be a spatial channel matrix or a right singular vector of the spatial channel matrix. The output of the AI / ML model is predicted spatial channel information at one or more time points Y. The predicted spatial channel information can be a spatial channel matrix or a right singular vector of the spatial channel matrix. None of the one or more time points X is later than any of the one or more time points Y. In addition, the AI / ML model may also include a preprocessing module. The preprocessing module may also not be included in the AI / ML model. An example of preprocessing performed by the preprocessing module is singular value decomposition (SVD), another example is two-dimensional discrete Fourier transform (DFT), or other preprocessing may be used.

[0012] As shown in Figure 1, the terminal device uses an AI / ML model to process information from one or more spatial channel matrices derived from channel estimation to obtain a prediction of the CSI at a future time.

[0013] The inventors of this application have discovered that in the method of using an AI / ML model to predict CSI, it is sometimes necessary to perform AI / ML model performance monitoring on the terminal device side, for example, the terminal device calculates a metric used to describe the performance of the AI / ML model. Since downlink transmission may have more than one spatial layer, and the AI / ML model may predict more than one channel at a future time when performing CSI prediction, when performing AI / ML model performance monitoring, the network device and the terminal device need to reach a consensus on the information of the indicator describing the performance and / or the threshold information. However, the prior art does not provide for the information of the indicator and / or the threshold information.

[0014] In response to at least one of the above problems or other similar problems, embodiments of the present application provide a method, device and communication system for monitoring model performance, thereby enabling configuration of information used to monitor model performance, thereby enabling monitoring of model performance.

[0015] According to one aspect of an embodiment of the present application, there is provided an apparatus for monitoring model performance, which is applied to a terminal device. The apparatus includes a first processing unit that controls the terminal device to perform the following operations:

[0016] receiving at least a first part of first information sent by a network device, and / or the terminal device reporting at least a second part of the first information to the network device,

[0017] The first information includes information on indicators and / or thresholds used to monitor the performance of the first model, and the first model is an artificial intelligence model.

[0018] According to another aspect of an embodiment of the present application, there is provided an apparatus for monitoring model performance, which is applied to a network device. The apparatus includes a second processing unit that controls the network device to perform the following operations:

[0019] sending at least a first part of the first information to the terminal device, and / or, the network device receiving at least a second part of the first information reported by the terminal device,

[0020] The first information includes information on indicators and / or thresholds used to monitor the performance of the first model, and the first model is an artificial intelligence model.

[0021] One of the beneficial effects of the embodiments of the present application is that the information used to monitor the performance of the model can be configured, thereby enabling monitoring of the model performance.

[0022] With reference to the following description and accompanying drawings, specific embodiments of the present application are disclosed in detail, indicating the manner in which the principles of the present application can be employed. It should be understood that the embodiments of the present application are not limited in scope. Within the spirit and scope of the appended claims, the embodiments of the present application include many variations, modifications and equivalents.

[0023] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0024] It should be emphasized that the term "include / comprising" when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The elements and features described in one figure or one embodiment of the present application can be combined with the elements and features shown in one or more other figures or embodiments. In addition, in the accompanying drawings, similar reference numerals represent corresponding parts in several figures and can be used to indicate corresponding parts used in more than one embodiment.

[0026] Figure 1 is a schematic diagram of CSI prediction based on AI / ML;

[0027] FIG2 is a schematic diagram of the communication system of the present application;

[0028] FIG3 is a schematic diagram of a method for monitoring model performance according to an embodiment of the first aspect of the present application;

[0029] FIG4 is a schematic diagram of a method for monitoring model performance according to an embodiment of the second aspect of the present application;

[0030] FIG5 is a schematic diagram of an apparatus for monitoring model performance according to an embodiment of the third aspect of the present application;

[0031] FIG6 is a schematic diagram of an apparatus for monitoring model performance according to an embodiment of the fourth aspect of the present application;

[0032] FIG7 is a schematic diagram of a terminal device according to an embodiment of the fifth aspect;

[0033] FIG8 is a schematic diagram of a network device according to an embodiment of the fifth aspect. DETAILED DESCRIPTION

[0034] The above and other features of the present application will become apparent through the following description with reference to the accompanying drawings. In the description and the accompanying drawings, specific embodiments of the present application are disclosed in detail, which illustrate some embodiments in which the principles of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the present application includes all modifications, variations and equivalents that fall within the scope of the appended claims.

[0035] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements from the name, but do not indicate the spatial arrangement or temporal order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0036] In the embodiments of this application, the singular forms "a," "the," etc. include plural forms and should be broadly understood to mean "a" or "a type" rather than being limited to "one." Furthermore, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. Furthermore, the term "according to" should be understood to mean "at least in part based on...", and the term "based on" should be understood to mean "at least in part based on...", unless the context clearly indicates otherwise.

[0037] In the embodiments of the present application, the term "communication network" or "wireless communication network" may refer to a network that complies with any of the following communication standards, such as New Radio (NR), Long Term Evolution (LTE), Enhanced Long Term Evolution (LTE-A, LTE-Advanced), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Packet Access (HSPA), etc.

[0038] Furthermore, communication between devices in the communication system may be carried out according to communication protocols of any stage, for example, including but not limited to the following communication protocols: 1G (generation), 2G, 2.5G, 2.75G, 3G, 4G, 4.5G and 5G, New Radio (NR), etc., and / or other communication protocols currently known or to be developed in the future.

[0039] In the embodiments of the present application, the term "network device" refers to, for example, a device in a communication system that connects a terminal device to a communication network and provides services for the terminal device. Network devices may include, but are not limited to, the following devices: an integrated access and backhaul node (IAB-node), a base station (BS), an access point (AP), a transmission reception point (TRP), a broadcast transmitter, a mobile management entity (MME), a gateway, a server, a radio network controller (RNC), a base station controller (BSC), and the like.

[0040] Base stations may include, but are not limited to, NodeBs (NBs), evolved NodeBs (eNodeBs or eNBs), and 5G base stations (gNBs), among others. They may also include remote radio heads (RRHs), remote radio units (RRUs), relays, or low-power nodes (e.g., femeto, pico, etc.). The term "base station" may include some or all of their functions, and each base station may provide communication coverage for a specific geographic area. The term "cell" may refer to a base station and / or its coverage area, depending on the context in which the term is used.

[0041] In the embodiments of the present application, the term "user equipment" (UE) or "terminal equipment" (TE) refers to, for example, a device that accesses a communication network through a network device and receives network services. A terminal device can be fixed or mobile and may also be referred to as a mobile station (MS), a terminal, a subscriber station (SS), an access terminal (AT), a station, and so on.

[0042] Among them, terminal devices may include but are not limited to the following devices: cellular phones, personal digital assistants (PDAs), wireless modems, wireless communication devices, handheld devices, machine-type communication devices, laptop computers, cordless phones, smart phones, smart watches, digital cameras, etc.

[0043] For another example, in scenarios such as the Internet of Things (IoT), the terminal device can also be a machine or device for monitoring or measurement, including but not limited to: machine type communication (MTC) terminal, vehicle-mounted communication terminal, device-to-device (D2D) terminal, machine-to-machine (M2M) terminal, and so on.

[0044] In addition, the term "network side" or "network device side" refers to one side of the network, which can be a base station or one or more network devices as mentioned above. The term "user side" or "terminal side" or "terminal device side" refers to the user or terminal side, which can be a UE or one or more terminal devices as mentioned above.

[0045] In the following description, the terms "uplink control signal" and "uplink control information (UCI)" or "physical uplink control channel (PUCCH)" are interchangeable, and the terms "uplink data signal" and "uplink data information" or "physical uplink shared channel (PUSCH)" are interchangeable to avoid confusion.

[0046] The terms "downlink control signal" and "downlink control information (DCI)" or "physical downlink control channel (PDCCH)" are interchangeable, and the terms "downlink data signal" and "downlink data information" or "physical downlink shared channel (PDSCH)" are interchangeable.

[0047] In addition, sending or receiving PUSCH can be understood as sending or receiving uplink data carried by PUSCH, sending or receiving PUCCH can be understood as sending or receiving uplink information carried by PUCCH, and sending or receiving PRACH can be understood as sending or receiving preamble carried by PRACH; uplink signals can include uplink data signals and / or uplink control signals, etc., and can also be referred to as uplink transmission (UL transmission) or uplink information or uplink channels. Sending uplink transmission on uplink resources can be understood as sending the uplink transmission using the uplink resources. Similarly, downlink data / signals / channels / information can be understood accordingly.

[0048] In the embodiments of the present application, the high-layer signaling may be, for example, radio resource control (RRC) signaling; for example, an RRC message, including, for example, an MIB, system information, or a dedicated RRC message; or an RRC information element (RRC IE). The high-layer signaling may also be, for example, MAC (Medium Access Control) signaling; or a MAC control element (MAC CE). However, the present application is not limited thereto.

[0049] The following describes the scenarios of the embodiments of the present application through examples, but the present application is not limited thereto.

[0050] Figure 2 is a schematic diagram of the communication system of the present application, which schematically illustrates a situation taking a terminal device and a network device as an example. As shown in Figure 2, the communication system 100 may include a network device 201 and a terminal device 202 (for simplicity, Figure 2 only illustrates one terminal device as an example).

[0051] In the embodiment of the present application, existing services or future services can be carried out between the network device 201 and the terminal device 202. For example, these services include but are not limited to: enhanced mobile broadband (eMBB), massive machine type communication (mMTC), and ultra-reliable and low-latency communication (URLLC), etc.

[0052] Among them, the terminal device 202 can send data to the network device 201, for example, using an authorized or unauthorized transmission mode. The network device 201 can receive data sent by one or more terminal devices 202 and feedback information to the terminal device 202, such as confirmation ACK / non-confirmation NACK information. The terminal device 202 can confirm the end of the transmission process, or can continue new data transmission, or can retransmit the data based on the feedback information.

[0053] In the following description of this application, artificial intelligence (AI) models may also be referred to as artificial intelligence / machine learning (AI / ML) models, and these two terms are interchangeable.

[0054] In the following embodiments of the present application, the signaling sent by the network device to the terminal device can be sent through downlink control information (DCI), and / or media access control element (MAC CE), and / or radio resource control (RRC) signaling.

[0055] In the following embodiments of this application, the AI / ML model is on the terminal device side.

[0056] In various embodiments of the present application, reporting may refer to an action of a terminal device sending information to a network device. For example, a terminal device reporting a CSI report may refer to the terminal device sending a CSI report to a network device.

[0057] Embodiments of the first aspect

[0058] Because generating and transmitting CSI takes time, the CSI received by the base station from a terminal device only reflects the channel information at a historical moment. In some practical application scenarios, such as those with fast-moving terminal devices, the channel properties change rapidly. In these cases, CSI feedback is subject to the channel aging problem, meaning that the fed-back CSI cannot describe the channel characteristics at the moment it is received. To address this issue, before generating CSI, spatial channel information can be used to predict spatial channel information and / or CSI at future moments. CSI can then be generated later, if necessary, to improve the accuracy of CSI feedback.

[0059] CSI prediction can be achieved using an artificial intelligence (AI / ML) model. For example, the input of the artificial intelligence (AI / ML) model is spatial channel information at historical moments and / or the current moment, and the output of the artificial intelligence (AI / ML) model is spatial channel information at future moments. The artificial intelligence (AI / ML) model used for CSI prediction can be on the terminal device side or on the network device side. This application considers the scenario where the artificial intelligence (AI / ML) model is on the terminal device side.

[0060] An embodiment of the first aspect of the present application provides a method for monitoring model performance, which is applied to a terminal device.

[0061] FIG3 is a schematic diagram of a method for monitoring model performance. As shown in FIG3 , the method includes:

[0062] 301. The terminal device receives at least a first part of the first information sent by the network device, and / or the terminal device reports at least a second part of the first information to the network device, wherein the first information includes information on indicators and / or threshold information used to monitor the performance of a first model, and the first model is an artificial intelligence model.

[0063] In the present application, the first part can be the part of the first information sent by the network device to the terminal device, can be a part of the first information, or can be all of the first information; the second part can be the part of the first information determined by the terminal device and reported to the network device, can be a part of the first information, or can be all of the first information.

[0064] In the present application, the first information may further include a third part, which may be the part of the first information specified by a standard or pre-agreed upon, or may be a part of the first information or the entire first information.

[0065] In which, at least the third part of the first information is different from or at least partially identical to at least the first part of the first information; at least the third part of the first information is different from or at least partially identical to at least the second part of the first information; at least the first part of the first information is different from or at least partially identical to at least the second part of the first information.

[0066] The first information obtained by the terminal device may be the first part, the second part, the third part or a combination thereof.

[0067] For example, the indicator information in the first information may be specified by a standard, and / or pre-agreed, and / or configured by a network device, and / or set by a terminal device and reported to the network device.

[0068] The information of the indicator may be an index or a name of the indicator. For example, for the indicator squared generalized cosine similarity (SGCS), the information of the indicator may be its index (eg, 000) or the name of the indicator (eg, SGCS).

[0069] The threshold information may be an index or a value of the threshold.

[0070] In the present application, the first model may be the aforementioned artificial intelligence (AI / ML) model for CSI prediction.

[0071] For example, the information input by the first model includes at least one first moment (for example, at least one first moment is moment t1, t2, ..., t m ), that is, the information input to the first model may be more than one first channel information; the information output by the first model includes at least one second moment (for example, at least one second moment is moment s1, s2, ..., s n ), that is, the information output by the first model may be more than one piece of second channel information. The second moment is not earlier than the first moment, that is, no moment in the second moment is earlier than all moments in the first moment. For example, on the time axis, the time value corresponding to the second moment is not less than the time value corresponding to the first moment.

[0072] The first channel information is the spatial channel information estimated by the terminal device using the Channel State Information Reference Signal (CSI-RS) received by the terminal device. The second channel information is the spatial channel information output by the first model. The first channel information can also be called the true value, and the second channel information can also be called the predicted value.

[0073] The method of the present application is described in detail below through different embodiments, and the contents of each embodiment can be combined.

[0074] Example 1:

[0075] Consider monitoring the AI / ML model (i.e., the first model) in the CSI prediction based on the AI / ML model on the terminal device side.

[0076] The terminal device estimates the downlink channel using the received CSI-RS. The terminal device uses the AI / ML model to predict the channel information at a future time based on the information of the estimated downlink channel. That is, the input of the AI / ML model is the information of the estimated downlink channel (i.e., the first channel information at at least one first time), such as the spatial channel matrix or the right singular vector of the spatial channel matrix. The output of the AI / ML model is the channel information at a future time (i.e., the second channel information at at least one second time), such as the spatial channel matrix or the right singular vector of the spatial channel matrix. The monitoring method is to calculate the value of a metric (the value of the metric is used to represent the performance of the CSI prediction based on AI / ML), and compare the calculated value of this metric with a certain threshold value. If the value of the metric is higher than the threshold value, the performance of the AI / ML model is considered to be good enough; if the value of the metric is lower than the threshold value, the performance of the AI / ML model is considered to be not good enough.

[0077] That is, the first channel information at time B is input into the first model, and the first model predicts the channel information at time A (time A is a future time relative to time B) (i.e., outputs the second channel information at time A), and calculates the value of the indicator based on the second channel information at time A (i.e., the predicted value at time A) and the first channel information at time A (i.e., the true value at time A). The value of the indicator is used to evaluate the performance of the first model.

[0078] For example, for AI / ML-based CSI prediction, the spatial channel matrix H = [h i,m,n ] is a three-dimensional matrix, where the subscript i represents the frequency domain dimension and the subscripts m and n represent the spatial dimension. For each frequency domain unit i, there is a two-dimensional channel matrix H with spatial dimension i , then H=[H1,H2,…,H N ], N is the total number of frequency domain units. Assume that H is the spatial channel matrix at time A (i.e., the first time), also known as the ground truth. It is the spatial channel matrix at moment A predicted from the spatial channel matrix information at a historical moment (the historical moment is earlier than the moment A, for example, the historical moment is moment B), also called a predicted value.

[0079] Let r be the value corresponding to the rank indicator (RI) in the CSI report selected by the terminal device. Then let the matrix H i The set of right singular vectors is {w i,1 ,w i,1 ,…,w i,r}, the set of singular values ​​is {σ i,1 ,σ i,1 ,…,σ i,r}, i=1,2,…N. matrix The set of right singular vectors of i=1,2,…N. If there exists an i∈{1,2,…N}, the matrix H i The rank (r i ) is less than r, it can be considered that for k∈{r i +1,…,r},w i,k Is a zero vector. If there exists an i∈{1,2,…N}, the matrix Rank (s i ) is less than r, it can be considered that for k∈{s i +1,…,r}, is the zero vector.

[0080] One possible metric is the normalized mean squared error (NMSE), which is defined as formula (1):

[0081] Another possible indicator is the squared generalized cosine similarity (SGCS), which is defined as follows: for the k-th spatial layer, k∈{1,2…,r}, there is formula (2):

[0082] Where E{·} represents the relationship between multiple The average operation of the sample, ‖·‖ means “l 2 norm”, r is the value corresponding to the rank indicator (RI) in the CSI report selected by the terminal device.

[0083] Example 2:

[0084] In the second embodiment, the relationship between the index and the spatial layer is described.

[0085] For the second channel information at the same time:

[0086] If the number of spatial domain layers is more than one, using a normalized mean square error (NMSE) between the second channel information and the first channel information as the indicator; or

[0087] If the number of spatial domain layers is one, using the squared generalized cosine similarity (SGCS) between the second channel information and the first channel information as the indicator; or

[0088] If the number of spatial domain layers is more than one, the square generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator corresponding to the spatial domain layer, or the arithmetic average or weighted average of the square generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator.

[0089] The weight of the squared generalized cosine similarity (SGCS) of each spatial layer is related to the singular value corresponding to the spatial layer.

[0090] In some embodiments, the normalized mean square error (NMSE) is used as a metric to measure the performance of the AI / ML model. If the number of spatial layers is one or more, the normalized mean square error formula (1) given in Example 1 can be used as a metric to monitor the performance of the AI / ML model.

[0091] In some embodiments, the squared generalized cosine similarity (SGCS) is used as a metric to measure the performance of the AI / ML model. If the number of spatial domain layers is one, the SGCS formula (2) given in Example 1 can be used as a metric to monitor the performance of the AI / ML model. If the number of spatial domain layers is more than one, the following possibilities may exist:

[0092] Possibility 1: Use the SGCS (Formula (2)) of each spatial layer as a metric to monitor the performance of the AI / ML model at each spatial layer. This is beneficial in that it is applicable to monitoring the AI / ML model separately at each spatial layer.

[0093] Possibility 2: Use formula (3)

[0094] This metric is used to monitor the performance of AI / ML models. It monitors the average performance of the AI / ML model on the spatial layer, with all spatial layers equally weighted. This is beneficial when monitoring the AI / ML model on the spatial layer as a whole, with all spatial layers treated equally.

[0095] Possibility 3: Use formula (4)

[0096] It is used as a metric to monitor the performance of AI / ML models. The average performance of the AI / ML model on the spatial layer is monitored, where the weight of the spatial layer may be different, which is related to the singular value corresponding to the spatial layer. The larger the singular value (a non-negative real number), the stronger the power of the spatial layer, and the performance of the AI / ML model in the spatial layer will be considered with a higher weight in the AI / ML model performance monitoring. Its beneficial effect is that it is applicable to the situation where more than one spatial layer is monitored as a whole for AI / ML models, and focuses on AI / ML model monitoring that describes the performance of the spatial layer with strong energy.

[0097] Possibility 4: Use formula (5)

[0098] It is used as a metric to monitor the performance of AI / ML models. The average performance of the AI / ML model on the spatial layer is monitored, where the weight of the spatial layer may be different, which is related to the singular value corresponding to the spatial layer. The larger the singular value (a non-negative real number), the stronger the power of the spatial layer, and the performance of the AI / ML model in the spatial layer will be considered with a higher weight in the AI / ML model performance monitoring. Compared with Possible Three, Possible Four squares the weighting coefficient. Its beneficial effect is that it is suitable for monitoring the AI / ML model of more than one spatial layer as a whole, and compared with Possible Three, it increases the weight of the spatial layer corresponding to the large singular value, which is suitable for AI / ML model monitoring that focuses more on describing the performance of the energy-rich spatial layer than Possible Three.

[0099] In some implementations, different indicators can be selected based on different application scenarios. This selection can be specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or selected and reported by the terminal device to the network device. In other words, there is a mapping relationship between the type of indicator and the application scenario, and the mapping relationship is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0100] For example, if you want to monitor the AI / ML model (i.e., the first model) separately for the spatial layer, the network device can send configuration information to instruct the terminal device to use the squared generalized cosine similarity (SGCS). For AI / ML model detection on the terminal device side, the network device needs to know what indicators the terminal device uses. Only then will the indicator values ​​reported by the terminal device or the results of model monitoring have clear meaning to the network device, and the network device can take the next action based on them.

[0101] Example 3:

[0102] In the third embodiment, the relationship between the indicator and the time is described.

[0103] In some implementations of Example 3:

[0104] The normalized mean square error (NMSE) between the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or

[0105] The normalized mean square error (NMSE) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or

[0106] The normalized mean square error (NMSE) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or

[0107] The first model outputs information at more than one time instant, and the arithmetic average or weighted average of the normalized mean square error (NMSE) of the second channel information and the first channel information corresponding to the more than one time instant is used as the indicator.

[0108] In some implementations of Example 3, for the same spatial layer:

[0109] The squared generalized cosine similarity (SGCS) between the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or

[0110] The squared generalized cosine similarity (SGCS) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or

[0111] The squared generalized cosine similarity (SGCS) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or

[0112] The first model outputs information at more than one time instant, and an arithmetic average or a weighted average of squared generalized cosine similarities (SGCS) corresponding to the second channel information and the first channel information at the more than one time instant is used as the indicator.

[0113] Specifically, in the third embodiment, the AI / ML model is used to make CSI predictions, which can predict the channel information for more than one moment in the future. For example, the first channel information used for prediction is m≥1 is a positive integer, that is is the input of the AI / ML model. The second channel information output by the AI / ML model is n≥1 is a positive integer. Then s1,s2,…,s n is the predicted time (i.e., one or more second time points), and t1<t2<…<t m <s1<s2<…<s n Since the earliest moment among all the prediction moments (s1 in this example) and the moment corresponding to the channel information used for prediction (the moment corresponding to the channel information input to the AI / ML model, in this example, is t1, t2, ..., t m) is the most recent, so the predicted channel information of the earliest moment among all the prediction moments (in this case ) and the channel information used for prediction (in this case ), so the forecast at the earliest time (s1 in this example) is statistically the most accurate. The forecast at s2 is statistically the second most accurate, and so on.

[0114] From the above description, we can see that the corresponding prediction performance is different at different prediction moments. The simulation results show that when the interval between s1, s2, s3, and s4 is 5 milliseconds (ms) and the terminal device is moving at a speed of 30 kilometers per hour (km / h), the performance SGCS of the strongest spatial layer predicted at time s1 (using formula (1)) is 0.2 higher than the performance SGCS of the strongest spatial layer predicted at time s4, which is a very significant difference. Therefore, for predicting channel information for more than one time in the future and monitoring the performance of the AI / ML model, a problem that needs to be solved is: the network device needs to know or instruct the terminal device to monitor the performance of the AI / ML model at which time. Based on this, the value of the indicator reported by the terminal device or the result of the model monitoring has a clear meaning for the network device, and the network device can take the next action based on it. For example, if the terminal device reports the performance SGCS (formula (1)) of the AI / ML model as 0.7, this value is not good for time s1, but good for time s4. The network device needs to know whether the performance of the AI / ML model used for CSI prediction reported by the terminal device corresponds to the corresponding indicator, such as the time at which it was reported.

[0115] In some examples of the present application, the terminal device (receives the configuration of the network device, or the terminal device selects and reports, or is specified in the standard or pre-agreed) monitors the performance of the AI / ML model at the earliest moment of all prediction moments, for example, the indicator is calculated based on the second channel information at moment s1 and the first channel information at moment s1. The beneficial effect is that it is applicable to scenarios where decisions are made based on the results of monitoring the AI / ML model at the moment with the best performance. The CSI prediction performance of the AI / ML model at the earliest prediction moment is the best. By monitoring the CSI prediction performance at this moment, a decision can be made whether to switch to a non-AI / ML method based on the results of the AI / ML model monitoring. That is, if the performance at the moment with the highest CSI prediction accuracy cannot meet the requirements, the CSI prediction performance at the remaining moments cannot meet the requirements either.

[0116] In some examples of the present application, the terminal device (receives the configuration of the network device, or the terminal device selects and reports, or is specified in the standard or pre-agreed) monitors the performance of the AI / ML model at the latest of all prediction moments, for example, the indicator is calculated based on the second channel information at moment s4 and the first channel information at moment s4. The beneficial effect is that it is applicable to scenarios where decisions are made based on the results of monitoring the AI / ML model at the moment with the worst performance. The CSI prediction performance of the AI / ML model at the latest prediction moment may be the worst. By monitoring the CSI prediction performance at this moment, a decision can be made on whether to activate the inactive AI / ML model based on the results of the AI / ML model monitoring, that is, if the performance at the latest moment of CSI prediction accuracy meets the requirements, the CSI prediction performance at the remaining moments can also meet the requirements. Another beneficial effect is to reduce the overhead of the reference signal. Specifically, if the CSI prediction performance of the AI / ML model at the latest prediction moment is good enough, that is, from the latest moment to the last moment before it when the channel is estimated using CSI-RS, the predicted channel information can be used instead of the channel information estimated by CSI-RS. In this way, less CSI-RS can be sent at these moments and the moments before them, reducing the overhead of the reference signal.

[0117] In some examples of this application, a terminal device (receives a configuration from a network device, or selects and reports it, or is specified by a standard or pre-agreed upon) monitors the performance of an AI / ML model at a specific moment among all prediction moments. For example, the specific moment is s3, and the indicator is calculated based on the second channel information at s3 and the first channel information at s3. This has the beneficial effect of selectively monitoring the performance of the AI / ML model at a specific moment based on specific needs, which is highly targeted and the model monitoring results can best meet the needs.

[0118] In some examples of the present application, the terminal device (receives the configuration of the network device, or the terminal device selects and reports, or according to the standard or pre-agreed) monitors the average performance of the AI / ML model at all prediction moments. For example, the index at time s2 is calculated based on the second channel information at time s2 and the first channel information at time s2, the index at time s3 is calculated based on the second channel information at time s3 and the first channel information at time s3, the index at time s4 is calculated based on the second channel information at time s4 and the first channel information at time s4, and then the average of the index at time s2, the index at time s3 and the index at time s4 is calculated. Among them, the average value can be an arithmetic mean or a weighted average value (according to actual needs, weights are added to the indicators according to the importance of different moments to meet the needs). Its beneficial effect is that it is suitable for scenarios where the performance of AI / ML models is comprehensively considered for all prediction moments.

[0119] For all of the above embodiments and examples, the indicators may be configured by the network device, and / or specified by a standard, and / or pre-agreed, and / or determined and reported by the terminal device to the network device. For all of the above embodiments, the indicators used to monitor the AI / ML model for CSI prediction may be determined by the network device configuration, and / or determined and reported by the terminal device to the network device, and / or pre-agreed, and / or specified by a standard, based on the needs of the monitored spatial layer, according to the embodiments described in Example 1.

[0120] Example 4:

[0121] In the above embodiment, the terminal device calculates the performance index of the AI / ML model for CSI prediction. The next step, such as switching to a non-AI / ML method, activating an AI / ML model, switching AI / ML models, or selecting an AI / ML model, is determined based on the relationship between the index value and the threshold.

[0122] Example 4 illustrates the threshold information required for AI / ML model monitoring. The threshold can also be called a threshold value.

[0123] In some embodiments, the threshold or threshold information is used to evaluate the performance of the first model. The value of the performance indicator of the AI / ML model calculated by the terminal device is compared with the threshold value to provide a basis for the next operation. The first possibility is that the terminal device makes the comparison and decides the next operation. The second possibility is that the terminal device makes the comparison and makes a recommendation for the next operation, and the terminal device sends the recommendation to the network device, and the network device makes a decision on the next operation based on it. The network device notifies the terminal device of the decision (as described in Example 5). The third possibility is that the terminal device sends the value of the indicator to the network device, and the network device compares it with the threshold value and decides the next operation based on it. The network device notifies the terminal device of the decision.

[0124] In some embodiments, the threshold or threshold information is used to evaluate the performance of the first model compared to the performance of a second model, wherein the second model is an artificial intelligence model or a non-artificial intelligence model, and the first model is different from the second model.

[0125] For example, the threshold value of the difference is obtained by subtracting the value of the possible SGCS at the earliest prediction moment of CSI prediction using AI / ML from the value of the possible SGCS at the earliest prediction moment of CSI prediction using auto-regression (AR) algorithm. The non-AI / ML method can be a network device configuration, and / or a terminal device decision, and / or a pre-agreed agreement, and / or a standard requirement. The non-AI / ML method can be the CSI prediction using the autoregressive algorithm, or it can be the channel obtained by channel estimation based on the received CSI-RS by the terminal device at the most recent moment before the moment B as the predicted channel at the moment B. The process of this implementation method can also be similar to the above three possibilities of the previous implementation method.

[0126] In some implementations, there may be more than one threshold. For example, a single indicator at a given moment may have a single threshold value. This provides clarity and ease of implementation. There may also be more than one threshold value for a single indicator at a given moment. This also allows for different threshold values ​​to be used for different application scenarios and / or requirements for CSI prediction accuracy.

[0127] A mapping relationship is defined between the one or more thresholds and indicators corresponding to information output by the first model at one or more moments. The mapping relationship is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0128] For example:

[0129] The network device configures a threshold for an indicator corresponding to information output by the first model at a moment; or

[0130] The network device configures a threshold selected from one or more candidate thresholds preset or specified by a standard as the threshold for an indicator corresponding to information output by the first model at a moment; or

[0131] The terminal device selects a threshold as the threshold from one or more candidate thresholds that are preset or specified by a standard and / or configured by the network device for an indicator corresponding to the information output by the first model at a moment, and reports information of the selected threshold to the network device; or

[0132] The terminal device specifies a threshold as the threshold for an indicator corresponding to information output by the first model at a moment, and reports information of the specified threshold to the network device.

[0133] For all of the above embodiments, the threshold value may be configured by the network device, and / or specified by a standard, and / or pre-agreed upon, and / or determined and reported to the network device by the terminal device. For example, the network device may configure a threshold value for a particular indicator at a given moment. For another example, if the standard specifies more than one threshold value, the network device may select one of them (if there are more than one threshold value) and configure it to the terminal device. For another example, if the network device and the terminal device agree on more than one threshold value, the terminal device may select one of them (if there are more than one threshold value) and report it to the network device.

[0134] In some embodiments, there is a mapping relationship between the threshold value and the indicator, that is, the indicator monitored by the AI / ML model and the threshold value corresponding to the indicator.

[0135] The threshold information and the indicator information are both configured by the network device; or

[0136] The threshold information is configured by the network device, and the terminal device determines the indicator information and reports the indicator information to the network device; or

[0137] The indicator information is configured by the network device, and the terminal device determines the threshold information and reports the threshold information to the network device; or

[0138] The terminal device determines the threshold information and the indicator information, and reports the threshold information and / or the indicator information to the network device.

[0139] For example, in combination with the implementation methods given in Examples 2 and 3, there may be the following examples.

[0140] Example 1: Both the threshold and the metric are configured by the network device. This corresponds to a situation where the network device fully controls the AI / ML model monitoring. The network device has clear requirements for the metrics to be measured and the thresholds corresponding to the metrics it requires. In this example, the terminal device can report the value of the metric; it can also compare the value of the metric with the threshold and report the comparison result (e.g., below the threshold, not below the threshold) to the network device.

[0141] Example 2: The indicator is selected by the terminal device and reported to the network device, and the threshold value is determined by the network device. In this case, the terminal device may be more aware of its own capabilities and the environment or power consumption situation in which it is located. For example, the terminal device is capable of calculating CSI predictions based on AI / ML, and is also capable of calculating CSI predictions based on autoregressive methods. The terminal device decides to calculate the difference between the possible SGCS values ​​at the earliest prediction moment of the two methods, and reports the indicator used and the value of the indicator to the network device. The network device can select the threshold value based on the report of the terminal device, the network device can decide the next operation, and the network device can also inform the terminal device of the decision through signaling.

[0142] Embodiment 5:

[0143] Example 5 is used to illustrate relevant processes and signaling, including configuration and reporting.

[0144] As shown in Figure 3, the method for monitoring model performance also includes:

[0145] 302. The terminal device calculates the value of the indicator based on the information of the indicator and using the information output by the first model.

[0146] As shown in Figure 3, the method for monitoring model performance also includes:

[0147] 303. The terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, and performs an operation based on the evaluation result; or

[0148] The terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, generates candidate operations based on the evaluation result, sends the candidate operations to the network device, and receives indication information of the network device based on the candidate operations; or

[0149] The terminal device sends the value of the indicator to the network device and receives indication information from the network device, where the indication information is generated according to a result of the network device evaluating the performance of the first model based on the value of the indicator and the threshold.

[0150] The instruction information is used to instruct the terminal device to perform an operation:

[0151] The first model is switched to a third model, or the first model is deactivated, or a fourth model is selected, or the first model is activated, and the third model and the fourth model are each different from the first model.

[0152] The third model is an artificial intelligence model or a non-artificial intelligence model, and the fourth model is an artificial intelligence model.

[0153] The information sent by the terminal device to the network device is carried by at least one of the following messages and / or resources:

[0154] Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

[0155] As shown in Figure 3, the method for monitoring model performance also includes:

[0156] 304. The terminal device receives a signaling sent by the network device to activate the terminal device to monitor the performance of the first model; or

[0157] The terminal device sends a request message to the network device to request monitoring of the performance of the first model.

[0158] The signaling is media access control element (MAC CE) signaling and / or downlink control information (DCI) signaling, or the request information is carried by at least one of the following messages and / or resources:

[0159] Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

[0160] Specifically:

[0161] In some embodiments, AI / ML model performance monitoring on the terminal device side is triggered by a network device.

[0162] The first possibility is that the network device sends a first configuration via an RRC message, where the first configuration includes information about the indicator and / or information about the threshold value. The first configuration may also include the number of times (number of samples) to calculate the value of the indicator, or a minimum value of the number of times (number of samples), or a maximum value of the number of times (number of samples).

[0163] The second possibility is: the network device activates / triggers the AI / ML model performance monitoring on the terminal device side through DCI or MAC CE signaling. This possibility corresponds to that before activating / triggering the AI / ML model performance monitoring on the terminal device side (the AI / ML model performance monitoring is in an inactive state), the network device has configured the indicator information and / or threshold value information (which can be one or more options) through the RRC message, or the configuration does not contain the indicator information and / or threshold value information.

[0164] In some embodiments, the AI / ML model performance monitoring on the terminal device side is triggered by the terminal device. The terminal device sends a request for AI / ML model performance monitoring to the network device. The request can be sent through terminal device auxiliary information (UE assistant information, UAI); it can also be sent through an RRC message; it can also be sent through MAC CE signaling, and the MAC CE signaling can be newly defined (i.e., separately defined); it can also be sent through uplink control information (UCI). The behavior of the network device after receiving the request can be any of the two possibilities mentioned above in this embodiment.

[0165] In some implementations, the terminal device reports the value of the indicator monitored by the AI / ML model to the network device. This corresponds to a scenario where the first configuration of the network device configuration does not include information about the threshold value.

[0166] In some embodiments, the terminal device reports information about the indicator and the value of the indicator monitored by the AI / ML model to the network device. This corresponds to a scenario where the first configuration of the network device configuration does not include information about the indicator, or where the first configuration does not include information about the indicator or threshold value.

[0167] In some embodiments, the terminal device reports information about the results of its monitoring of the AI / ML model to the network device. The information about the results of the monitoring of the AI / ML model can be used to indicate below a threshold value, above a threshold value, not below a threshold value, or not above a threshold value. The corresponding scenario in this case can be that the first configuration of the network device configuration includes information about the indicator and the threshold value.

[0168] The above-mentioned report of the terminal device to the network device can be sent through RRC message; or MAC CE signaling, and the MAC CE signaling can be newly defined (i.e., separately defined); or sent through a dedicated physical random access channel (dedicated PRACH); or sent through a physical uplink control channel resource (PUCCH SR-like resource) similar to a scheduling request; or sent through a physical uplink shared channel transmission in a two-step random access channel (PUSCH transmission in two-step RACH).

[0169] The network device may make a decision based on the received second information (e.g., the second information includes the value of the indicator reported by the terminal device), where the decision includes determining whether the performance of the AI / ML model (e.g., the first model) is better / worse / not worse / not worse than a threshold value or the performance of a non-AI / ML method, and / or the next action after the AI / ML model detection. The second information is the indicator and / or the value of the indicator.

[0170] The first possibility is that the performance of the activated AI / ML model is worse than / inferior to the threshold value or the performance of the non-AI / ML method. The network device sends information about the next action to the terminal device, which may be sent via an RRC message. The information about the next action may be used to indicate switching to a non-AI / ML method, AI / ML model switching, AI / ML model deactivation, or AI / ML model selection.

[0171] Possible 2: If the performance of the inactive AI / ML model is better than / not worse than / a threshold value or the performance of a non-AI / ML method, the network device sends information about the next action to the terminal device, which may be sent via an RRC message. This information about the next action may be used to indicate AI / ML model activation or AI / ML model selection.

[0172] The network device can determine the next action after the AI / ML model monitoring based on the information about the AI / ML model monitoring results reported by the terminal device. The information about the AI / ML model monitoring results can be used to indicate whether the performance of the AI / ML model is better / worse / not worse than / not worse than a threshold value or the performance of a non-AI / ML method.

[0173] The first possibility is that the performance of the activated AI / ML model is worse than / inferior to the threshold value or the performance of the non-AI / ML method. The network device sends information about the next action to the terminal device, which may be sent via an RRC message. The information about the next action may be used to indicate switching to a non-AI / ML method, AI / ML model switching, AI / ML model deactivation, or AI / ML model selection.

[0174] Possible 2: If the performance of the inactive AI / ML model is better than / not worse than / a threshold value or the performance of a non-AI / ML method, the network device sends information about the next action to the terminal device, which may be sent via an RRC message. This information about the next action may be used to indicate AI / ML model activation or AI / ML model selection.

[0175] According to the embodiment of the first aspect of the present application, the information used to monitor the performance of the model can be configured, thereby enabling monitoring of the model performance.

[0176] Embodiments of the second aspect

[0177] The embodiment of the second aspect provides a method for monitoring model performance, which is applied to a network device, such as the network device 201 in Figure 2. For the parts of the embodiment of the second aspect that are the same as those of the embodiment of the first aspect, reference can be made to the description of the embodiment of the first aspect, which will not be repeated here.

[0178] FIG4 is a schematic diagram of a method for monitoring model performance according to an embodiment of the second aspect. The method comprises:

[0179] 401. The network device sends at least a first part of the first information to the terminal device, and / or the network device receives at least a second part of the first information reported by the terminal device, wherein the first information includes information on indicators and / or threshold information used to monitor the performance of a first model, and the first model is an artificial intelligence model.

[0180] In some embodiments, at least the third part of the first information is specified by a standard or pre-agreed, at least the third part of the first information is different from or at least partially identical to at least the first part of the first information, at least the third part of the first information is different from or at least partially identical to at least the second part of the first information, and at least the first part of the first information is different from or at least partially identical to at least the second part of the first information.

[0181] In some embodiments, the information input by the first model includes first channel information at least one first moment, and the information output by the first model includes second channel information at least one second moment, the first channel information is information of the spatial channel estimated by the terminal device through the channel state information reference signal (CSI-RS) received by the terminal device, and the second channel information is information of the spatial channel output by the first model.

[0182] In some embodiments, the second time is no earlier than the first time.

[0183] In some embodiments, the indicator information is an index or a name, and the threshold information is an index or a value.

[0184] In some embodiments, for the same moment:

[0185] The number of spatial domain layers is one or more, and a normalized mean square error (NMSE) between the second channel information and the first channel information is used as the indicator; or

[0186] The number of spatial domain layers is one, and the squared generalized cosine similarity (SGCS) between the second channel information and the first channel information is used as the indicator; or

[0187] If the number of spatial domain layers is more than one, the square generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator corresponding to the spatial domain layer, or the arithmetic average or weighted average of the square generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator.

[0188] In some embodiments, the weight of the squared generalized cosine similarity (SGCS) of each spatial layer is related to the singular value corresponding to the spatial layer.

[0189] In some embodiments, there is a mapping relationship between the types of indicators and application scenarios.

[0190] In some embodiments, the mapping relationship is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0191] In some embodiments, the normalized mean square error (NMSE) of the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or

[0192] The normalized mean square error (NMSE) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or

[0193] The normalized mean square error (NMSE) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or

[0194] The first model outputs information at more than one time instant, and the arithmetic average or weighted average of the normalized mean square error (NMSE) of the second channel information and the first channel information corresponding to the more than one time instant is used as the indicator.

[0195] In some embodiments, for the same spatial layer:

[0196] The squared generalized cosine similarity (SGCS) between the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or

[0197] The squared generalized cosine similarity (SGCS) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or

[0198] The squared generalized cosine similarity (SGCS) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or

[0199] The first model outputs information at more than one time instant, and an arithmetic average or a weighted average of squared generalized cosine similarities (SGCS) corresponding to the second channel information and the first channel information at the more than one time instant is used as the indicator.

[0200] In some embodiments, the indicator information is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0201] In some embodiments, the threshold or information about the threshold is used to evaluate the performance of the first model.

[0202] In some embodiments, the threshold or threshold information is used to evaluate the comparison result between the first model performance and the second model performance.

[0203] In some embodiments, the second model is an artificial intelligence model or a non-artificial intelligence model, and the first model is different from the second model.

[0204] In some embodiments, there is more than one threshold.

[0205] In some embodiments, there is a mapping relationship between the one or more thresholds and indicators corresponding to the information output by the first model at one or more moments.

[0206] In some embodiments, the mapping relationship is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0207] In some embodiments, the network device configures a threshold for an indicator corresponding to information output by the first model at a moment; or

[0208] The network device configures a threshold selected from one or more candidate thresholds preset or specified by a standard as the threshold for an indicator corresponding to information output by the first model at a moment; or

[0209] The terminal device selects a threshold as the threshold from one or more candidate thresholds that are preset or specified by a standard and / or configured by the network device for an indicator corresponding to the information output by the first model at a moment, and the network device receives information of the selected threshold reported by the terminal device; or

[0210] The terminal device specifies a threshold as the threshold for an indicator corresponding to information output by the first model at a moment, and the network device receives information of the specified threshold reported by the terminal device.

[0211] In some embodiments, there is a mapping relationship between one or more thresholds in the threshold information and one or more indicators in the indicator information.

[0212] In some embodiments, the threshold information and the indicator information are both configured by the network device; or

[0213] The threshold information is configured by the network device, the terminal device determines the indicator information, and the network device receives the indicator information reported by the terminal device; or

[0214] The indicator information is configured by the network device, the terminal device determines the threshold information, and the network device receives the threshold information reported by the terminal device; or

[0215] The terminal device determines the threshold information and the indicator information, and the network device receives the threshold information and / or the indicator information reported by the terminal device.

[0216] In some embodiments, as shown in FIG4 , the method further includes:

[0217] 402. The network device receives a candidate operation sent by the terminal device, and sends indication information to the terminal device based on the candidate operation, wherein the terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, and generates the candidate operation based on the evaluation result; or

[0218] The network device receives the value of the indicator sent by the terminal device and sends indication information to the terminal device, where the indication information is generated according to a result of the network device evaluating the performance of the first model based on the value of the indicator and the threshold.

[0219] In some embodiments, the instruction information is used to instruct the terminal device to perform the following operations:

[0220] The first model is switched to a third model, or the first model is deactivated, or a fourth model is selected, or the first model is activated, and the third model and the fourth model are each different from the first model.

[0221] In some embodiments, the third model is an artificial intelligence model or a non-artificial intelligence model, and the fourth model is an artificial intelligence model.

[0222] In some embodiments,

[0223] The network device receives the information sent by the terminal device through at least one of the following messages and / or resources:

[0224] Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

[0225] In some embodiments, as shown in FIG4 , the method further includes:

[0226] 403. The network device sends a signaling to the terminal device to activate the terminal device to monitor the performance of the first model; or

[0227] The network device receives request information sent by the terminal device for requesting to monitor the performance of the first model.

[0228] The signaling is media access control element (MAC CE) signaling and / or downlink control information (DCI) signaling; or, the request information is carried by at least one of the following messages and / or resources:

[0229] Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

[0230] Embodiments of the third aspect

[0231] At least for the same problem as the embodiment of the first aspect, the embodiment of the third aspect of the present application provides an apparatus for monitoring model performance, which is applied to a terminal device and corresponds to the embodiment of the first aspect.

[0232] Fig. 5 is a schematic diagram of an apparatus for monitoring model performance according to an embodiment of the third aspect. As shown in Fig. 5 , the apparatus 500 for monitoring model performance includes: a first processing unit 501 .

[0233] The first processing unit 501 controls the terminal device to perform the following operations:

[0234] Receive at least a first part of the first information sent by the network device, and / or the terminal device reports at least a second part of the first information to the network device, wherein the first information includes information on indicators and / or threshold information used when monitoring the performance of a first model, and the first model is an artificial intelligence model.

[0235] In some embodiments, at least the third portion of the first information is specified by a standard or pre-agreed upon, and the at least the third portion of the first information is different from or at least partially identical to the at least the first portion of the first information.

[0236] At least a third portion of the first information is different from or at least partially identical to at least a second portion of the first information,

[0237] At least a first portion of the first information is different from or at least partially identical to at least a second portion of the first information.

[0238] In some embodiments, the information input by the first model includes first channel information at least one first moment, and the information output by the first model includes second channel information at least one second moment, the first channel information is information of the spatial channel estimated by the terminal device through the channel state information reference signal (CSI-RS) received by the terminal device, and the second channel information is information of the spatial channel output by the first model.

[0239] In some embodiments, the second time is no earlier than the first time.

[0240] In some embodiments, the indicator information is an index or a name, and the threshold information is an index or a value.

[0241] In some embodiments, for the same moment:

[0242] The number of spatial domain layers is one or more, and a normalized mean square error (NMSE) between the second channel information and the first channel information is used as the indicator; or

[0243] The number of spatial domain layers is one, and the squared generalized cosine similarity (SGCS) between the second channel information and the first channel information is used as the indicator; or

[0244] If the number of spatial domain layers is more than one, the square generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator corresponding to the spatial domain layer, or the arithmetic average or weighted average of the square generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator.

[0245] In some embodiments, the weight of the squared generalized cosine similarity (SGCS) of each spatial layer is related to the singular value corresponding to the spatial layer.

[0246] In some embodiments, there is a mapping relationship between the types of indicators and application scenarios.

[0247] In some embodiments, the mapping relationship is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0248] In some embodiments, the normalized mean square error (NMSE) of the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or

[0249] The normalized mean square error (NMSE) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or

[0250] The normalized mean square error (NMSE) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or

[0251] The first model outputs information at more than one time instant, and the arithmetic average or weighted average of the normalized mean square error (NMSE) of the second channel information and the first channel information corresponding to the more than one time instant is used as the indicator.

[0252] In some embodiments, for the same spatial layer:

[0253] The squared generalized cosine similarity (SGCS) between the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or

[0254] The squared generalized cosine similarity (SGCS) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or

[0255] The squared generalized cosine similarity (SGCS) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or

[0256] The first model outputs information at more than one time instant, and an arithmetic average or a weighted average of squared generalized cosine similarities (SGCS) corresponding to the second channel information and the first channel information at the more than one time instant is used as the indicator.

[0257] In some embodiments, the indicator information is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0258] In some embodiments, the threshold or information about the threshold is used to evaluate the performance of the first model.

[0259] In some embodiments, the threshold or threshold information is used to evaluate the comparison result between the first model performance and the second model performance.

[0260] In some embodiments, the second model is an artificial intelligence model or a non-artificial intelligence model,

[0261] The first model is different from the second model.

[0262] In some embodiments, there is more than one threshold.

[0263] In some embodiments, there is a mapping relationship between the one or more thresholds and indicators corresponding to the information output by the first model at one or more moments.

[0264] In some embodiments, the mapping relationship is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0265] In some embodiments, the network device configures a threshold for an indicator corresponding to information output by the first model at a moment; or

[0266] The network device configures a threshold selected from one or more candidate thresholds preset or specified by a standard as the threshold for an indicator corresponding to information output by the first model at a moment; or

[0267] The terminal device selects a threshold as the threshold from one or more candidate thresholds that are preset or specified by a standard and / or configured by the network device for an indicator corresponding to the information output by the first model at a moment, and reports information of the selected threshold to the network device; or

[0268] The terminal device specifies a threshold as the threshold for an indicator corresponding to information output by the first model at a moment, and reports information of the specified threshold to the network device.

[0269] In some embodiments, there is a mapping relationship between one or more thresholds in the threshold information and one or more indicators in the indicator information.

[0270] In some embodiments, the threshold information and the indicator information are both configured by the network device; or

[0271] The threshold information is configured by the network device, and the terminal device determines the indicator information and reports the indicator information to the network device; or

[0272] The indicator information is configured by the network device, and the terminal device determines the threshold information and reports the threshold information to the network device; or

[0273] The terminal device determines the threshold information and the indicator information, and reports the threshold information and / or the indicator information to the network device.

[0274] In some embodiments, the operations further include:

[0275] The terminal device calculates the value of the indicator based on the information of the indicator and using the information output by the first model.

[0276] In some embodiments, the operations further include:

[0277] The terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, and performs an operation based on the evaluation result; or

[0278] The terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, generates candidate operations based on the evaluation result, sends the candidate operations to the network device, and receives indication information of the network device based on the candidate operations; or

[0279] The terminal device sends the value of the indicator to the network device and receives indication information from the network device, where the indication information is generated according to a result of the network device evaluating the performance of the first model based on the value of the indicator and the threshold.

[0280] In some embodiments, the instruction information is used to instruct the terminal device to perform the following operations:

[0281] The first model is switched to a third model, or the first model is deactivated, or a fourth model is selected, or the first model is activated, and the third model and the fourth model are each different from the first model.

[0282] In some embodiments, the third model is an artificial intelligence model or a non-artificial intelligence model, and the fourth model is an artificial intelligence model.

[0283] In some embodiments, the information sent by the terminal device to the network device is carried by at least one of the following messages and / or resources:

[0284] Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

[0285] In some embodiments, the operations further include:

[0286] The terminal device receives a signaling sent by the network device to activate the terminal device to monitor the performance of the first model; or

[0287] The terminal device sends a request message to the network device to request monitoring of the performance of the first model.

[0288] In some embodiments,

[0289] The signaling is media access control element (MAC CE) signaling and / or downlink control information (DCI) signaling; or

[0290] The request information is carried by at least one of the following messages and / or resources:

[0291] Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

[0292] Embodiments of the fourth aspect

[0293] An embodiment of the fourth aspect of the present application provides a device for monitoring model performance, which is applied to a network device and corresponds to the method of the embodiment of the second aspect.

[0294] Fig. 6 is a schematic diagram of an apparatus for monitoring model performance according to an embodiment of the fourth aspect. As shown in Fig. 6 , the apparatus 600 includes: a second processing unit 601 .

[0295] In at least one embodiment, the second processing unit 601 controls the network device to perform the following operations:

[0296] sending at least a first part of the first information to the terminal device, and / or, the network device receiving at least a second part of the first information reported by the terminal device,

[0297] The first information includes information on indicators and / or thresholds used to monitor the performance of the first model, and the first model is an artificial intelligence model.

[0298] In some embodiments, at least the third part of the first information is specified by a standard or pre-agreed, at least the third part of the first information is different from or at least partially identical to at least the first part of the first information, at least the third part of the first information is different from or at least partially identical to at least the second part of the first information, and at least the first part of the first information is different from or at least partially identical to at least the second part of the first information.

[0299] In some embodiments, the information input by the first model includes first channel information at least one first moment, and the information output by the first model includes second channel information at least one second moment, the first channel information is information of the spatial channel estimated by the terminal device through the channel state information reference signal (CSI-RS) received by the terminal device, and the second channel information is information of the spatial channel output by the first model.

[0300] In some embodiments, the second time is no earlier than the first time.

[0301] In some embodiments, the indicator information is an index or a name, and the threshold information is an index or a value.

[0302] In some embodiments, for the same moment:

[0303] The number of spatial domain layers is one or more, and a normalized mean square error (NMSE) between the second channel information and the first channel information is used as the indicator; or

[0304] The number of spatial domain layers is one, and the squared generalized cosine similarity (SGCS) between the second channel information and the first channel information is used as the indicator; or

[0305] If the number of spatial domain layers is more than one, the square generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator corresponding to the spatial domain layer, or the arithmetic average or weighted average of the square generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator.

[0306] In some embodiments, the weight of the squared generalized cosine similarity (SGCS) of each spatial layer is related to the singular value corresponding to the spatial layer.

[0307] In some embodiments, there is a mapping relationship between the types of indicators and application scenarios.

[0308] In some embodiments, the mapping relationship is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0309] In some embodiments, the normalized mean square error (NMSE) of the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or

[0310] The normalized mean square error (NMSE) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or

[0311] The normalized mean square error (NMSE) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or

[0312] The first model outputs information at more than one time instant, and the arithmetic average or weighted average of the normalized mean square error (NMSE) of the second channel information and the first channel information corresponding to the more than one time instant is used as the indicator.

[0313] In some embodiments, for the same spatial layer:

[0314] The squared generalized cosine similarity (SGCS) between the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or

[0315] The squared generalized cosine similarity (SGCS) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or

[0316] The squared generalized cosine similarity (SGCS) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or

[0317] The first model outputs information at more than one time instant, and an arithmetic average or a weighted average of squared generalized cosine similarities (SGCS) corresponding to the second channel information and the first channel information at the more than one time instant is used as the indicator.

[0318] In some embodiments, the indicator information is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0319] In some embodiments, the threshold or information about the threshold is used to evaluate the performance of the first model.

[0320] In some embodiments, the threshold or threshold information is used to evaluate the comparison result between the first model performance and the second model performance.

[0321] In some embodiments, the second model is an artificial intelligence model or a non-artificial intelligence model, and the first model is different from the second model.

[0322] In some embodiments, there is more than one threshold.

[0323] In some embodiments, there is a mapping relationship between the one or more thresholds and indicators corresponding to the information output by the first model at one or more moments.

[0324] In some embodiments, the mapping relationship is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0325] In some embodiments, the network device configures a threshold for an indicator corresponding to information output by the first model at a moment; or

[0326] The network device configures a threshold selected from one or more candidate thresholds preset or specified by a standard as the threshold for an indicator corresponding to information output by the first model at a moment; or

[0327] The terminal device selects a threshold as the threshold from one or more candidate thresholds that are preset or specified by a standard and / or configured by the network device for an indicator corresponding to the information output by the first model at a moment, and the network device receives information of the selected threshold reported by the terminal device; or

[0328] The terminal device specifies a threshold as the threshold for an indicator corresponding to information output by the first model at a moment, and the network device receives information of the specified threshold reported by the terminal device.

[0329] In some embodiments, there is a mapping relationship between one or more thresholds in the threshold information and one or more indicators in the indicator information.

[0330] In some embodiments, the threshold information and the indicator information are both configured by the network device; or

[0331] The threshold information is configured by the network device, the terminal device determines the indicator information, and the network device receives the indicator information reported by the terminal device; or

[0332] The indicator information is configured by the network device, the terminal device determines the threshold information, and the network device receives the threshold information reported by the terminal device; or

[0333] The terminal device determines the threshold information and the indicator information, and the network device receives the threshold information and / or the indicator information reported by the terminal device.

[0334] In some embodiments, the operations further include:

[0335] The network device receives the candidate operation sent by the terminal device, and the network device sends indication information to the terminal device based on the candidate operation, wherein the terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, and generates the candidate operation based on the evaluation result; or

[0336] The network device receives the value of the indicator sent by the terminal device and sends indication information to the terminal device, where the indication information is generated according to a result of the network device evaluating the performance of the first model based on the value of the indicator and the threshold.

[0337] In some embodiments, the instruction information is used to instruct the terminal device to perform the following operations:

[0338] The first model is switched to a third model, or the first model is deactivated, or a fourth model is selected, or the first model is activated, and the third model and the fourth model are each different from the first model.

[0339] In some embodiments, the third model is an artificial intelligence model or a non-artificial intelligence model, and the fourth model is an artificial intelligence model.

[0340] In some embodiments, the network device receives the information sent by the terminal device through at least one of the following messages and / or resources:

[0341] Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

[0342] In some embodiments, the operations further include:

[0343] The network device sends a signaling to the terminal device to activate the terminal device to monitor the performance of the first model; or

[0344] The network device receives request information sent by the terminal device for requesting to monitor the performance of the first model.

[0345] In some embodiments, the signaling is media access control element (MAC CE) signaling and / or downlink control information (DCI) signaling; or

[0346] The request information is carried by at least one of the following messages and / or resources:

[0347] Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

[0348] Embodiments of the fifth aspect

[0349] An embodiment of the fifth aspect of the present application provides a communication system, which may include a network device and a terminal device.

[0350] FIG7 is a schematic diagram of a terminal device according to an embodiment of the fifth aspect. As shown in FIG7 , the terminal device 700 (e.g., corresponding to the terminal device 202 in FIG2 ) may include a processor 710 and a memory 720; the memory 720 stores data and programs and is coupled to the processor 710. It should be noted that this diagram is exemplary; other types of structures may be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0351] For example, the processor 710 may be configured to execute a program to implement the method according to the second embodiment.

[0352] As shown in Figure 7 , the terminal device 700 may further include: a communication module 730, an input unit 740, a display 750, and a power supply 760. The functions of these components are similar to those in the prior art and are not described in detail here. It is worth noting that the terminal device 700 does not necessarily include all of the components shown in Figure 7 , and these components are not essential. Furthermore, the terminal device 700 may also include components not shown in Figure 7 , for which reference may be made to the prior art.

[0353] FIG8 is a schematic diagram of a network device according to an embodiment of the fifth aspect. As shown in FIG8 , network device 800 (e.g., corresponding to network device 201 in FIG2 ) may include a processor 810 (e.g., a central processing unit (CPU)) and a memory 820; the memory 820 is coupled to the processor 810. The memory 820 may store various data and may also store an information processing program 830, which is executed under the control of the processor 88.

[0354] For example, the processor 88 may be configured to execute a program to implement the method described in the embodiment of the first aspect.

[0355] In addition, as shown in FIG8 , network device 800 may further include: a transceiver 840 and an antenna 850, etc.; wherein, the functions of the above components are similar to those in the prior art and are not described in detail here. It is worth noting that network device 800 does not necessarily include all the components shown in FIG8 ; in addition, network device 800 may also include components not shown in FIG8 , and reference may be made to the prior art for details.

[0356] An embodiment of the present application further provides a computer program, wherein when the program is executed in a terminal device, the program enables the terminal device to execute the method described in the above embodiment.

[0357] An embodiment of the present application further provides a storage medium storing a computer program, wherein the computer program enables a terminal device to execute the method described in the above embodiment.

[0358] An embodiment of the present application further provides a computer program, wherein when the program is executed in a network device, the program enables the network device to execute the method described in the above embodiment.

[0359] An embodiment of the present application further provides a storage medium storing a computer program, wherein the computer program enables a network device to execute the method described in the above embodiment.

[0360] The above devices and methods of the present application can be implemented by hardware or by a combination of hardware and software. The present application relates to such a computer-readable program that, when executed by a logic component, enables the logic component to implement the devices or components described above, or enables the logic component to implement the various methods or steps described above. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0361] The method / device described in conjunction with the embodiments of the present application can be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figure and / or one or more combinations of functional block diagrams can correspond to various software modules of the computer program flow or to various hardware modules. These software modules can respectively correspond to the various steps shown in the figure. These hardware modules can be implemented by solidifying these software modules, for example, using a field programmable gate array (FPGA).

[0362] The software module may be located in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in the memory of the mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the device (such as a mobile terminal) uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0363] One or more of the functional blocks and / or one or more combinations of functional blocks described in the accompanying drawings may be implemented as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any appropriate combination thereof for performing the functions described in this application. One or more of the functional blocks and / or one or more combinations of functional blocks described in the accompanying drawings may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0364] The present application has been described above in conjunction with specific embodiments. However, those skilled in the art should understand that these descriptions are merely illustrative and are not intended to limit the scope of protection of the present application. Those skilled in the art may make various modifications and variations to the present application based on the spirit and principles of the present application, and such modifications and variations are also within the scope of the present application.

[0365] Regarding the implementation methods including the above embodiments, the following additional notes are also disclosed:

[0366] 1. A method for monitoring model performance, applied to a network device, comprising:

[0367] The network device sends at least a first part of the first information to the terminal device, and / or the network device receives at least a second part of the first information reported by the terminal device,

[0368] The first information includes information on indicators and / or thresholds used to monitor the performance of the first model, and the first model is an artificial intelligence model.

[0369] The information input by the first model includes first channel information at least one first moment, and the information output by the first model includes second channel information at least one second moment. The first channel information is information of the spatial channel estimated by the terminal device through the channel state information reference signal (CSI-RS) received by the terminal device, and the second channel information is information of the spatial channel output by the first model.

[0370] 2. The method as described in Note 1, wherein:

[0371] The second time is not earlier than the first time.

[0372] 3. The method as described in Note 1, wherein:

[0373] The information of the indicator is an index or a name, and the information of the threshold is an index or a value.

[0374] 4. The method as described in Note 1, wherein:

[0375] At the same moment,

[0376] The number of spatial domain layers is one or more, and a normalized mean square error (NMSE) between the second channel information and the first channel information is used as the indicator; or

[0377] The number of spatial domain layers is one, and the squared generalized cosine similarity (SGCS) between the second channel information and the first channel information is used as the indicator; or

[0378] If the number of spatial domain layers is more than one, the squared generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator corresponding to the spatial domain layer, or the arithmetic average or weighted average of the squared generalized cosine similarity (SGCS) of each spatial domain layer of the second channel information and the first channel information is used as the indicator.

[0379] The weight of the squared generalized cosine similarity (SGCS) of each spatial layer is related to the singular value corresponding to the spatial layer.

[0380] 5. The method as described in Note 1, wherein:

[0381] There is a mapping relationship between the types of indicators and application scenarios.

[0382] 6. The method as described in Note 5, wherein:

[0383] The mapping relationship is specified by a standard, and / or agreed in advance, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0384] 7. The method as described in Note 1, wherein:

[0385] The information of the indicator is specified by a standard, and / or pre-agreed, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

[0386] 8. The method as described in Supplementary Note 1, wherein the method further comprises:

[0387] The network device receives the candidate operation sent by the terminal device, and the network device sends indication information to the terminal device based on the candidate operation, wherein the terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, and generates the candidate operation based on the evaluation result; or

[0388] The network device receives the value of the indicator sent by the terminal device and sends indication information to the terminal device, where the indication information is generated according to a result of the network device evaluating the performance of the first model based on the value of the indicator and the threshold.

[0389] 9. The method as described in Supplementary Note 8, wherein:

[0390] The instruction information is used to instruct the terminal device to perform an operation:

[0391] The first model is switched to a third model, or the first model is deactivated, or a fourth model is selected, or the first model is activated, and the third model and the fourth model are each different from the first model.

[0392] 10. The method as described in Supplementary Note 9, wherein:

[0393] The third model is an artificial intelligence model or a non-artificial intelligence model, and the fourth model is an artificial intelligence model.

Claims

1. A device for monitoring model performance, applied to a terminal device, comprising a first processing unit configured to control the terminal device to perform the following operations: receiving at least a first part of first information sent by a network device, and / or the terminal device reporting at least a second part of the first information to the network device, in, The first information includes information about indicators and / or thresholds used when monitoring the performance of a first model, and the first model is an artificial intelligence model.

2. The device according to claim 1, wherein At least the third part of the first information is specified by a standard or pre-agreed upon, At least a third portion of the first information is different from or at least partially identical to at least the first portion of the first information, At least a third portion of the first information is different from or at least partially identical to at least a second portion of the first information, At least a first portion of the first information is different from or at least partially identical to at least a second portion of the first information.

3. The device according to claim 1, wherein The information input by the first model includes first channel information at least one first moment, and the information output by the first model includes second channel information at least one second moment. The first channel information is information of the spatial channel estimated by the terminal device through the channel state information reference signal (CSI-RS) received by the terminal device, and the second channel information is information of the spatial channel output by the first model.

4. The device according to claim 3, wherein At the same moment, The number of spatial domain layers is one or more, and a normalized mean square error (NMSE) between the second channel information and the first channel information is used as the indicator; or The number of spatial domain layers is one, and the squared generalized cosine similarity (SGCS) between the second channel information and the first channel information is used as the indicator; or The number of spatial domain layers is more than one, and the second channel information and the first channel information of each spatial domain layer are The squared generalized cosine similarity (SGCS) is used as the indicator corresponding to the spatial layer, or the arithmetic average or weighted average of the squared generalized cosine similarities (SGCS) of each spatial layer of the second channel information and the first channel information is used as the indicator.

5. The device according to claim 3, wherein The normalized mean square error (NMSE) between the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or The normalized mean square error (NMSE) between the second channel information at the latest moment among the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or The normalized mean square error (NMSE) between the second channel information at a specified moment among the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or The first model outputs information at more than one time instant, and the arithmetic average or weighted average of the normalized mean square error (NMSE) of the second channel information and the first channel information corresponding to the more than one time instant is used as the indicator.

6. The device according to claim 3, wherein The same airspace layer, The squared generalized cosine similarity (SGCS) between the second channel information at the earliest moment among the information at more than one moment output by the first model and the first channel information at the earliest moment is used as the indicator; or The squared generalized cosine similarity (SGCS) between the second channel information at the latest moment in the information at more than one moment output by the first model and the first channel information at the latest moment is used as the indicator; or The squared generalized cosine similarity (SGCS) between the second channel information at a specified moment in the information at one or more moments output by the first model and the first channel information at the specified moment is used as the indicator; or The first model outputs information at more than one time instant, the second channel information and the first channel information The arithmetic average or weighted average of the squared generalized cosine similarities (SGCS) corresponding to the more than one time points is used as the indicator.

7. The device according to claim 1, wherein The threshold or threshold information is used to evaluate the performance of the first model.

8. The device according to claim 1, wherein The threshold or threshold information is used to evaluate the comparison result between the first model performance and the second model performance.

9. The device according to claim 8, wherein The second model is an artificial intelligence model or a non-artificial intelligence model, The first model is different from the second model.

10. The device of claim 1, wherein There is more than one threshold.

11. The device according to claim 10, wherein There is a mapping relationship between the one or more thresholds and indicators corresponding to the information output by the first model at one or more moments.

12. The device according to claim 11, wherein The mapping relationship is specified by a standard, and / or agreed in advance, and / or configured by the network device, and / or set by the terminal device and reported to the network device.

13. The device of claim 12, wherein: The network device configures a threshold for an indicator corresponding to information output by the first model at a moment; or The network device configures a threshold selected from one or more candidate thresholds preset or specified by a standard as the threshold for an indicator corresponding to information output by the first model at a moment; or The terminal device selects a threshold as the threshold from one or more candidate thresholds that are preset or specified by a standard and / or configured by the network device for an indicator corresponding to information output by the first model at a moment, and reports information of the selected threshold to the network device; or The terminal device specifies a threshold as the threshold for an indicator corresponding to information output by the first model at a moment, and reports information of the specified threshold to the network device.

14. The device of claim 1, wherein: There is a mapping relationship between one or more thresholds in the threshold information and one or more indicators in the indicator information.

15. The apparatus of claim 1, wherein: The operations further include: The terminal device calculates the value of the indicator based on the information of the indicator and using the information output by the first model.

16. The apparatus of claim 15, wherein: The operations further include: The terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, and performs an operation based on the evaluation result; or The terminal device evaluates the performance of the first model based on the value of the indicator and the threshold, generates candidate operations based on the evaluation result, sends the candidate operations to the network device, and receives indication information of the network device based on the candidate operations; or The terminal device sends the value of the indicator to the network device and receives indication information from the network device, where the indication information is generated according to a result of the network device evaluating the performance of the first model based on the value of the indicator and the threshold.

17. The apparatus of claim 15, wherein: The information sent by the terminal device to the network device is carried by at least one of the following messages and / or resources: Terminal equipment auxiliary information (UAI, UE assistant information), radio resource control (RRC) message, media access control control element (MAC CE) signaling, physical uplink control channel resources similar to scheduling request (PUCCH SR-like resource).

18. The apparatus of claim 1, wherein: The operations further include: The terminal device receives a signaling sent by the network device to activate the terminal device to monitor the performance of the first model; or The terminal device sends a request message to the network device to request monitoring of the performance of the first model.

19. The apparatus of claim 18, wherein: The signaling is media access control element (MAC CE) signaling and / or downlink control information (DCI) signaling; or The request information is carried by at least one of the following messages and / or resources: UE assistant information (UAI), radio resource control (RRC) message information, media access control element (MAC CE) signaling, and physical uplink control channel resources (PUCCH SR-like resources) similar to scheduling requests.

20. A device for monitoring model performance, applied to a network device, the device comprising a second processing unit configured to control the network device to perform the following operations: sending at least a first part of the first information to the terminal device, and / or, the network device receiving at least a second part of the first information reported by the terminal device, in, The first information includes information about indicators and / or thresholds used when monitoring the performance of a first model, and the first model is an artificial intelligence model.

Images