Model performance monitoring method, device, communication system, communication device, and storage medium

Abstract

The present disclosure relates to the technical field of communications, and relates to a model performance monitoring method, a device, a communication system, a communication device, and a storage medium. The method comprises: a first device receives a first request sent by a second device, the first request being used for requesting the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and performance monitoring comprising determining a performance index and / or performance monitoring result; and the first device performs performance monitoring on the first model on the basis of the first request sent by the second device to obtain the performance index and / or performance monitoring result of the first model. Signaling enhancement for model performance monitoring is implemented.

Description

Model performance monitoring method and device, communication system, communication device, storage medium Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a model performance monitoring method and device, a communication system, a communication device, and a storage medium. Background Art

[0002] Artificial Intelligence (AI) / Machine Learning (ML), as an important component of 5G communication technology, is becoming increasingly important in the research and application of 5G or 6G communication standards.

[0003] Summary of the Invention

[0004] The embodiments of the present disclosure provide a model performance monitoring method and device, a communication system, a communication device, and a storage medium, which can be used in the field of communication technology to implement signaling enhancement for model performance monitoring.

[0005] According to a first aspect of an embodiment of the present disclosure, a model performance monitoring method is proposed, which is executed by a first device and includes: receiving a first request sent by a second device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring including determining performance indicators and / or performance monitoring results.

[0006] According to the second aspect of an embodiment of the present disclosure, a model performance monitoring method is proposed, which is executed by a second device, including: sending a first request to a first device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring including determining performance indicators and / or performance monitoring results.

[0007] According to the third aspect of an embodiment of the present disclosure, a first device is proposed, including a transceiver module for receiving a first request sent by a second device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring including determining performance indicators and / or performance monitoring results.

[0008] According to the fourth aspect of an embodiment of the present disclosure, a second device is proposed, including a transceiver module, for sending a first request to a first device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring including determining performance indicators and / or performance monitoring results.

[0009] According to a fifth aspect of an embodiment of the present disclosure, a communication device is proposed, comprising one or more processors; wherein the one or more processors are used to call instructions so that the communication device executes the method described in any one of the first and second aspects.

[0010] According to the sixth aspect of an embodiment of the present disclosure, a communication system is proposed, comprising a first device and a second device, wherein the first device is configured to implement the model performance monitoring method of the first aspect, and the second device is configured to implement the model performance monitoring method of the second aspect.

[0011] According to a seventh aspect of an embodiment of the present disclosure, a storage medium is proposed, wherein the storage medium stores instructions. When the instructions are executed on a communication device, the communication device executes any one of the methods of the first and second aspects.

[0012] According to the model performance monitoring method proposed in this disclosure, a second device sends a first request to a first device, requesting the first device to monitor the performance of a first model deployed on the first device to obtain performance indicators. This signaling enhancement addresses the performance monitoring issue of AI positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following drawings required for describing the embodiments are introduced. The following drawings are merely some embodiments of the present disclosure and do not impose specific limitations on the protection scope of the present disclosure.

[0014] FIG1A is a schematic diagram of the functional architecture of AI / ML according to an embodiment of the present disclosure;

[0015] FIG1B is a schematic diagram of the architecture of a communication system provided according to an embodiment of the present disclosure;

[0016] FIG2 is an interactive diagram of a model performance monitoring method provided according to an embodiment of the present disclosure;

[0017] FIG3A is a schematic flow chart of a method for monitoring model performance of a first device according to an embodiment of the present disclosure;

[0018] FIG3B is a schematic flow chart of a method for monitoring model performance of a first device according to an embodiment of the present disclosure;

[0019] FIG4A is a schematic flow chart of a method for monitoring model performance of a second device according to an embodiment of the present disclosure;

[0020] FIG4B is a schematic diagram of a flow chart of a method for monitoring model performance of a second device according to an embodiment of the present disclosure;

[0021] FIG5 is an interactive diagram of a model performance monitoring method according to an embodiment of the present disclosure;

[0022] FIG6A is a schematic structural diagram of a first device provided according to an embodiment of the present disclosure;

[0023] FIG6B is a schematic structural diagram of a second device provided according to an embodiment of the present disclosure;

[0024] FIG7A is a schematic structural diagram of a communication device according to an embodiment of the present disclosure;

[0025] FIG7B is a schematic diagram of the structure of the chip proposed in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The embodiments of the present disclosure provide a model performance monitoring method and device, a communication system, a communication device, and a storage medium.

[0027] In a first aspect, an embodiment of the present disclosure provides a model performance monitoring method, which is executed by a first device and includes: receiving a first request sent by a second device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring including determining performance indicators and / or performance monitoring results.

[0028] In the above embodiment, the first device can perform performance monitoring on the first model based on the request sent by the second device to obtain the first model performance indicator and / or performance monitoring result, thereby achieving signaling enhancement for performance monitoring of the model.

[0029] In combination with some embodiments of the first aspect, in some embodiments, the method further includes receiving a second request sent by a second device, where the second request is used to indicate information used by the first device to determine performance indicators and / or performance monitoring results.

[0030] In conjunction with some embodiments of the first aspect, in some embodiments, the information includes input parameters and / or output parameters of the first model.

[0031] In the above embodiment, through the second request sent by the second device to the first device, the first device can determine which parameter to use for performance monitoring based on the first request and the second request to obtain performance monitoring indicators and / or performance monitoring results.

[0032] In combination with some embodiments of the first aspect, in some embodiments, the method further includes: receiving first indication information sent by the second device, the first indication information being used to indicate a threshold for determining a performance monitoring result, the first indication information including at least one of the following: a first threshold, the first threshold being used to evaluate a first performance indicator, the first performance indicator being obtained by comparing an input parameter with input data in the training data of the first model; a second threshold, the second threshold being used to evaluate a second performance indicator, the second performance indicator being obtained by comparing a current input parameter with a previous input parameter; a third threshold, the third threshold being used to evaluate a third performance indicator, the third performance indicator being obtained by comparing a statistical value of an output parameter with a statistical value of output data in the training data of the first model; a fourth threshold, the fourth threshold being used to evaluate a fourth performance indicator, the fourth performance indicator being obtained by comparing a current output parameter with a previous output parameter; a fifth threshold, the fifth threshold being used to evaluate a fifth performance indicator, the fifth performance indicator being obtained by comparing a statistical value of the output parameter with a statistical value of the output data in the training data of the first model; and a sixth threshold, the sixth threshold being used to evaluate a sixth performance indicator, the sixth performance indicator being obtained by comparing the output parameter with a result obtained by a preset positioning method.

[0033] In the above embodiment, through the first indication information sent by the second device to the first device, the first device can determine the performance monitoring result of the first model based on the first request and the first indication information, or determine the performance monitoring result of the first model based on the first request, the second request and the first indication information.

[0034] In combination with some embodiments of the first aspect, in some embodiments, the method further includes: receiving second indication information sent by the second device, where the second indication information is used to assist the first device in performance monitoring.

[0035] In combination with some embodiments of the first aspect, in some embodiments, the second indication information includes the geographic reality label of the first terminal and / or the statistical value of the training data of the first model, and the geographic reality label includes at least one of the following: the measurement value of the first terminal; the location of the first terminal; the timestamp corresponding to the location of the first terminal; and the quality indication of the measurement value of the first terminal.

[0036] In the above embodiment, the second device sends the measurement value, timestamp, quality of the measurement value and actual location obtained by positioning other terminals through the first model to the first device, so as to assist the first device in monitoring the performance of the first model to obtain performance indicators and / or performance monitoring results.

[0037] In combination with some embodiments of the first aspect, in some embodiments, the method further includes performing performance monitoring on the first model based on at least one of the first request, the second request, the first indication information, and the second indication information.

[0038] In the above embodiment, the first device can perform performance monitoring on the first model based on the first request or the second request or the first indication information or the second indication information, or can perform performance monitoring on the first model based on any two or any more of them to obtain performance indicators and / or performance monitoring results of the first model.

[0039] In combination with some embodiments of the first aspect, in some embodiments, based on at least one of the first request, the second request, the first indication information, and the second indication information, performance monitoring of the first model includes at least one of the following: comparing the input parameters of the first model with the input data in the training data of the first model; comparing the statistical values ​​of the input parameters of the first model with the statistical values ​​of the input data in the training data of the first model; comparing the current input parameters of the first model with the previous input parameters of the first model; comparing the statistical values ​​of the output parameters of the first model with the statistical values ​​of the output data in the training data of the first model; comparing the current output parameters of the first model with the previous output parameters of the first model; comparing the statistical values ​​of the current output parameters of the first model with the statistical values ​​of the previous output parameters of the first model; and comparing the output parameters of the first model with the results obtained by a preset positioning method.

[0040] In the above embodiment, the performance monitoring method for the first model is determined by different parameters, that is, different parameters correspond to different comparison parameters, and performance monitoring indicators and performance monitoring results based on the parameters can be obtained.

[0041] In combination with some embodiments of the first aspect, in some embodiments, the method further includes receiving third indication information sent by the second device, where the third indication information is used to indicate a preset positioning method.

[0042] In the above embodiment, the performance monitoring index of the first model can be obtained by instructing a preset positioning method to position the first model, outputting the result of the preset positioning method, and comparing it with the output parameters of the first model.

[0043] In combination with some embodiments of the first aspect, in some embodiments, the input parameters of the first model include channel measurement-related parameters and / or reference signal measurement results, and the channel measurement-related parameters and / or reference signal measurement results include at least one of the following: channel impulse response CIR; power delay spectrum PDP; delay characteristic DP; signal to interference and noise ratio PRS-SINR of positioning reference signal; reference signal received power PRS-RSRP of positioning reference signal; reference signal received power SSB-RSRP of synchronization signal block; signal to interference and noise ratio SSB-SI NR of synchronization signal block; reference signal time difference RSTD; UE receive transmit time difference UERx-Tx time difference; gNB receive transmit time difference gNB Rx-Tx time difference; signal to interference and noise ratio SRS-SINR of uplink positioning reference signal; uplink positioning reference signal received power SR S-RSRP.

[0044] In combination with some embodiments of the first aspect, in some embodiments, the output parameter includes at least one of the following: terminal position; L os indication; NLoS indication; flight time; measurement quantity based on positioning reference signal.

[0045] In combination with some embodiments of the first aspect, in some embodiments, the method further includes receiving a third request sent by the second device, where the third request is used to request the first device to send performance indicators and / or performance monitoring results.

[0046] In combination with some embodiments of the first aspect, in some embodiments, the method further includes: sending the performance indicator and / or performance monitoring result to the second device based on the third request.

[0047] In the above embodiment, the first device monitors the performance of the first model and can obtain performance monitoring indicators and / or performance monitoring results, and can also send the performance indicators and / or performance monitoring results to the second device at the request of the second device.

[0048] In combination with some embodiments of the first aspect, in some embodiments, the method further includes: sending fourth indication information to the second device, where the fourth indication information is used to instruct the second device to determine the performance monitoring result of the first model based on the performance indicator.

[0049] In the above embodiment, the second device obtains the performance monitoring result based on the received performance indicator according to the fourth indication information sent by the first device.

[0050] In combination with some embodiments of the first aspect, in some embodiments, the method further includes: sending fifth indication information to the second device, where the fifth indication information is used to indicate a statistical value of the training data of the first model.

[0051] In the above embodiment, the first device sends the statistical value of the training data of the first model to the second device, so that the second device can obtain the performance monitoring result of the first model.

[0052] In combination with some embodiments of the first aspect, in some embodiments, the performance indicator includes at least one of the following: channel impulse response CIR; power delay profile PDP; delay characteristic DP; signal to interference plus noise ratio PRS-SINR of the positioning reference signal; reference signal received power PRS-RSRP of the positioning reference signal; reference signal received power SSB-RSRP of the synchronization signal block; signal to interference plus noise ratio SSB-SINR of the synchronization signal block; reference signal time difference RSTD; UE receive transmit time difference UERx-Tx time difference; gNB receive transmit time difference gNB Rx-Tx time difference.; signal to interference plus noise ratio SRS-SINR of the uplink positioning reference signal; uplink positioning reference signal received power SRS-RSRP; terminal position obtained by the first model; terminal position obtained by the preset positioning method; flight time.

[0053] In combination with some embodiments of the first aspect, in some embodiments, the performance monitoring results include at least one of the following: the first model is no longer applicable; the second model is expected to be applicable; a model update indication; a performance indicator for determining that the first model is no longer applicable; and a method for model performance monitoring.

[0054] In combination with some embodiments of the first aspect, in some embodiments, the first model includes at least one of the following: a model currently used by the first device; a model deployed on the first device and not yet used; a model running in the first device; a model deployed on the first device and not yet running.

[0055] In the above embodiment, the first device can perform performance monitoring on the first model based on the request or instruction information and request sent by the second device. Different parameters can be used for performance monitoring to obtain different performance indicators. The performance indicators can be judged based on the thresholds sent by the second device to obtain the performance monitoring results of the first model. Alternatively, the first device can send the performance indicators to the second device and instruct the second device to judge the performance results. This achieves signaling enhancement for performance monitoring of the model.

[0056] In a second aspect, an embodiment of the present disclosure provides a model performance monitoring method, which is executed by a second device and includes: sending a first request to a first device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring including determining performance indicators and / or performance monitoring results.

[0057] In combination with some embodiments of the second aspect, in some embodiments, the method further includes sending a second request to the first device, where the second request is used to indicate information used by the first device when determining performance indicators and / or performance monitoring results.

[0058] In conjunction with some embodiments of the second aspect, in some embodiments, the information includes input parameters and / or output parameters of the first model.

[0059] In combination with some embodiments of the second aspect, in some embodiments, the method also includes sending first indication information to the first device, the first indication information is used to indicate a threshold for determining a performance monitoring result, and the first indication information includes at least one of the following: a first threshold, the first threshold is used to evaluate a first performance indicator, and the first performance indicator is obtained by comparing the input parameter with the input data in the training data of the first model; a second threshold, the second threshold is used to evaluate a second performance indicator, and the second performance indicator is obtained by comparing the current input parameter with the previous input parameter; a third threshold, the third threshold is used to evaluate a third performance indicator, and the third performance indicator is obtained by comparing the statistical value of the output parameter with the statistical value of the output data in the training data of the first model; a fourth threshold, the fourth threshold is used to evaluate a fourth performance indicator, and the fourth performance indicator is obtained by comparing the current output parameter with the previous output parameter; a fifth threshold, the fifth threshold is used to evaluate a fifth performance indicator, and the fifth performance indicator is obtained by comparing the statistical value of the output parameter with the statistical value of the output data in the training data of the first model; a sixth threshold, the sixth threshold is used to evaluate a sixth performance indicator, and the sixth performance indicator is obtained by comparing the output parameter with the result obtained by a preset positioning method.

[0060] In combination with some embodiments of the second aspect, in some embodiments, the method further includes: sending second indication information to the first device, where the second indication information is used to assist the first device in performance monitoring.

[0061] In conjunction with some embodiments of the second aspect, in some embodiments, the second indication information includes a geographic ground truth tag of the first terminal and / or statistical values ​​of the training data of the first model, where the geographic ground truth tag includes at least one of the following: a measurement value of the first terminal; the location of the first terminal; a timestamp corresponding to the location of the first terminal; or a quality indication of the measurement value of the first terminal. In conjunction with some embodiments of the second aspect, in some embodiments, the method further includes sending third indication information to the first device, where the third indication information is used to indicate a preset positioning method.

[0062] In combination with some embodiments of the second aspect, in some embodiments, the input parameters of the first model include channel measurement-related parameters and / or reference signal measurement results, and the channel measurement-related parameters and / or reference signal measurement results include at least one of the following: channel impulse response CIR; power delay spectrum PDP; delay characteristics DP; signal to interference plus noise ratio PRS-SINR of positioning reference signal; reference signal received power PRS-RSRP of positioning reference signal; reference signal received power SSB-RSRP of synchronization signal block; signal to interference plus noise ratio SSB-SI NR of synchronization signal block; reference signal time difference RSTD; UE receive transmit time difference UERx-Tx time difference; gNB receive transmit time difference gNB Rx-Tx time difference; signal to interference plus noise ratio SRS-SINR of uplink positioning reference signal; uplink positioning reference signal received power SR S-RSRP.

[0063] In combination with some embodiments of the second aspect, in some embodiments, the output parameter includes at least one of the following: terminal position; L os indication; NLoS indication; flight time; measurement value based on positioning reference signal.

[0064] In combination with some embodiments of the second aspect, in some embodiments, the method further includes: sending a third request to the first device, where the third request is used to request the first device to send performance indicators and / or performance monitoring results.

[0065] In combination with some embodiments of the second aspect, in some embodiments, the method further includes: receiving performance indicators and / or performance monitoring results sent by the first device.

[0066] In combination with some embodiments of the second aspect, in some embodiments, the method also includes: receiving fourth indication information sent by the first device, the fourth indication information being used to instruct the second device to determine the performance monitoring result of the first model based on the performance indicator; and determining the performance monitoring result based on the fourth indication information.

[0067] In combination with some embodiments of the second aspect, in some embodiments, the method further includes: receiving fifth indication information sent by the first device, where the fifth indication information is used to indicate a statistical value of the training data of the first model.

[0068] In combination with some embodiments of the second aspect, in some embodiments, the performance indicator includes at least one of the following: channel impulse response CIR; power delay spectrum PDP; delay characteristic DP; signal to interference plus noise ratio PRS-SINR of the positioning reference signal; reference signal received power PRS-RSRP of the positioning reference signal; reference signal received power SSB-RSRP of the synchronization signal block; signal to interference plus noise ratio SSB-SINR of the synchronization signal block; reference signal time difference RSTD; UE receive transmit time difference UERx-Tx time difference; gNB receive transmit time difference gNB Rx-Tx time difference; signal to interference plus noise ratio SRS-SINR of the uplink positioning reference signal; uplink positioning reference signal received power S RS-RSRP; terminal position obtained by the first model; terminal position obtained by the preset positioning method; flight time.

[0069] In combination with some embodiments of the second aspect, in some embodiments, the performance monitoring results include at least one of the following: the first model is no longer applicable; the expectation of applying the second model; a model update indication; a performance indicator for determining that the first model is no longer applicable; and a method for model monitoring.

[0070] In combination with some embodiments of the second aspect, in some embodiments, the first model includes at least one of the following: a model currently used by the first device; a model deployed on the first device and not yet used; a model running in the first device; a model deployed on the first device and not yet running.

[0071] In the above embodiment, the second device sends a request for performance monitoring to the first device, and sends information, auxiliary information and thresholds for performance monitoring through indication information for the first device to perform performance monitoring. The second device can also infer the performance monitoring results based on the performance indicators sent by the first device to obtain the performance monitoring results, thereby realizing signaling enhancement of model performance monitoring.

[0072] In a third aspect, an embodiment of the present disclosure provides a first device, including: a transceiver module, the transceiver module is used to receive a first request sent by a second device, the first request is used to request the first device to perform performance monitoring on a first model, the first model is deployed on the first device, and the performance monitoring includes determining performance indicators and / or performance monitoring results.

[0073] In a fourth aspect, an embodiment of the present disclosure provides a second device, including: a transceiver module, the transceiver module is used to send a first request to a first device, the first request is used to request the first device to perform performance monitoring on a first model, the first model is deployed on the first device, and the performance monitoring includes determining performance indicators and / or performance monitoring results.

[0074] In a fifth aspect, an embodiment of the present disclosure provides a communication device, comprising: one or more processors; wherein the one or more processors are used to call instructions so that the communication device executes the method described in any one of the embodiments of the first and second aspects.

[0075] In the sixth aspect, an embodiment of the present disclosure provides a communication system, comprising: a first device and a second device, wherein the first device is used to execute the method described in any one of the embodiments in the first aspect of the present disclosure; the second device is used to execute the method described in any one of the embodiments in the second aspect of the present disclosure.

[0076] In a seventh aspect, an embodiment of the present disclosure provides a storage medium storing instructions. When the instructions are executed on a communication device, the communication device executes the method described in any one of the embodiments of the first and second aspects of the present disclosure.

[0077] In an eighth aspect, an embodiment of the present disclosure proposes a program product. When the program product is executed by a communication device, the communication device executes the method described in the optional implementation of the first and second aspects.

[0078] In a ninth aspect, an embodiment of the present disclosure proposes a computer program, which, when executed on a computer, enables the computer to execute the method described in the optional implementation of the first and second aspects.

[0079] In a tenth aspect, an embodiment of the present disclosure provides a chip or a chip system, which includes a processing circuit configured to execute the method described in the optional implementation of the first and second aspects above.

[0080] It is understandable that the first device, the second device, the communication system, the storage medium, the program product, the computer program, the chip, or the chip system described above are all used to perform the method proposed in the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding methods and will not be repeated here.

[0081] The embodiments of the present disclosure provide a model performance monitoring method and device, a communication system, a communication device, and a storage medium.

[0082] In some embodiments, terms such as model performance monitoring method and information processing method can be replaced with each other, terms such as first device, second device and information processing device, communication device can be replaced with each other, and terms such as information processing system, communication system can be replaced with each other.

[0083] The embodiments of the present disclosure are not exhaustive and are merely illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined. For example, some or all steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0084] In each embodiment of the present disclosure, unless otherwise specified or provided for by logic, the terms and / or descriptions between the embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form a new embodiment based on their inherent logical relationships.

[0085] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.

[0086] In the embodiments of the present disclosure, unless otherwise specified, elements expressed in the singular, such as "a", "an", "the", "the", "the", etc., can mean "one and only one", or "one or more", "at least one", etc. For example, when using articles such as "a", "an", "the" in English in translation, the noun following the article can be understood as a singular expression or a plural expression.

[0087] In the embodiments of the present disclosure, “plurality” refers to two or more.

[0088] In some embodiments, the terms "at least one of", "at least one of", "at least one of", "one or more", "a plurality of", "multiple", etc. can be used interchangeably.

[0089] In the embodiments of the present disclosure, descriptions such as “at least one of A, B, C…”, “A and / or B and / or C…”, etc. include the situation where any one of A, B, C… exists alone, and also include any combination of any multiple of A, B, C…, and each situation can exist alone; for example, “at least one of A, B, C” includes the situation where A exists alone, B exists alone, C exists alone, the combination of A and B, the combination of A and C, the combination of B and C, and the combination of A, B, and C; for example, A and / or B includes the situation where A exists alone, B exists alone, and the combination of A and B.

[0090] In some embodiments, descriptions such as "in one case A, in another case B," or "in response to one case A, in response to another case B," may include the following technical solutions depending on the situation: executing A independently of B (in some embodiments, A); executing B independently of A (in some embodiments, B); selectively executing A and B (in some embodiments, selecting between A and B); and executing both A and B (in some embodiments, A and B). The same applies when there are more branches, such as A, B, and C.

[0091] The prefixes such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different description objects and do not constitute any restriction on the position, order, priority, quantity or content of the description objects. For the statement of the description object, please refer to the description in the context of the claims or embodiments, and no unnecessary restriction should be constituted due to the use of prefixes. For example, if the description object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields". "First" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the description object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of description objects is not limited by the ordinal number and can be one or more. Taking "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes can be the same or different. For example, if the description object is "device", then the "first device" and the "second device" can be the same device or different devices, and their types can be the same or different; for another example, if the description object is "information", then the "first information" and the "second information" can be the same information or different information, and their contents can be the same or different.

[0092] In some embodiments, “including A,” “comprising A,” “used to indicate A,” and “carrying A” can be interpreted as directly carrying A or indirectly indicating A.

[0093] In some embodiments, terms such as "time / frequency" and "time / frequency domain" refer to the time domain and / or the frequency domain.

[0094] In some embodiments, terms such as "in response to...", "in response to determining...", "in the case of...", "at the time of...", "when...", "if...", "if...", etc. can be used interchangeably.

[0095] In some embodiments, terms such as "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not less than", and "above" can be replaced with each other, and terms such as "less than", "less than or equal to", "not greater than", "less than", "less than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", and "below" can be replaced with each other.

[0096] In some embodiments, devices, etc. can be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. Terms such as "device", "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", and "subject" can be used interchangeably.

[0097] In some embodiments, "network" can be interpreted as devices included in the network (eg, access network equipment, core network equipment, etc.).

[0098] In some embodiments, the terms “access network device (AN device)”, “radio access network device (RAN device)”, “base station (BS)”, “radio base station”, “fixed station”, “node”, “access point”, “transmission point (TP)”, “reception point (RP)”, “transmission / reception point (TRP)”, “panel”, “antenna panel”, “antenna array”, “cell”, “macro cell”, “small cell”, “femto cell”, “pico cell”, “sector”, “cell group”, “carrier”, “component carrier”, “bandwidth part (BWP)” and the like may be used interchangeably.

[0099] In some embodiments, the terms "terminal", "terminal device", "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client and the like may be used interchangeably.

[0100] In some embodiments, the access network device, the core network device, or the network device can be replaced by a terminal. For example, the various embodiments of the present disclosure can also be applied to a structure in which the communication between the access network device, the core network device, or the network device and the terminal is replaced by communication between multiple terminals (for example, device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, it is also possible to set the structure in which the terminal has all or part of the functions of the access network device. In addition, terms such as "uplink" and "downlink" can also be replaced by terms corresponding to communication between terminals (for example, "side"). For example, uplink channels, downlink channels, etc. can be replaced by side channels, and uplinks, downlinks, etc. can be replaced by side links.

[0101] In some embodiments, the terminal may be replaced by an access network device, a core network device, or a network device. In this case, the access network device, the core network device, or the network device may have a structure that has all or part of the functions of the terminal.

[0102] In some embodiments, obtaining data, information, etc. may comply with the laws and regulations of the country where the data is obtained.

[0103] In some embodiments, data, information, etc. may be obtained with the user's consent.

[0104] In addition, each element, each row, or each column in the table of the embodiment of the present disclosure can be implemented as an independent embodiment, and the combination of any elements, any rows, and any columns can also be implemented as an independent embodiment.

[0105] Figure 1A is a functional architecture diagram of AI / ML. As shown in Figure 1A, data collection provides input data to model training, management, and inference modules; model training is the module that performs AI / ML model training, verification, and testing, and can generate model performance indicators as part of the model testing process; management is the function of supervising the operation (e.g., selection / (de)activation / switching / fallback) and supervision (e.g., performance) of AI / ML models or AI / ML functions; inference is the application of artificial intelligence / machine learning models or artificial intelligence / machine learning functions in the process of providing output, using data provided by data collection (i.e., inference data) as input; model storage is responsible for storing trained / updated models, which can be used to perform inference functions. In AI / ML applications, for AI-based positioning, the inference node in Figure 1A is the node where AI functions are deployed. AI positioning mainly includes the following:

[0106] 1. Direct positioning of AI / ML:

[0107] AI / ML model output: The terminal's location. For example, different channel values ​​such as CIR, SNR, or SRP are collected at different points and used as input for the AI ​​model.

[0108] 2. AI / ML output measurement results:

[0109] AI / ML model outputs: New results and / or improvements to existing results.

[0110] For example, LOS / NLOS (line of sight / non-line of sight) identification, timing and / or measurement angles, possibility of measurement.

[0111] More specifically, the present disclosure refers to the following cases:

[0112] Case 1: Device-based positioning using a device-side model, AI / ML direct positioning, or AI / ML-assisted positioning;

[0113] Case 2a: Device-assisted / LMF positioning using the device-side model, and AI / ML-assisted positioning;

[0114] Case 2b: Terminal-assisted / LMF-based positioning, LMF-side model, and AI / ML-based direct positioning;

[0115] Case 3a: NG-RAN node-assisted positioning gNB side model, AI / ML-assisted positioning;

[0116] Case 3b: NG-RAN node assists in positioning the LMF side model, with direct AI / ML positioning.

[0117] For AI positioning, it is necessary to test the performance of AI positioning. As for which node generates performance monitoring indicators, the following conclusions are drawn:

[0118] 1. An entity that obtains performance monitoring indicators. For example, this is the management node in Figure 1A. The monitoring process can be after model training, during model use, or before the model is used, by performing performance monitoring on an unused model to obtain the model's performance monitoring indicators and determine whether the model is usable.

[0119] 2. Terminal: In the above case 1 and case 2a, this is the solution of the terminal-side model.

[0120] 3. gNB, in the above case 3a, that is, the gNB side model solution.

[0121] 4. LMF, in the above cases 2b and 3b, that is, the LMF side model solution.

[0122] For AI / ML based positioning, the LMF for case 2a (with UE side model) and case 3a (with gNB side model) is used as the entity to derive performance monitoring metrics, at least when the monitoring metrics are derived based on the provided ground truth labels (or their approximations).

[0123] For AI models or functions deployed on the device side, gNB side, and LMF side, the following signaling enhancements may be required for performance monitoring:

[0124] 1. For the terminal-side and gNB-side models monitoring AI / ML positioning:

[0125] (1) A signal is sent from the LMF and the monitoring entity obtains the monitoring indicators.

[0126] (2) A signal is sent from the monitoring entity to request measurement results. The monitoring entity here may be the LMF or gNB. That is, the terminal may request the LMF or gNB to provide some measurement results. These measurement results are based on those obtained by other terminals and are used for comparison with the terminal during the positioning process.

[0127] (3) Request to report the monitoring indicators.

[0128] (4) Marking: There may not be any special impact.

[0129] 2. LMF side model for monitoring AI / ML positioning.

[0130] (1) LMF sends a signal to request performance monitoring.

[0131] Although it has been determined that signaling enhancement is required for performance monitoring of AI positioning, the specific signaling process has not yet been determined.

[0132] Therefore, the present disclosure proposes a model performance monitoring method and device, a communication system, a communication device, and a storage medium. A first device receives a first request sent by a second device, requesting the first device to perform performance monitoring on a first model deployed on the first device. Performance monitoring includes determining performance indicators and / or performance monitoring results. This defines the signaling process for AI positioning performance monitoring and achieves signaling enhancement for model performance monitoring.

[0133] The method proposed in the present disclosure is applicable to various communication systems, including but not limited to 4G, 5G, 5G-advance and subsequent communication technologies (such as 6G, etc.).

[0134] In some embodiments, "AI model", "artificial intelligence model", "machine learning (ML) model", "ML model", AI, AI function, model, ML function, artificial intelligence function, and machine learning function can be replaced with each other.

[0135] In some embodiments, “first model”, “first function”, “first AI model”, and “first AI function” can be interchangeable.

[0136] FIG1B is a schematic diagram illustrating an architecture of a communication system according to an embodiment of the present disclosure. As shown in FIG1B , a communication system 100 may include a first device 101 and a second device 102 .

[0137] In some embodiments, the first device 101 may be a device that receives the first request.

[0138] In some embodiments, the first device 101 may be the device that receives the second request.

[0139] In some embodiments, the first device 101 may be a device that receives the first indication information.

[0140] In some embodiments, the first device 101 may be a device that receives the second indication information.

[0141] In some embodiments, the first device 101 may be a device that receives the third indication information.

[0142] In some embodiments, the first device 101 may be a device that receives the third request.

[0143] In some embodiments, the first device 101 may be a device that performs performance monitoring on the first model.

[0144] In some embodiments, the first device 101 may be a device that sends performance indicators and / or performance monitoring results.

[0145] In some embodiments, the first device 101 may be a device that sends the fourth indication information.

[0146] In some embodiments, the first device 101 may be a device that sends the fifth indication information.

[0147] In some embodiments, the first device 101 may be a device on which the first model is deployed.

[0148] In some embodiments, the first device 101 may be a device using a first model.

[0149] In some embodiments, the first device 101 may be a device running a first model.

[0150] In some embodiments, the first device 101 may be a terminal or an access network device, for example, a gNB.

[0151] In some embodiments, the name of the first device 101 is not limited, and it can be, for example, "a device for model performance monitoring", "a device for determining performance monitoring results", or "a device for deploying the first model".

[0152] In some embodiments, the second device 102 may be the device that sends the first request.

[0153] In some embodiments, the second device 102 may be the device that sends the second request.

[0154] In some embodiments, the second device 102 may be a device that sends the first indication information.

[0155] In some embodiments, the second device 102 may be a device that sends the second indication information.

[0156] In some embodiments, the second device 102 may be a device that sends the third indication information.

[0157] In some embodiments, the second device 102 may be the device that sends the third request.

[0158] In some embodiments, the second device 102 may be a device that receives performance indicators and / or performance monitoring results.

[0159] In some embodiments, the second device 102 may be a device that receives the fourth indication information.

[0160] In some embodiments, the second device 102 may be a device that receives the fifth indication information.

[0161] In some embodiments, the second device 102 may be a device that determines the performance monitoring result of the first model.

[0162] In some embodiments, the second device 102 may be a network device, for example, a LMF.

[0163] In some embodiments, the name of the second device 102 is not limited, and it can be, for example, "a receiving device for performance indicators", "a receiving device for performance monitoring results", "a device for model performance monitoring", etc. In some embodiments, the terminal may include at least one of a mobile phone, a wearable device, an Internet of Things device, a car with communication functions, a smart car, a tablet computer, a computer with wireless transceiver functions, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in a smart grid, a wireless terminal device in transportation safety, a wireless terminal device in a smart city, and a wireless terminal device in a smart home, but is not limited thereto.

[0164] In some embodiments, the access network device may include at least one of an evolved NodeB (eNB), a next generation eNB (ng-eNB), a next generation NodeB (gNB), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an open base station (Open RAN), a cloud base station (Cloud RAN), a base station in other communication systems, and an access node in a Wi-Fi system, but is not limited thereto.

[0165] In some embodiments, the technical solution of the present disclosure can be applied to the Open RAN architecture. In this case, the interfaces between or within the access network devices involved in the embodiments of the present disclosure can be transformed into internal interfaces of the Open RAN, and the processes and information interactions between these internal interfaces can be implemented through software or programs.

[0166] In some embodiments, the access network device can be composed of a centralized unit (CU) and a distributed unit (DU), where the CU can also be called a control unit. The CU-DU structure can be used to split the protocol layer of the access network device, with the functions of some protocol layers centrally controlled by the CU, and the functions of the remaining part or all of the protocol layers distributed in the DU, which is centrally controlled by the CU, but is not limited to this.

[0167] In some embodiments, a core network device may be a device including one or more network elements, or may be multiple devices or device groups, each including all or part of the one or more network elements. The network element may be virtual or physical. The core network may include, for example, at least one of an Evolved Packet Core (EPC), a 5G Core Network (5GCN), and a Next Generation Core (NGC).

[0168] It can be understood that the communication system described in the embodiment of the present disclosure is for the purpose of more clearly illustrating the technical solution of the embodiment of the present disclosure, and does not constitute a limitation on the technical solution proposed in the embodiment of the present disclosure. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution proposed in the embodiment of the present disclosure is also applicable to similar technical problems.

[0169] The following embodiments of the present disclosure may be applied to the communication system 100 shown in FIG2 , or a portion thereof, but are not limited thereto. The entities shown in FIG1 are illustrative only. The communication system may include all or part of the entities shown in FIG1 , or may include other entities outside of FIG1 . The number and form of the entities may be arbitrary. The connection relationship between the entities is illustrative only. The entities may be connected or disconnected, and the connection may be in any manner, including direct or indirect, wired or wireless.

[0170] The embodiments of the present disclosure may be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G New Radio (NR), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New Radio Access (NX), Future Generation Radio Access (FX), Global System for Mobile Communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.17 (WiMAX (registered trademark)), IEEE 802.18 (WiMAX (registered trademark)), IEEE 802.19 (WiMAX (registered trademark)), IEEE 802.20 (WiMAX (registered trademark)), IEEE 802.21 (WiMAX (registered trademark)), IEEE 802.22 (WiMAX (registered trademark)), IEEE 802.23 (WiMAX (registered trademark)), IEEE 802.24 (WiMAX (registered trademark)), IEEE 802.25 (WiMAX (registered trademark)), IEEE 802.26 (WiMAX (registered trademark)), IEEE 802.27 (WiMAX (registered trademark)), IEEE 802.28 (WiMAX (registered trademark)), IEEE 802.29 (WiMAX (registered trademark)), IEEE 802.30 (WiMAX (registered trademark)), IEEE 802.31 (WiMAX (registered trademark)), IEEE 802.32 (WiMAX (registered trademark)), IEEE 802.33 (WiMAX 802.20, Ultra-WideBand (UWB), Bluetooth (registered trademark), Public Land Mobile Network (PLMN) networks, Device-to-Device (D2D) systems, Machine-to-Machine (M2M) systems, Internet of Things (IoT) systems, Vehicle-to-Everything (V2X), systems using other user plane path establishment methods, and next-generation systems based on these. Furthermore, multiple systems may be combined (for example, a combination of LTE or LTE-A with 5G).

[0171] Figure 2 is an interactive diagram of a model performance monitoring method provided by an embodiment of the present disclosure. As shown in Figure 2, an embodiment of the present disclosure relates to a model performance monitoring method, which can be executed by a communication system, such as the communication system 100 shown in Figure 1B. The communication system includes a first device and a second device. The interactive method may include the following steps:

[0172] Step 2101: The second device sends a first request to the first device.

[0173] In some embodiments, the first request may be a request for requesting the first device to perform performance monitoring on the first model, wherein the first model is deployed on the first device.

[0174] In some embodiments, the first device receives a first request sent by the second device.

[0175] In some embodiments, the first device may be a terminal or an access network device, such as a gNB; the second device may be a network device, such as a LMF. For example, the LMF sends a first request to the terminal or gNB, requesting the terminal or gNB to perform performance monitoring on the first model.

[0176] In some embodiments, the first model may be a model currently used by the first device.

[0177] In some embodiments, the first model may be a model that is deployed on the first device and has not yet been used.

[0178] In some embodiments, the first model may be a model currently running in the first device.

[0179] In some embodiments, the first model may be a model that is deployed on the first device and has not yet been run.

[0180] In some embodiments, performance monitoring includes determining a performance indicator and / or a performance monitoring result of the first model.

[0181] In some embodiments, based on the first request, the first device performs performance monitoring on the model currently in use or running, and also performs performance monitoring on models that have not yet been used or run.

[0182] In some embodiments, based on the first request, the first device simultaneously monitors the performance of the currently used or running model and the model that has not been used or run, determines the performance comparison of the models, and implements the switching of the models.

[0183] In some embodiments, based on the first request, the first device monitors the performance of a model that has not been used or run to determine the performance of an alternative model.

[0184] In some embodiments, the first model includes at least one AI model or AI function.

[0185] In some embodiments, the first request may be sent via an LPP message, for example, via an LPP location request message or an LPP providing assistance information, and an LCS (Location Service) message.

[0186] In some embodiments, the first request may be sent via an NR Positioning Protocol A (NRPPa) message.

[0187] Step 2102: The second device sends a second request to the first device.

[0188] In some embodiments, the first device receives a second request sent by the second device.

[0189] In some embodiments, the second request may be used to indicate information used by the first device when determining the performance indicator and / or the performance monitoring result. In some embodiments, the information includes input parameters and / or output parameters of the first model.

[0190] For example, the second device instructs the first device to use all or part of the input parameters of the first model to monitor the performance of the first model, or the second device instructs the first device to use all or part of the output parameters of the first model to monitor the performance of the first model, or the second device instructs the first device to use all or part of the input parameters and output parameters of the first model to monitor the performance of the first model.

[0191] In some embodiments, the input parameters include channel measurement related parameters or reference signal measurement results.

[0192] In some embodiments, channel measurement-related parameters may include channel impulse response (CIR), power delay profile (PDP), and delay profile (DP). Reference signal measurement results may include positioning reference signal signal-to-interference-and-noise ratio (PRS-SINR), positioning reference signal received power (PRS-RSRP), synchronization signal block reference signal received power (SSB-RSRP), synchronization signal block signal-to-interference-and-noise ratio (SSB-SINR); reference signal time difference (RSTD); UE receive-transmit time difference (UERx-Tx time difference), gNB receive-transmit time difference (gNB Rx-Tx time difference), and uplink positioning reference signal signal signal-to-interference-and-noise ratio (SNR). uplink positioning reference signal received power (SRS-RSRP), uplink positioning reference signal received power (SRS-RSRP).

[0193] In some embodiments, the output parameters may include terminal location, Loss indication, NLoS indication, flight time, and a measurement quantity based on a positioning reference signal. The measurement quantity based on a positioning reference signal may be a measurement quantity of a reference signal time difference (RS TD). For example, when the first device is a terminal, the output parameters include terminal location, Loss / NLos indication, and flight time (TOA); when the first device is a gNB, the output parameters include Loss / NLos indication and flight time (TOA).

[0194] For example, the second request sent by the second device to the first device instructs the first device to use the channel impulse response CIR for performance monitoring. The first device then uses the channel impulse response CIR to perform performance monitoring on the first model to obtain performance indicators and / or performance monitoring results.

[0195] In some embodiments, the second request may be sent via an LPP message, for example, via an LPP location request message or an LPP providing assistance information, and an LCS (Location Service) message.

[0196] In some embodiments, the second request may be sent via an NRPPa message.

[0197] Step 2103: The second device sends first indication information to the first device.

[0198] In some embodiments, the first indication information is used to indicate a threshold for determining a performance monitoring result.

[0199] In some embodiments, the first device receives first indication information sent by the second device.

[0200] In some embodiments, the first indication information includes at least one of the following: a first threshold value, the first threshold value is used to evaluate the first performance indicator, and the first performance indicator is obtained by comparing the input parameter with the input data in the training data of the first model; a second threshold value, the second threshold value is used to evaluate the second performance indicator, and the second performance indicator is obtained by comparing the current input parameter with the previous input parameter; a third threshold value, the third threshold value is used to evaluate the third performance indicator, and the third performance indicator is obtained by comparing the statistical value of the output parameter with the statistical value of the output data in the training data of the first model; a fourth threshold value, the fourth threshold value is used to evaluate the fourth performance indicator, and the fourth performance indicator is obtained by comparing the current output parameter with the previous output parameter; a fifth threshold value, the fifth threshold value is used to evaluate the fifth performance indicator, and the fifth performance indicator is obtained by comparing the statistical value of the output parameter with the statistical value of the output data in the training data of the first model; a sixth threshold value, the sixth threshold value is used to evaluate the sixth performance indicator, and the sixth performance indicator is obtained by comparing the output parameter with the result obtained by the preset positioning method.

[0201] In the above embodiment, the second device sends the first indication information to the first device, so that the first device can judge the obtained performance indicator to obtain the performance monitoring result of the first model.

[0202] Step 2104: The second device sends second indication information to the first device.

[0203] In some embodiments, the first device receives second indication information sent by the second device.

[0204] In some embodiments, the second indication information may be information used to assist the first device in performing performance monitoring.

[0205] In some embodiments, the second indication information includes a geographic reality tag of the first terminal and / or a statistical value of training data of the first model.

[0206] The geographic real-time tag includes at least one of a measurement value of the first terminal, a location of the first terminal, a timestamp corresponding to the location of the first terminal, and a quality indicator of the measurement value of the first terminal.

[0207] In some embodiments, the measurement value of the first terminal may be the channel impulse response CIR, the power delay spectrum PDP, the delay characteristic DP, the signal to interference plus noise ratio PRS-SINR of the positioning reference signal, the reference signal received power PRS-RSRP of the positioning reference signal, the reference signal received power SSB-RSRP of the synchronization signal block, the signal to interference plus noise ratio SSB-SINR of the synchronization signal block; the reference signal time difference RSTD; the UE receive and transmit time difference UERx-Tx time difference, the NB receive and transmit time difference gNB Rx-Tx time difference, the signal to interference plus noise ratio SRS-SINR of the uplink positioning reference signal, and the uplink positioning reference signal received power SRS-RSRP.

[0208] In some embodiments, the training data for the first model includes input data and output data.

[0209] Among them, the input data of the training data can be the channel impulse response CIR, the power delay spectrum PDP, the delay characteristic DP, the signal to interference and noise ratio PRS-SINR of the positioning reference signal, the reference signal received power PRS-RSRP of the positioning reference signal, the reference signal received power SSB-RSRP of the synchronization signal block, the signal to interference and noise ratio SSB-SINR of the synchronization signal block; the reference signal time difference RSTD; the UE receive and transmit time difference UERx-Tx time difference, the NB receive and transmit time difference gNB Rx-Tx time difference, the signal to interference and noise ratio SRS-SINR of the uplink positioning reference signal, and the uplink positioning reference signal received power SRS-RSRP.

[0210] The output data of the training data may include terminal position, Los indication, NLoS indication, flight time, and measurement quantity based on positioning reference signal. The measurement quantity based on positioning reference signal may be measurement quantity of reference signal time difference (RS TD).

[0211] For example, the measurement value of the first terminal includes a measurement value of the first terminal for a reference signal, such as a measurement value of a positioning reference signal.

[0212] For example, the measurement value of the first terminal has a corresponding relationship with the position of the first terminal. For example, the position of the first terminal is obtained according to the measurement value of the first terminal.

[0213] For example, the first device performs performance monitoring on the first model and obtains a performance indicator obtained by referring to the measurement values ​​and positions of other terminals in the second indication information to assist in obtaining a monitoring result of the positioning performance of the first model.

[0214] For example, according to the second indication information, the first device uses the measurement value of the first terminal to monitor part or all of the input parameters of the first model, thereby achieving performance monitoring of the first model.

[0215] For example, according to the second indication information, the first device uses the location of the first terminal to monitor part or all of the output parameters of the first model, thereby achieving performance monitoring of the first model.

[0216] For example, according to the second indication information, the first device uses the measurement value and location of the first terminal to monitor part or all of the input and output parameters of the first model, thereby achieving performance monitoring of the first model.

[0217] For example, the first model is trained by other devices and then used by the first device. The second device sends the statistical value of the training data of the first model to the first device so that the first device can monitor the performance of the first model.

[0218] For example, the first model is trained by the second device and then given to the first device for use. The second device sends the statistical value of the training data of the first model to the first device so that the first device can monitor the performance of the first model.

[0219] For example, according to the second indication information, the first device uses part or all of the input parameters of the first model training data to monitor part or all of the input parameters of the first model, thereby achieving performance monitoring of the first model.

[0220] For example, according to the second indication information, the first device uses part or all of the output parameters of the first model training data to monitor part or all of the output parameters of the first model, thereby achieving performance monitoring of the first model.

[0221] For example, according to the indicated information, the first device uses part or all of the output and input parameters of the first model training data to monitor part or all of the output and input parameters of the first model, thereby achieving performance monitoring of the first model.

[0222] In the above embodiment, the second device sends the second indication information to the first device. The second indication information can assist the first device in performing performance monitoring on the first model, thereby determining the performance index and / or performance monitoring result of the first model.

[0223] Step 2105: The second device sends third indication information to the first device.

[0224] In some embodiments, the third indication information may be used to indicate a preset positioning method.

[0225] In some embodiments, the first device receives third indication information sent by the second device.

[0226] For example, the preset positioning method may be a positioning method independent of wireless access technology, such as WiFi or Bluetooth, or a positioning method based on wireless access technology, such as downlink time difference of arrival positioning method DL-TDOA.

[0227] In some embodiments, when the second device instructs the first device to use the output parameters of the model for performance monitoring, the second device instructs the first device to use the output results obtained by the specified positioning method, such as reference signal measurement results, UE location information, multipath LOS / NLOS indication, and compare them with the output results of the model to achieve performance monitoring of the model.

[0228] The execution order of step 2101, step 2102, step 2103, step 2104, and step 2105 is not limited and they can be executed separately or simultaneously.

[0229] Step 2106: The first device performs performance monitoring on the first model.

[0230] In some embodiments, the first device performs performance monitoring on the first model based on at least one of the first request, the second request, the first indication information, and the second indication information.

[0231] For example, the first device performs performance monitoring on the first model based on the first request to obtain a performance indicator of the first model. The manner in which the first device performs performance monitoring on the first model is not limited.

[0232] For example, the first device monitors the performance of the first model based on the first request and the second request, that is, instructs the first device to monitor the performance of the first model using the input parameters and / or output parameters of the first model to obtain the performance indicators of the first model.

[0233] For example, the first device performs performance monitoring on the first model based on the first request, the second request, and the first indication information, and can obtain performance indicators and performance monitoring results of the first model.

[0234] For example, the first device performs a performance check on the first model based on the first request, the second request, the first indication information, and the second indication information, and can obtain the performance indicators and performance monitoring results of the first model.

[0235] In some embodiments, the first device performs performance monitoring on the first model in order to obtain a performance indicator or a performance monitoring result of the first model.

[0236] In some embodiments, the performance monitoring may be that the first device performs performance monitoring on the output parameters of the first model.

[0237] In some embodiments, the performance monitoring may be that the first device performs performance monitoring on input parameters of the first model.

[0238] In some embodiments, the performance monitoring may be that the first device performs performance monitoring on the output and input parameters of the first model.

[0239] In some embodiments, performance monitoring may be performed by comparing input parameters of the first model with input data in the training data of the first model. For example, the CIR of the first model input may be compared with the CIR of the input data in the training data of the first model to obtain a performance indicator related to the CIR of the input parameters.

[0240] In some embodiments, performance monitoring may be comparing output parameters of the first model with output data in training data of the first model.

[0241] For example, the terminal position in the output parameters of the first model may be compared with the terminal position output in the training data to obtain a performance indicator of the first model related to the terminal position.

[0242] In some embodiments, performance monitoring may be performed by comparing the statistical value of the input parameter of the first model with the statistical value of the input data in the training data of the first model. For example, the performance indicator of the statistical value of the CIR of the input of the first model may be obtained by comparing the mean CIR of the input of the first model with the mean CIR of the input in the training data of the first model.

[0243] In some embodiments, performance monitoring can be performed by comparing the current input parameters of the first model with the previous input parameters of the first model. For example, the current input parameter is the channel impulse response CIR input to the first model this time, and the previous input parameter is the channel impulse response CIR input to the first model last time. By comparing the CIR input this time with the CIR input last time, the performance index of the first model related to CIR can be obtained.

[0244] In some embodiments, performance monitoring may involve comparing statistical values ​​of an output parameter of the first model with statistical values ​​of output data in the training data of the first model. For example, if the output parameter of the first model is the location of a terminal, the average of the terminal locations output by the first model multiple times may be compared with the average of the terminal locations output multiple times in the training data of the first model to obtain a performance indicator of the first model related to the terminal location.

[0245] In some embodiments, performance monitoring may be performed by comparing a current output parameter of the first model with a previous output parameter of the first model. For example, the current output parameter of the first model is the flight time outputted this time, and the previous output parameter is the flight time outputted last time. The flight time outputted this time is compared with the flight time outputted last time to obtain a performance indicator of the first model related to the flight time.

[0246] In some embodiments, performance monitoring may be comparing statistical values ​​of current output parameters of the first model with statistical values ​​of previous output parameters of the first model.

[0247] In some embodiments, performance monitoring may include comparing output parameters of the first model with results obtained by a preset positioning method. For example, the preset positioning method may be Bluetooth positioning. By comparing the terminal location output by the first model with the terminal location obtained by Bluetooth positioning, performance indicators related to the terminal location can be obtained.

[0248] In some embodiments, the statistical value is the mean or variance of any one of multiple input parameters, output parameters, input data, output data, current output parameters, and previous output parameters of the same model.

[0249] In some embodiments, the statistical value is the mean or variance of any one of multiple input parameters, output parameters, input data, output data, current output parameters, and previous output parameters of the same model.

[0250] For example, the first device may compare the output parameter flight time of the first model with the output parameter flight time obtained by a positioning method using a downlink time difference of arrival observation positioning method DL-TDOA to obtain a performance indicator.

[0251] In some embodiments, the performance indicators include at least one of the following: channel impulse response CIR, power delay profile PDP, delay characteristic DP, signal to interference plus noise ratio PRS-SINR of positioning reference signal, reference signal received power PRS-RSRP of positioning reference signal, reference signal received power SSB-RSRP of synchronization signal block, signal to interference plus noise ratio SSB-SINR of synchronization signal block, reference signal time difference RSTD, UE receive transmit time difference UERx-Tx time difference, gNB receive transmit time difference gNB Rx-Tx time difference, signal to interference plus noise ratio SRS-SINR of uplink positioning reference signal, uplink positioning reference signal received power SRS-RSRP, terminal position obtained by the first model, terminal position obtained by the preset positioning method; flight time.

[0252] In some embodiments, the first device may perform performance monitoring on the first model based on the first request, the second request, and the first indication information to obtain a performance monitoring result of the first model.

[0253] In some embodiments, the performance monitoring result can be determined based on a first threshold, wherein the first threshold is used to evaluate a first performance indicator, which is obtained by comparing the input parameters with the input data in the training data of the first model.

[0254] In some embodiments, the performance monitoring result may be determined based on a second threshold, where the second threshold is used to evaluate a second performance indicator obtained by comparing a current input parameter with a previous input parameter.

[0255] In some embodiments, the performance monitoring result can be determined based on a third threshold, wherein the third threshold is used to evaluate a third performance indicator, and the third performance indicator is obtained by comparing the statistical value of the output parameter with the statistical value of the output data in the training data of the first model.

[0256] In some embodiments, the performance monitoring result may be determined based on a fourth threshold, where the fourth threshold is used to evaluate a fourth performance indicator obtained by comparing a current output parameter with a previous output parameter.

[0257] In some embodiments, the performance monitoring result can be determined based on a fifth threshold, wherein the fifth threshold is used to evaluate a fifth performance indicator, which is obtained by comparing the statistical value of the output parameter with the statistical value of the output data in the training data of the first model.

[0258] In some embodiments, the performance monitoring result may be determined based on a sixth threshold, where the sixth threshold is used to evaluate a sixth performance indicator, which is obtained by comparing the output parameter with a result obtained by a preset positioning method.

[0259] In some embodiments, the performance monitoring result may be at least one of the following: the first model is no longer applicable, the expectation of applying the second model, a model update indication, a performance indicator for determining that the first model is no longer applicable, and a method for model performance monitoring.

[0260] For example, the change in PRS-SINR may be lower than a set threshold value, or the statistical value of PRS-SINR may be lower than a set threshold value, and it may be concluded that the first model is no longer applicable, or the performance indicator for determining that the first model is no longer applicable is the signal to interference plus noise ratio PRS-SINR of the positioning reference signal, or the second model is expected to be applicable, and the second model refers to a model deployed on the first device but not currently used.

[0261] For example, a model update indication may be obtained when a change between an input parameter CIR of the first model and an input data CIR in the training data is lower than a first threshold.

[0262] In the above embodiment, the first device performs performance monitoring on the first model based on the performance monitoring request sent by the second device or one or more of the information or auxiliary information used for performance monitoring, and can obtain the performance indicators of the first model. It can also obtain the performance monitoring results of the first model based on the threshold sent by the second device.

[0263] Step 2107: The second device sends a third request to the first device.

[0264] In some embodiments, the first device receives a third request sent by the second device.

[0265] In some embodiments, the third request may be a request from the second device to the first device to send it a performance indicator or a performance monitoring result of the first model, or a performance indicator and a performance monitoring result.

[0266] For example, the second device LMF requests the terminal to send the performance indicator to the LMF, and the LMF can use the performance indicator to perform further operations.

[0267] For example, the second device LMF requests the gNB to send the performance indicator to the LMF, and the LMF can use the performance indicator for further operations.

[0268] For example, the second device LMF requests the terminal to send the performance monitoring result to the LMF, and the LMF can use the performance monitoring result to perform further operations.

[0269] For example, the second device LMF requests the gNB to send the performance monitoring results to the LMF, and the LMF can use the performance monitoring results to perform further operations.

[0270] Step 2108: The first device sends the performance indicator and / or performance monitoring result to the second device.

[0271] In some embodiments, the second device receives the performance indicators and / or performance monitoring results sent by the first device.

[0272] In some embodiments, the first device sends the performance indicator to the second device based on a third request sent by the second device.

[0273] In some embodiments, the first device sends the performance monitoring result to the second device based on a third request sent by the second device.

[0274] In some embodiments, the first device sends the performance indicator and the performance monitoring result to the second device based on the third request sent by the second device.

[0275] For example, the terminal sends the performance indicator to the LMF based on the third request, or the gNB sends the performance indicator to the LMF based on the third request, or the terminal sends the performance monitoring result to the LMF based on the third request, or the terminal sends the performance indicator and the performance monitoring result to the LMF based on the third request.

[0276] For example, the terminal sends the performance indicator based on the input parameter channel impulse response CIR obtained by performing performance monitoring on the first model to the LMF based on the third request.

[0277] For example, the terminal sends the performance indicator based on the output parameter terminal position obtained by performing performance monitoring on the first model to the LMF based on the third request.

[0278] For example, the terminal sends the performance indicator based on the input parameter CIR and the output parameter terminal location obtained by performance monitoring the first model to the LMF based on the third request.

[0279] Step 2109: The first device sends fourth indication information to the second device.

[0280] In some embodiments, the second device receives fourth indication information sent by the first device.

[0281] In some embodiments, the fourth indication information may be used to instruct the second device to determine the performance monitoring result of the first model according to the performance indicator.

[0282] For example, the terminal sends an instruction to the LMF, requesting the LMF to determine the performance monitoring result of the first model according to the input parameter CIR sent by the terminal.

[0283] For example, the gNB sends an indication to the LMF, requesting the LMF to determine the performance monitoring result of the first model based on the output parameter terminal location sent by the gNB.

[0284] For example, the terminal or gNB sends an indication to the LMF, requesting the LMF to determine the performance monitoring result of the first model based on the input parameter CIR and the output parameter terminal location.

[0285] Step 2110: The first device sends fifth indication information to the second device.

[0286] In some embodiments, the second device receives fifth indication information sent by the first device.

[0287] In some embodiments, the fifth indication information may be a statistical value indicating the training data of the first model.

[0288] For example, the terminal sends the statistical value of the training data of the first model to the LMF, so that the LMF uses the statistical value of the training data to monitor the performance of the first model.

[0289] In some embodiments, the first device sends the mean CIR of the input data in the training data of the first model to the LMF, and the LMF determines the performance monitoring result of the first model based on the received performance indicators of the first model related to CIR and the mean CIR in the training data.

[0290] In some embodiments, the first device sends the mean of the positions of the output data terminals in the training data of the first model to the LMF, and the LMF judges the performance monitoring results of the first model based on the received performance indicators of the terminal positions of the first model and the mean of the terminal positions in the training data.

[0291] Step 2111: The second device determines the performance monitoring result of the first model.

[0292] In some embodiments, the second device determines the performance monitoring result of the first model based on the performance indicator and the fourth indication information sent by the first device.

[0293] In some embodiments, the second device determines the performance monitoring result of the first model based on the performance indicator, the fourth indication information, and the fifth indication information sent by the first device.

[0294] In some embodiments, the performance monitoring result of the first model includes at least one of the following: the first model is no longer applicable; the expectation of applying the second model; a model update indication; a performance indicator for determining that the first model is no longer applicable; and a method for model performance monitoring.

[0295] For example, the change in PRS-SINR may be lower than a set threshold value, or the statistical value of PRS-SINR may be lower than a set threshold value, and it may be concluded that the first model is no longer applicable, or the performance indicator for determining that the first model is no longer applicable is the signal to interference plus noise ratio PRS-SINR of the positioning reference signal, or the second model is expected to be applicable, and the second model refers to a model deployed on the first device but not currently used.

[0296] For example, a model update indication may be obtained when a change between an input parameter CIR of the first model and an input data CIR in the training data is lower than a first threshold.

[0297] In some embodiments, the second device determines the performance monitoring result of the first model in the same manner as the first device determines the performance monitoring result of the first model in step 2106, and is not repeated here.

[0298] In some embodiments, the second device may send the first request, the second request, the first indication information, the second indication information, the third indication information, and the third request to the first device via one message or multiple messages, which is not limited in this disclosure.

[0299] In some embodiments, the first device may receive the first request, the second request, the first indication information, the second indication information, the third indication information, and the third request sent by the second device through one message or multiple messages, which is not limited by the present disclosure.

[0300] In some embodiments, the first device may send the performance indicator and / or performance monitoring result, the fourth indication information, and the fifth indication information to the second device via one message or multiple messages, which is not limited in this disclosure.

[0301] In some embodiments, the second device may receive the performance indicator and / or performance monitoring result, the fourth indication information, and the fifth indication information sent by the first device through one message or multiple messages, which is not limited by the present disclosure. The model performance monitoring method involved in the embodiments of the present disclosure may include at least one of steps 2101 to 2111. For example, step 2101 can be tried as an independent embodiment, step 2106 can be implemented as an independent embodiment, and so on, but is not limited to this. Step 2101+2106, Step 2101+2102+2106, Step 2101+2102+2103+2106, Step 2101+2102+2104+2106, Step 2101+2102+2103+2104+2106, Step 2101+2102+2105+2106, Step 2101+2102+2103+2105+2106, Step 2101+2102+2103+2104+2105+2106, Step 2101+2102+2104+2105+2106 , step 2101+2106+2107, step 2101+2102+2106+2107, step 2101+2102+2103+2106+2107, step 2101+2102+2104+2106+2107, step 2101+2102+2103+2104+2106+2107, step 2101+2102+2105+2106+2107, step 2101+2102+2103+2104+ 2105+2106+2107, Step 2101+2102+2104+2105+2106+2107, Step 2101+2106+2107+2108, Step 2101+2102+2106+2107+2108, Step 2101+2102+2103+2106+2107+2108, Step 2101+2102+2104+2106+2107+2108, Step 2101+2102+2103+2104+2106+2107+2108 2+2105+2106+2107+2108, step 2101+2102+2103+2105+2106+2107+2108, step 2101+2102+2103+2104+2105+2106+2107+2108, step 2101+2102+2104+2105+2106+2107+2108, step 2101+2106+2107+2108+2109+2111, step 2101+2102+2106+2107+2108+2109+2111,Steps 2101+2102+2103+2106+2107+2108+2109+2111, Steps 2101+2102+2104+2106+2107+2108+2109+2111, Steps 2101+2102+2103+2104+2106+2107+2108+2109+2111, Steps 2101+2102+2105+2106+2107+2108+2109+2111, Steps 2101+2102+2103+2104+2106+2107+2108+2109+2111 05+2106+2107+2108+2109+2111, Steps 2101+2102+2103+2104+2105+2106+2107+2108+2109+2111, Steps 2101+2102+2104+2105+2106+2107+2108+2109+2111, Steps 2101+2106+2107+2108+2109+2110+2111, Steps 2101+2102+2106+2107+2108+21 09+2110+2111, Steps 2101+2102+2103+2106+2107+2108+2109+2110+2111, Steps 2101+2102+2104+2106+2107+2108+2109+2110+2111, Steps 2101+2102+2103+2104+2106+2107+2108+2109+2110+2111, Steps 2101+2102+2105+2106+2107+2108+2 109+2110+2111, steps 2101+2102+2103+2105+2106+2107+2108+2109+2110+2111, steps 2101+2102+2104+2105+2106+2107+2108+2109+2110+2111, and steps 2101+2102+2103+2104+2105+2106+2107+2108+2109+2110+2111 can be implemented as independent embodiments, but are not limited thereto.

[0302] In some embodiments, steps 2102, 2103, 2104, 2105, 2109, 2110, and 2111 are optional, and any one or more of the steps may be omitted or replaced in different embodiments.

[0303] In some embodiments, the order of step 2101, step 2102, step 2103, step 2104, and step 2105 is not limited, and the order of each step can be exchanged in different embodiments.

[0304] In some embodiments, the order of step 2107, step 2109, and step 2110 is not limited, and the order of each step can be exchanged in different embodiments.

[0305] In this embodiment or example, unless there is any contradiction, each step can be independent, arbitrarily combined or exchanged in order, the optional methods or optional examples can be arbitrarily combined, and can be arbitrarily combined with any steps of other embodiments or other examples.

[0306] FIG3A is a flow chart of a method for monitoring model performance of a first device according to an embodiment of the present disclosure. The present disclosure embodiment relates to a method for monitoring model performance, and the method includes:

[0307] Step 3101: Receive a first request sent by a second device.

[0308] The optional implementation of step 3101 can refer to the optional implementation of step 2101 in Figure 2 and other related parts in the embodiment involved in Figure 2, which will not be repeated here.

[0309] Step 3102: Receive a second request sent by a second device.

[0310] For optional implementations of step 3102, please refer to the optional implementations of step 2102 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0311] Step 3103: Receive first indication information sent by the second device.

[0312] For optional implementations of step 3103, please refer to the optional implementations of step 2103 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0313] Step 3104: Receive second indication information sent by the second device.

[0314] For optional implementations of step 3104, please refer to the optional implementations of step 2104 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0315] Step 3105: Receive third indication information sent by the second device.

[0316] For optional implementations of step 3105, please refer to the optional implementations of step 2105 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0317] Step 3106: Monitor the performance of the first model.

[0318] For optional implementations of step 3106, please refer to the optional implementations of step 2106 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0319] Step 3107: Receive a third request sent by the second device.

[0320] For optional implementations of step 3107, please refer to the optional implementations of step 2107 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0321] Step 3108: Send the performance indicator and / or performance monitoring result to the second device.

[0322] For optional implementations of step 3108, please refer to the optional implementations of step 2108 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0323] Step 3109: Send fourth indication information to the second device.

[0324] For optional implementations of step 3109, please refer to the optional implementations of step 2109 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0325] Step 3110: Send fifth indication information to the second device.

[0326] For optional implementations of step 3110, please refer to the optional implementations of step 2110 in Figure 2 and other related parts in the embodiment involved in Figure 2, which will not be repeated here.

[0327] The model identification determination method involved in the embodiment of the present disclosure may include at least one of steps 3101 to 3110. For example, step 3101 can be implemented as an independent embodiment, and step 3106 can be implemented as an independent embodiment. And so on, but not limited to this. Step 3101+3106, step 3101+3102+3106, step 3101+3102+3103+3106, step 3101+3102+3104+3106, step 3101+3102+3103+3104+3106, step 3101+3102+3105+3106, step 3101+3102+3103+3105+3106, step 3101+3102+3104+3106 103+3104+3105+3106, step 3101+3102+3104+3105+3106, step 3101+3106+3107, step 3101+3102+3106+3107, step 3101+3102+3103+3106+3107, step 3101+3102+3104+3106+2107, step 3101+3102+3103+3104+3106+2107 06+3107, step 3101+3102+3105+3106+3107, step 3101+3102+3103+3105+3106+3107, step 3101+3102+3103+3104+3105+3106+3107, step 3101+3102+3104+3105+3106+3107, step 3101+3106+3107+3108, step 310 1+3102+3106+3107+3108, step 3101+3102+3103+3106+3107+3108, step 3101+3102+3104+3106+3107+3108, step 3101+3102+3103+3104+3106+3107+3108, step 3101+3102+3105+3106+3107+3108, step 3101+3102+3105+3106+3107+3108 02+3103+3105+3106+3107+3108, step 3101+3102+3104+3105+3106+3107+3108, and step 3101+3102+3103+3104+3105+3106+3107+3108 can be implemented as independent embodiments, but are not limited to this.

[0328] In some embodiments, step 3102, step 3103, step 3104, step 3105, step 3109, and step 3110 are optional, and any one or more of the steps may be omitted or replaced in different embodiments.

[0329] In some embodiments, the order of step 3101, step 3102, step 3103, step 3104, step 3105, and step 3107 is not limited, and the order of each step can be exchanged in different embodiments.

[0330] In some embodiments, the order of step 3108, step 3109, and step 3110 is not limited, and the order of each step can be exchanged in different embodiments.

[0331] FIG3B is a flow chart of a method for monitoring model performance of a first device according to an embodiment of the present disclosure. The present disclosure embodiment relates to a method for monitoring model performance, and the method includes:

[0332] Step 3201: Receive a first request sent by a second device.

[0333] The first request is used to request the first device to perform performance monitoring on the first model. The first model is deployed on the first device. The performance monitoring includes determining performance indicators and / or performance monitoring results.

[0334] Optional implementations of step 3201 can be found in step 2101 of FIG. 2 , optional implementations of step 3101 of FIG. 3A , and other related parts in the embodiments involved in FIG. 2 and FIG. 3A , which will not be described in detail here.

[0335] In an embodiment of the present disclosure, step 3201 may be combined with step 3102 in FIG. 3A , and step 3201 may be combined with step 3106 in FIG. 3A .

[0336] FIG4A is a flow chart of a method for monitoring the performance of a model of a second device according to an embodiment of the present disclosure. The present disclosure embodiment relates to a method for monitoring the performance of a model, and the method includes:

[0337] Step 4101: Send a first request to a first device.

[0338] The optional implementation of step 4101 can refer to the optional implementation of step 2101 in Figure 2 and other related parts in the embodiment involved in Figure 2, which will not be repeated here.

[0339] Step 4102: Send a second request to the first device.

[0340] For optional implementations of step 4102, please refer to the optional implementations of step 2102 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0341] Step 4103: Send first indication information to the first device.

[0342] For optional implementations of step 4103, please refer to the optional implementations of step 2103 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0343] Step 4104: Send second indication information to the first device.

[0344] For optional implementations of step 4104, please refer to the optional implementations of step 2104 in Figure 2 and other related parts in the embodiment involved in Figure 2, which will not be repeated here.

[0345] Step 4105: Send third indication information to the first device.

[0346] For optional implementations of step 4105, please refer to the optional implementations of step 2105 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0347] Step 4106: Send a third request to the first device.

[0348] For optional implementations of step 4106, please refer to the optional implementations of step 2107 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0349] Step 4107: Receive the performance indicators and / or performance monitoring results sent by the first device.

[0350] For optional implementations of step 4107, please refer to the optional implementations of step 2108 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0351] Step 4108: Receive fourth indication information sent by the first device.

[0352] For optional implementations of step 4108, please refer to the optional implementations of step 2109 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0353] Step 4109: Receive the fifth indication information sent by the first device.

[0354] For optional implementations of step 4109, please refer to the optional implementations of step 2110 in FIG. 2 and other related parts in the embodiment involved in FIG. 2 , which will not be described in detail here.

[0355] Step 4110: Determine the performance monitoring results of the first model.

[0356] For optional implementations of step 4110, reference may be made to the optional implementations of step 2111 in FIG. 2 and other related parts of the embodiment involved in FIG. 2 , which will not be described in detail here.

[0357] The model identification determination method involved in the embodiments of the present disclosure may include at least one of steps 4101 to 4110. For example, step 4101 may be implemented as an independent embodiment, and step 4106 may be implemented as an independent embodiment, and so on, but the present invention is not limited thereto. Step 4101+4102, Step 4101+4102+4103, Step 4101+4102+4103+4104, Step 4101+4102+4105, Step 4101+4102+4103+4105, Step 4101+4102+4103+4104+4105, Step 4101+4102+4106+4107, Step 4101+4102+4103+4106+4107, Step 4101+4102+4103+4104+4105+4106+4107, Step 4101+4102+4103+4104+4105+4106+4107 106+4107+4108+4110, step 4101+4102+4106+4107+4108++4109+4110, step 4101+4102+4103+4106+4107+4110, step 4101+4102+4103+4104+4106+4107+4110, step 4101+4102+4103+4104+4106+4107+4108+4109+4110, and step 4101+4102+4103+4104+4105+4106+4107+4108+4109+4110 can be implemented as independent embodiments, but are not limited to this.

[0358] In some embodiments, step 4103, step 4104, step 4105, step 4108, and step 4109 are optional, and any one or more of the steps may be omitted or replaced in different embodiments.

[0359] In some embodiments, the order of step 4101, step 4102, step 4103, step 4104, step 4105, and step 4106 is not limited, and the order of each step can be exchanged in different embodiments.

[0360] In some embodiments, the order of step 4108 and step 4109 is not limited, and the order of each step can be exchanged in different embodiments.

[0361] FIG4B is a flow chart of a method for monitoring the performance of a model of a second device according to an embodiment of the present disclosure. The present disclosure embodiment relates to a method for monitoring the performance of a model, and the method includes:

[0362] Step 4201: Send a first request to a first device.

[0363] The first request is used to request the first device to perform performance monitoring on the first model. The first model is deployed on the first device. The performance monitoring includes determining performance indicators and / or performance monitoring results.

[0364] The optional implementation of step 4201 can be found in step 2101 of Figure 2, step 3101 of Figure 3A, step 3201 of Figure 3B, the optional implementation of step 4101 of Figure 4A, and other related parts in the embodiments involved in Figures 2, 3A, 3B, and 4A, which will not be repeated here.

[0365] In an embodiment of the present disclosure, step 4201 may be combined with step 4102 or step 4105 in FIG. 4A .

[0366] Figure 5 is an interactive diagram of a model performance monitoring method provided according to an embodiment of the present disclosure. As shown in Figure 5, an embodiment of the present disclosure relates to a model performance monitoring method, the method comprising:

[0367] Step 5101: The first device receives a first request sent by the second device.

[0368] The first request is used to request the first device to perform performance monitoring on the first model. The first model is deployed on the first device. The performance monitoring includes determining performance indicators and / or performance monitoring results.

[0369] For optional implementations of step 5101, please refer to the optional implementations of step 2101 in Figure 2, step 3101 in Figure 3A, step 3201 in Figure 3B, step 4101 in Figure 4A, step 4201 in Figure 4B, and other related parts in the embodiments involved in Figures 2, 3A, 3B, 4A, and 4B, which will not be repeated here.

[0370] In some embodiments, the above method may include the method described in the above embodiments of the first device side, the second device side, etc., which will not be repeated here.

[0371] In this embodiment or example, unless there is any contradiction, each step can be independent, arbitrarily combined or exchanged in order, the optional methods or optional examples can be arbitrarily combined, and can be arbitrarily combined with any steps of other embodiments or other examples.

[0372] The following is a specific embodiment of a model performance monitoring method provided in an embodiment of the present disclosure, including performance monitoring of an AI model / function when deployed on a terminal, and performance monitoring of an AI model / function when deployed on a gNB. Optionally, the AI ​​model / function can be a model currently used by the terminal or gNB, a model deployed on the terminal or gNB but not yet used, a model deployed on the terminal or gNB but not yet running, or a model currently running on the first device. Specifically, the method includes the following steps:

[0373] Example 1:

[0374] Step 1: The LMF requests the terminal and / or gNB to use the input or output of the AI ​​model for AI performance monitoring.

[0375] The inputs to an AI model may include:

[0376] (1) Input values ​​of the AI ​​model deployed on the terminal side: channel impulse response (CIR), power delay profile (PDP), delay profile (DP), positioning reference signal signal to interference plus noise ratio (PRS-SINR), positioning reference signal reference signal received power (PRS-RSRP), synchronization signal block reference signal received power (SSB-RSRP), synchronization signal block signal to interference plus noise ratio (SSB-SINR), reference signal time difference (RSTD), UE receive transmit time difference (UERx-Tx time difference), and other measurement results;

[0377] (2) Input values ​​of the AI ​​model deployed on the gNB side: channel impulse response (CIR), power delay profile (PDP), delay profile (DP), uplink positioning reference signal signal-to-interference-plus-noise ratio (SRS-SINR), uplink positioning reference signal received power (SRS-RSRP), and gNB receive-transmit time difference (gNB Rx-Tx time difference).

[0378] Step 2: Performance monitoring based on AI model input:

[0379] (1) Solution for deploying AI models on the terminal side: Compare the AI ​​input with the training data. For example, compare the CIR of the AI ​​input with the CIR of the training data, or compare the statistical characteristics of multiple CIRs of the same AI input model with the statistical characteristics of the training data (such as the mean or variance).

[0380] (2) Solution where the AI ​​model is deployed on the gNB side: Performance monitoring is performed based on changes in AI input, such as when the change in PRS SINR is lower than a certain threshold, or when the change in the statistical characteristics of PRS SINR is lower than or higher than a certain threshold.

[0381] Step 3: The output of the AI ​​model includes one of the following:

[0382] (1) AI model deployment on the UE side: terminal location, Loss / NLos indication, and Time of Arrival (TOA);

[0383] (2) Solution for deploying the AI ​​model on the gNB side: Loss / NLos indication and TOA.

[0384] Step 4: The terminal or gNB performs performance monitoring based on the output of the AI ​​model.

[0385] (1) Compare the statistics output by the AI ​​model with those output by the training data;

[0386] (2) The statistical data output by the AI ​​model is compared with its own previous reasoning output or with the change in the statistical data output by the AI ​​model;

[0387] (3) Compare the AI ​​output with the results of other non-AI positioning methods, such as the terminal's location or RSRP.

[0388] Step 5: If the LMF requests the terminal or gNB to compare the AI ​​output with the results of other non-AI positioning methods for performance monitoring, the LMF may also instruct the terminal to use the following positioning methods:

[0389] Positioning methods independent of wireless access technology: Wi-Fi, Bluetooth, etc.

[0390] Positioning methods based on wireless access technology: Downstream Determined Time Difference of Arrival (DL-TDOA) positioning method, etc.

[0391] Step 6: The LMF may also provide indication information to the terminal or gNB. The indication information is used by the terminal or gNB to determine the performance of the AI ​​model based on the monitoring indicators. For example:

[0392] The threshold at which AI input is compared with training data;

[0393] Thresholds for changes in AI input;

[0394] The threshold for comparing the statistics output by the AI ​​with the statistics of the training data;

[0395] Thresholds for comparing the statistics of the AI’s output with its own previous reasoning outputs;

[0396] The threshold for the amount of change in the statistical data output by the AI;

[0397] The threshold for comparing AI output with the results of other non-AI positioning methods.

[0398] Step 7: The LMF can also provide the following auxiliary information to the terminal or gNB. The auxiliary information is used by the terminal to monitor AI performance:

[0399] (1) Geographical facts label, including the following:

[0400] The measurement result corresponding to the terminal's position;

[0401] The location of the terminal;

[0402] Time to obtain measurement results;

[0403] The quality of the measurement results obtained by other terminals.

[0404] (2) Statistical characteristics of the AI ​​model’s training data.

[0405] Step 8: The terminal or gNB provides the performance monitoring results.

[0406] The AI ​​model / function is no longer applicable;

[0407] Indicate new AI models / capabilities;

[0408] AI model update instructions;

[0409] Determine which inputs or outputs are unavailable based on performance monitoring metrics;

[0410] Performance monitoring methods.

[0411] Example 2:

[0412] The LMF determines the performance monitoring results based on the performance indicators sent by the terminal or gNB:

[0413] Step 1. After step 3 of embodiment 1, the LMF requests the terminal or gNB to provide AI performance indicators.

[0414] Step 2: LMF instructs the terminal to use which traditional positioning method to obtain the terminal's position.

[0415] Step 3: The terminal or gNB reports the performance monitoring indicators of the AI ​​model based on the LMF request.

[0416] Performance monitoring indicators include: channel impulse response CIR, power delay profile PDP, delay characteristic DP, signal to interference and noise ratio PRS-SINR of positioning reference signal, reference signal received power PRS-RSRP of positioning reference signal, reference signal received power SSB-RSRP of synchronization signal block, signal to interference and noise ratio SSB-SINR of synchronization signal block, reference signal time difference RSTD, etc.; positioning results based on AI model and the location of the terminal using traditional positioning methods.

[0417] Step 4: The terminal or gNB may also provide the LMF with the statistical characteristics of the training data of the AI ​​model.

[0418] Step 5: The terminal or gNB may also provide indication information to the LMF, requesting the LMF to determine the performance of the AI ​​model based on the performance monitoring indicators, for example:

[0419] The threshold at which AI input is compared with training data;

[0420] Thresholds for changes in AI input;

[0421] The threshold for comparing the statistics output by the AI ​​with the statistics of the training data;

[0422] Thresholds for comparing the statistics of the AI’s output with its own previous reasoning outputs;

[0423] The threshold for the amount of change in the statistical data output by the AI;

[0424] The threshold for comparing AI output with the results of other non-AI positioning methods.

[0425] In the embodiments of the present disclosure, some or all of the steps and their optional implementations may be arbitrarily combined with some or all of the steps in other embodiments, or may be arbitrarily combined with the optional implementations of other embodiments.

[0426] The embodiments of the present disclosure further provide an apparatus for implementing any of the above methods. For example, an apparatus is provided, comprising units or modules for implementing each step performed by a terminal in any of the above methods. For another example, another apparatus is provided, comprising units or modules for implementing each step performed by a network device (e.g., an access network device, a core network function node, a core network device, etc.) in any of the above methods.

[0427] It should be understood that the division of the various units or modules in the above device is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a physical entity, or they may be physically separated. In addition, the units or modules in the device may be implemented in the form of a processor calling software: for example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or implement the functions of the various units or modules of the above device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory within the device or a memory outside the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuits, and the functions of some or all of the units or modules can be realized by designing the hardware circuits. The above-mentioned hardware circuits can be understood as one or more processors; for example, in one implementation, the above-mentioned hardware circuit is an application-specific integrated circuit (ASIC), which realizes the functions of some or all of the above units or modules by designing the logical relationship of the components in the circuit; for example, in another implementation, the above-mentioned hardware circuit can be realized by a programmable logic device (PLD). Taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, thereby realizing the functions of some or all of the above units or modules. All units or modules of the above devices can be realized in the form of software called by the processor, or in the form of hardware circuits, or in part by the form of software called by the processor, and the rest by hardware circuits.

[0428] In the embodiments of the present disclosure, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP); in another implementation, the processor can implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the above-mentioned hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by a processor as an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0429] Figure 6A is a schematic diagram of the structure of a first device provided according to an embodiment of the present disclosure. As shown in Figure 6A, the first device 6100 includes a transceiver module 6101. In some embodiments, the transceiver module is configured to receive a first request from a second device, the first request being configured to request the first device to perform performance monitoring on a first model deployed on the first device. The performance monitoring includes determining performance indicators and / or performance monitoring results.

[0430] Optionally, the above-mentioned transceiver module is used to execute at least one of the communication steps such as sending and / or receiving performed by the first device 6100 in any of the above methods (for example, step 2101, step 2102, step 2103, step 2104, step 2105, step 2107, step 2108, step 2109, step 2110, step 3101, step 3102, step 3103, step 3104, step 3105, step 3107, step 3108, step 3109, step 3110, step 3201, but not limited to these), which will not be repeated here.

[0431] Figure 6B is a schematic diagram of the structure of a second device 6200 provided according to an embodiment of the present disclosure. As shown in Figure 6B, the second device 6200 may include a transceiver module 6201. In some embodiments, the transceiver module is configured to send a first request to a first device, requesting the first device to perform performance monitoring on a first model deployed on the first device. The performance monitoring includes determining performance indicators and / or performance monitoring results.

[0432] Optionally, the above-mentioned transceiver module is used to execute at least one of the communication steps such as sending and / or receiving performed by the second device 6200 in any of the above methods (for example, step 2101, step 2102, step 2103, step 2104, step 2105, step 2107, step 2108, step 2109, step 2110, step 4101, step 4102, step 4103, step 4104, step 4105, step 4106, step 4107, step 4108, step 4109, step 4201, but not limited to these), which will not be repeated here.

[0433] In some embodiments, the transceiver module may include a transmitting module and / or a receiving module, and the transmitting module and the receiving module may be separate or integrated. Optionally, the transceiver module may be interchangeable with the transceiver.

[0434] Figure 7A is a schematic diagram of the structure of a communication device 7100 provided according to an embodiment of the present disclosure. Communication device 7100 can be a network device (e.g., an access network device, a core network device, etc.), a terminal (e.g., a user device, etc.), a chip, a chip system, or a processor that supports a network device to implement any of the above methods, or a chip, a chip system, or a processor that supports a terminal to implement any of the above methods. Communication device 7100 can be used to implement the methods described in the above method embodiments. For details, please refer to the description of the above method embodiments.

[0435] As shown in Figure 7A, the communication device 7100 includes one or more processors 7101. The processor 7101 can be a general-purpose processor or a dedicated processor, for example, a baseband processor or a central processing unit. The baseband processor can be used to process the communication protocol and communication data, and the central processing unit can be used to control the communication device (such as a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute programs, and process program data. Optionally, the communication device 7100 is used to perform any of the above methods. Optionally, one or more processors 7101 are used to call instructions to enable the communication device 7100 to perform any of the above methods.

[0436] In some embodiments, the communication device 7100 further includes one or more transceivers 7102. When the communication device 7100 includes one or more transceivers 7102, the transceiver 7102 performs the communication steps such as sending and / or receiving in the above method (e.g., step 2101, step 2102, step 2103, step 2104, step 2105, step 2107, step 2108, step 2109, step 2110, step 3101, step 3102, step 3103, step 3104, step 3105, step 3107, step 3108, At least one of steps 3109, 3110, 3201, 4101, 4102, 4103, 4104, 4105, 4106, 4107, 4108, 4109, 4201, and 5101 is performed, and the processor 7101 performs at least one of the other steps (e.g., steps 2106, 2111, 3106, and 4110, but not limited thereto). In an optional embodiment, the transceiver may include a receiver and / or a transmitter, and the receiver and transmitter may be separate or integrated. Optionally, the terms transceiver, transceiver unit, transceiver, transceiver circuit, interface circuit, and interface may be interchangeable, the terms transmitter, transmitting unit, transmitter, and transmitting circuit may be interchangeable, and the terms receiver, receiving unit, receiver, and receiving circuit may be interchangeable.

[0437] In some embodiments, the communication device 7100 further includes one or more memories 7103 for storing data. Alternatively, all or part of the memories 7103 may be located outside the communication device 7100. In alternative embodiments, the communication device 7100 may include one or more interface circuits 7104. Optionally, the interface circuits 7104 are connected to the memory 7102 and may be configured to receive data from the memory 7102 or other devices, or to send data to the memory 7102 or other devices. For example, the interface circuits 7104 may read data stored in the memory 7102 and send the data to the processor 7101.

[0438] In some embodiments, processor 7101 may store a computer program 7105. Computer program 7105, when executed on processor 7101, enables communication device 7000 to perform the methods described in the above method embodiments. Computer program 7105 may be embedded in processor 7101, in which case processor 7101 may be implemented by hardware.

[0439] The communication device 7100 described in the above embodiment may be a network device or a terminal, but the scope of the communication device 7100 described in the present disclosure is not limited thereto, and the structure of the communication device 7100 may not be limited by FIG. 7A. The communication device may be an independent device or may be part of a larger device. For example, the communication device may be: 1) an independent integrated circuit IC, or a chip, or a chip system or subsystem; (2) a collection of one or more ICs, optionally, the above IC collection may also include a storage component for storing data or programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, an intelligent terminal device, a cellular phone, a wireless device, a handheld device, a mobile unit, an in-vehicle device, a network device, a cloud device, an artificial intelligence device, etc.; (6) others, etc.

[0440] 7B is a schematic diagram of the structure of a chip 7200 proposed in an embodiment of the present disclosure. If the communication device 7100 can be a chip or a chip system, please refer to the schematic diagram of the structure of the chip 7200 shown in FIG7B , but the present disclosure is not limited thereto.

[0441] The chip 7200 includes one or more processors 7201. The chip 7200 is configured to execute any of the above methods.

[0442] In some embodiments, chip 7200 further includes one or more interface circuits 7202. Alternatively, terms such as interface circuit, interface, and transceiver pins may be used interchangeably. In some embodiments, chip 7200 further includes one or more memories 7203 for storing data. Alternatively, all or part of memory 7203 may be located external to chip 7200. Optionally, interface circuit 7202 is connected to memory 7203 and may be used to receive data from memory 7203 or other devices, or may be used to send data to memory 7203 or other devices. For example, interface circuit 7202 may read data stored in memory 7203 and send the data to processor 7201.

[0443] In some embodiments, the interface circuit 7202 performs at least one of the communication steps such as sending and / or receiving in the above method (for example, step 2101, step 2102, step 2103, step 2104, step 2105, step 2107, step 2108, step 2109, step 2110, step 3101, step 3102, step 3103, step 3104, step 3105, step 3107, step 3108, step 3109, step 3110, step 3201, step 4101, step 4102, step 4103, step 4104, step 4105, step 4106, step 4107, step 4108, step 4109, step 4201, step 5101, but not limited to these). The interface circuit 7202 performing the communication steps of sending and / or receiving in the above method, for example, means that the interface circuit 7202 performs data exchange between the processor 7201, the chip 7200, the memory 7203, or the transceiver device. In some embodiments, the processor 7201 performs at least one of the other steps (for example, step 2106, step 2111, step 3106, and step 4110, but not limited thereto).

[0444] The modules and / or devices described in various embodiments, such as virtual devices, physical devices, and chips, can be arbitrarily combined or separated according to circumstances. Optionally, some or all steps can also be performed collaboratively by multiple modules and / or devices, which is not limited here.

[0445] The present disclosure also provides a storage medium having instructions stored thereon. When the instructions are executed on the communication device 7100, the communication device 7100 executes any of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer-readable storage medium, but is not limited thereto and may also be a storage medium readable by other devices. Optionally, the storage medium may be a non-transitory storage medium, but is not limited thereto and may also be a temporary storage medium.

[0446] The present disclosure also provides a program product, which, when executed by the communication device 7100, enables the communication device 7100 to perform any of the above methods. Optionally, the program product is a computer program product.

[0447] The present disclosure also proposes a computer program, which, when executed on a computer, causes the computer to perform any one of the above methods.

Claims

1. A method for monitoring model performance, characterized in that, The method is executed by a first device, and the method includes: Receiving a first request sent by a second device, where the first request is used to request the first device to perform performance monitoring on a first model, the first model is deployed on the first device, and the performance monitoring includes determining performance metrics and / or performance monitoring results.

2. The method according to claim 1, wherein The method further includes: Receiving a second request sent by the second device, where the second request is used to indicate information used by the first device when determining the performance metrics and / or the performance monitoring results.

3. The method according to claim 2, wherein The information includes input parameters and / or output parameters of the first model.

4. The method according to any one of claims 1 to 3, characterized in that The method further includes: Receiving first indication information sent by the second device, where the first indication information is used to indicate a threshold for determining the performance monitoring results, and the first indication information includes at least one of the following: A first threshold, which is used to evaluate a first performance metric, and the first performance metric is obtained by comparing the input parameters with input data in the training data of the first model; A second threshold, which is used to evaluate a second performance metric, and the second performance metric is obtained by comparing current input parameters with prior input parameters; A third threshold, which is used to evaluate a third performance metric, and the third performance metric is obtained by comparing the statistical value of the output parameters with the statistical value of output data in the training data of the first model; A fourth threshold, which is used to evaluate a fourth performance metric, and the fourth performance metric is obtained by comparing current output parameters with prior output parameters; A fifth threshold, which is used to evaluate a fifth performance metric, and the fifth performance metric is obtained by comparing the statistical value of the output parameters with the statistical value of output data in the training data of the first model; A sixth threshold, which is used to evaluate a sixth performance metric, and the sixth performance metric is obtained by comparing the output parameters with the result obtained by a preset positioning method.

5. The method according to any one of claims 1 to 4, characterized in that The method further includes: Receiving second indication information sent by the second device, where the second indication information is used to assist the first device in performing performance monitoring.

6. The method according to claim 5, wherein The second indication information includes a ground truth label of a first terminal and / or a statistical value of the training data of the first model, and the ground truth label includes at least one of the following: A measurement value of the first terminal; A location of the first terminal; A timestamp corresponding to the location of the first terminal; A quality indication of the measurement value of the first terminal.

7. The method according to any one of claims 1 to 6, characterized in that The method further includes: Performing performance monitoring on the first model based on at least one of the first request, the second request, the first indication information, and the second indication information.

8. The method according to claim 7, wherein The performing performance monitoring on the first model based on at least one of the first request, the second request, the first indication information, and the second indication information includes at least one of the following: Comparing the input parameters of the first model with the input data in the training data of the first model; Comparing the statistical value of the input parameters of the first model with the statistical value of the input data in the training data of the first model; Compare the current input parameters of the first model with the previous input parameters of the first model; Compare the statistical value of the output parameters of the first model with the statistical value of the output data in the training data of the first model; Compare the current output parameters of the first model with the previous output parameters of the first model; Compare the statistical value of the current output parameters of the first model with the statistical value of the previous output parameters of the first model; Compare the output parameters of the first model with the result obtained by the preset positioning method.

9. The method according to claim 8, wherein The method further includes: Receiving third indication information sent by the second device, where the third indication information is used to indicate the preset positioning method.

10. The method according to any one of claims 2 to 9, characterized in that, The input parameters of the first model include channel measurement related parameters and / or reference signal measurement results, and the channel measurement related parameters and / or the reference signal measurement results include at least one of the following: Channel Impulse Response (CIR); Power Delay Profile (PDP); Delay Profile (DP); Signal to Interference plus Noise Ratio of the positioning reference signal (PRS-SINR); Received Signal Strength of the positioning reference signal (PRS-RSRP); Received Signal Strength of the Synchronization Signal Block (SSB-RSRP); Signal to Interference plus Noise Ratio of the Synchronization Signal Block (SSB-SINR); Reference Signal Time Difference (RSTD); UE Receive-Transmit Time Difference; gNB Receive-Transmit Time Difference; Signal to Interference plus Noise Ratio of the uplink positioning reference signal (SRS-SINR); Received Signal Strength of the uplink positioning reference signal (SRS-RSRP).

11. The method according to any one of claims 2 to 10, characterized in that, The output parameters include at least one of the following: Terminal location; Line-of-Sight Path Indication (Los indication); Non-Line-of-Sight Path Indication (NLos indication); Time of Flight; Measurement quantity based on the positioning reference signal.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: Receiving a third request sent by the second device, where the third request is used to request the first device to send the performance metrics and / or performance monitoring results.

13. The method according to claim 12, wherein The method further includes: Based on the third request, sending the performance metrics and / or performance monitoring results to the second device.

14. The method according to any one of claims 1 to 13, characterized in that, The method further includes: Sending fourth indication information to the second device, where the fourth indication information is used to indicate the second device to determine the performance monitoring results of the first model according to the performance metrics.

15. The method according to any one of claims 1 to 14, characterized in that The method further includes: Sending fifth indication information to the second device, where the fifth indication information is used to indicate the statistical value of the training data of the first model.

16. The method according to any one of claims 1 to 15, characterized in that, The performance metrics include at least one of the following: Channel Impulse Response (CIR); Power Delay Profile (PDP); Delay Profile (DP); Signal to Interference plus Noise Ratio of the positioning reference signal (PRS-SINR); Received Signal Strength of the positioning reference signal (PRS-RSRP); Received Signal Strength of the Synchronization Signal Block (SSB-RSRP); Signal to Interference plus Noise Ratio of the Synchronization Signal Block (SSB-SINR); Reference Signal Time Difference (RSTD); UE receives the Rx - Tx time difference; gNB receives the Rx - Tx time difference; The signal - to - interference - plus - noise ratio of the uplink positioning reference signal, SRS - SINR; The received signal strength of the uplink positioning reference signal, SRS - RSRP; The terminal position obtained through the first model; The terminal position obtained through a preset positioning method; Time of flight.

17. The method according to any one of claims 1 to 16, characterized in that, The performance monitoring result includes at least one of the following: The first model is no longer applicable; It is expected to apply the second model; Model update indication; The performance metric for determining that the first model is no longer applicable; A method for model performance monitoring.

18. The method according to any one of claims 1 to 17, characterized in that, The first model includes at least one of the following: The model currently used by the first device; The model deployed on the first device and not yet used; The model running in the first device; The model deployed on the first device and not yet running.

19. A method for monitoring model performance, characterized in that, The method is executed by a second device, and the method includes: Sending a first request to the first device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring includes determining performance metrics and / or performance monitoring results.

20. The method according to claim 19, wherein The method further includes: Sending a second request to the first device, the second request being used to indicate the information used by the first device when determining the performance metrics and / or the performance monitoring results.

21. The method according to claim 20, wherein The information includes the input parameters and / or output parameters of the first model.

22. The method according to any one of claims 19 to 21, characterized in that, The method further includes: Sending a first indication information to the first device, the first indication information being used to indicate the threshold for determining the performance monitoring result, and the first indication information includes at least one of the following: A first threshold, the first threshold being used to evaluate a first performance metric, and the first performance metric is obtained by comparing the input parameters with the input data in the training data of the first model; A second threshold, the second threshold being used to evaluate a second performance metric, and the second performance metric is obtained by comparing the current input parameters with the prior input parameters; A third threshold, the third threshold being used to evaluate a third performance metric, and the third performance metric is obtained by comparing the statistical value of the output parameters with the statistical value of the output data in the training data of the first model; A fourth threshold, the fourth threshold being used to evaluate a fourth performance metric, and the fourth performance metric is obtained by comparing the current output parameters with the prior output parameters; A fifth threshold, the fifth threshold being used to evaluate a fifth performance metric, and the fifth performance metric is obtained by comparing the statistical value of the output parameters with the statistical value of the output data in the training data of the first model; A sixth threshold, the sixth threshold being used to evaluate a sixth performance metric, and the sixth performance metric is obtained by comparing the output parameters with the result obtained by a preset positioning method.

23. The method according to any one of claims 19 to 22, characterized in that The method further includes: Sending a second indication information to the first device, the second indication information being used to assist the first device in performing performance monitoring.

24. The method according to claim 23, wherein The second indication information includes the geographical ground truth label of the first terminal and / or the statistical value of the training data of the first model, and the geographical ground truth label includes at least one of the following: The measurement value of the first terminal; The location of the first terminal; The timestamp corresponding to the location of the first terminal; The quality indication of the measurement value of the first terminal.

25. The method according to any one of claims 19 to 24, characterized in that, The method further includes: Sending third indication information to the first device, where the third indication information is used to indicate a preset positioning method.

26. The method according to any one of claims 20 to 25, characterized in that, The input parameters of the first model include channel measurement related parameters and / or reference signal measurement results, and the channel measurement related parameters and / or the reference signal measurement results include at least one of the following: Channel Impulse Response (CIR); Power Delay Profile (PDP); Delay Profile (DP); Signal-to-Interference-plus-Noise Ratio of the positioning reference signal (PRS-SINR); Reference Signal Received Power of the positioning reference signal (PRS-RSRP); Reference Signal Received Power of the Synchronization Signal Block (SSB-RSRP); Signal-to-Interference-plus-Noise Ratio of the Synchronization Signal Block (SSB-SINR); Reference Signal Time Difference (RSTD); UE Receive-Transmit Time Difference; gNB Receive-Transmit Time Difference; Signal-to-Interference-plus-Noise Ratio of the uplink positioning reference signal (SRS-SINR); Received Power of the uplink positioning reference signal (SRS-RSRP).

27. The method according to any one of claims 20 to 26, characterized in that, The output parameters include at least one of the following: Terminal location; Line-of-Sight Path Indication (Los indication); Non-Line-of-Sight Path Indication (NLos indication); Time of Flight; Measurement value based on the positioning reference signal.

28. The method according to any one of claims 19 to 27, characterized in that, The method further includes: Sending a third request to the first device, where the third request is used to request the first device to send the performance metrics and / or performance monitoring results.

29. The method according to claim 28, wherein The method further includes: receiving the performance metrics and / or performance monitoring results sent by the first device.

30. The method according to any one of claims 19 to 29, characterized in that The method further includes: Receiving fourth indication information sent by the first device, where the fourth indication information is used to indicate that the second device determines the performance monitoring result of the first model according to the performance metrics; Determining the performance monitoring result based on the fourth indication information.

31. The method according to any one of claims 19 to 30, characterized in that, The method further includes: Receiving fifth indication information sent by the first device, where the fifth indication information is used to indicate the statistical value of the training data of the first model.

32. The method according to any one of claims 19 to 31, characterized in that The performance metrics include at least one of the following: Channel Impulse Response (CIR); Power Delay Profile (PDP); Delay Profile (DP); Signal-to-Interference-plus-Noise Ratio of the positioning reference signal (PRS-SINR); Reference Signal Received Power of the positioning reference signal (PRS-RSRP); Reference Signal Received Power of the Synchronization Signal Block (SSB-RSRP); Signal-to-Interference-plus-Noise Ratio of the Synchronization Signal Block (SSB-SINR); Reference Signal Time Difference (RSTD); UE Receive-Transmit Time Difference; gNB Rx-Tx time difference SRS-SINR (Signal to Interference plus Noise Ratio of the uplink positioning reference signal) SRS-RSRP (Received Signal Strength of the uplink positioning reference signal) The terminal position obtained through the first model The terminal position obtained through a preset positioning method Time of Flight 33. The method according to any one of claims 19 to 32, characterized in that, The performance monitoring result includes at least one of the following: The first model is no longer applicable It is expected to apply the second model Model update indication The performance metrics for determining that the first model is no longer applicable A method for model monitoring 34. The method according to any one of claims 19 to 33, characterized in that, The first model includes at least one of the following: The model currently used by the first device The model deployed on the first device and not yet used The model running in the first device The model deployed on the first device and not yet running 35. A first device, characterized in that, Comprising a transceiver module for: Receiving a first request sent by a second device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring including determining performance metrics and / or performance monitoring results 36. A second device, characterized in that, Comprising a transceiver module for: Sending a first request to a first device, the first request being used to request the first device to perform performance monitoring on a first model, the first model being deployed on the first device, and the performance monitoring including determining performance metrics and / or performance monitoring results 37. A communication device, characterized in that, Comprising: One or more processors Wherein, the one or more processors are used to call instructions to cause the communication device to execute the method according to any one of claims 1-34 38. A communication system, characterized in that, Comprising a first device and a second device, wherein the first device is configured to implement the method according to any one of claims 1-18, and the second device is configured to implement the method according to any one of claims 19-34 39. A storage medium, the storage medium stores instructions, characterized in that, When the instructions run on the communication device, the communication device is caused to execute the method according to any one of claims 1-34

Images