Model performance monitoring methods and apparatuses, devices, system, and storage medium

Abstract

The present disclosure relates to model performance monitoring methods and apparatuses, devices, a system, and a storage medium. A method comprises: sending a first message to a second network element, wherein the first message is used for requesting first data from the second network element; and receiving a second message sent by the second network element, wherein the second message comprises the first data, and the first data is used for implementing model performance monitoring of a first model. By means of the solution of the present disclosure, the first data can be efficiently obtained, so as to implement model performance monitoring of the first model.

Description

Model performance monitoring method and apparatus, device, system, and storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and particularly relates to a model performance monitoring method and apparatus, device, system, and storage medium. BACKGROUND

[0002] With the development of communication technology, artificial intelligence technology can be combined with communication technology to improve the efficiency of communication networks and enhance the functions of communication networks.

[0003] SUMMARY

[0004] The present disclosure provides a model performance monitoring method and apparatus, communication device, communication system, storage medium, and computer program product.

[0005] According to a first aspect of the present disclosure, a model performance monitoring method is provided. The model performance monitoring method is performed by a first network element. The model performance monitoring method comprises: sending a first message to a second network element, wherein the first message is used to request first data from the second network element; receiving a second message sent by the second network element, wherein the second message comprises the first data; and wherein the first data is used to implement model performance monitoring on a first model.

[0006] According to a second aspect of the present disclosure, a model performance monitoring method is provided. The model performance monitoring method is performed by a second network element. The model performance monitoring method comprises: receiving a first message sent by a first network element, wherein the first message is used to request first data from the second network element; sending a second message to the first network element, wherein the second message comprises the first data; and wherein the first data is used to implement model performance monitoring on a first model.

[0007] According to a third aspect of the present disclosure, a model performance monitoring method is provided. The model performance monitoring method is performed by a third network element. The model performance monitoring method comprises: sending a third message to a first network element, wherein the third message is used to trigger the first network element to perform model performance monitoring on a first model.

[0008] According to a fourth aspect of the present disclosure, a model performance monitoring apparatus is provided. The model performance monitoring apparatus is arranged in a first network element. The model performance monitoring apparatus comprises a transceiver module. The transceiver module is configured to: send a first message to a second network element, wherein the first message is used to request first data from the second network element; receive a second message sent by the second network element, wherein the second message comprises the first data; and wherein the first data is used to implement model performance monitoring on a first model.

[0009] According to a fifth aspect of embodiments of the present disclosure, a model performance monitoring apparatus is provided. The model performance monitoring apparatus is arranged at a second network element. The model performance monitoring apparatus comprises a transceiver. The transceiver is configured to receive a first message sent by a first network element, wherein the first message is used to request the second network element for first data; and send a second message to the first network element, wherein the second message comprises the first data; and wherein the first data is used to implement model performance monitoring on a first model.

[0010] According to a sixth aspect of embodiments of the present disclosure, a model performance monitoring apparatus is provided. The model performance monitoring apparatus is arranged at a third network element. The model performance monitoring apparatus comprises a transceiver. The transceiver is configured to send a third message to a first network element, wherein the third message is used to trigger the first network element to perform model performance monitoring on a first model.

[0011] According to a seventh aspect of embodiments of the present disclosure, a communication device is provided. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to the first aspect.

[0012] According to an eighth aspect of embodiments of the present disclosure, a communication device is provided. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to the second aspect.

[0013] According to a ninth aspect of embodiments of the present disclosure, a communication device is provided. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to the third aspect.

[0014] According to a tenth aspect of embodiments of the present disclosure, a communication system is provided. The communication system comprises at least a first network element configured to implement the model performance monitoring method according to the first aspect, and a second network element configured to implement the model performance monitoring method according to the second aspect.

[0015] According to an eleventh aspect of embodiments of the present disclosure, a storage medium is provided. The storage medium stores instructions. The instructions, when executed on a communication device, cause the communication device to perform the model performance monitoring method according to the first aspect, the second aspect or the third aspect.

[0016] According to a twelfth aspect of embodiments of the present disclosure, a program product is provided. The program product, when executed by a communication device, causes the communication device to perform the model performance monitoring method according to the first aspect, the second aspect or the third aspect.

[0017] According to a thirteenth aspect of the embodiments of the present disclosure, a computer program is provided. The computer program, when running on a computer, causes the computer to perform the model performance monitoring method according to the first aspect, the second aspect, or the third aspect.

[0018] According to a fourteenth aspect of the embodiments of the present disclosure, a chip or chip system is provided. The chip or chip system includes processing circuitry. The processing circuitry is configured to perform the model performance monitoring method according to the first aspect, the second aspect, or the third aspect.

[0019] According to the embodiments of the present disclosure, the first network element can obtain relevant data for model performance monitoring of the first model from the second network element.

[0020] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not constitute a limitation on the embodiments of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following describes the drawings required for the embodiment description. The following drawings are only some embodiments of the present disclosure, and do not specifically limit the protection scope of the present disclosure.

[0022] FIG. 1 is an architecture schematic diagram of a communication system according to an embodiment of the present disclosure.

[0023] FIG. 2 is an interaction schematic diagram of model performance monitoring in the related art.

[0024] FIG. 3A is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0025] FIG. 3B is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0026] FIG. 4 is a flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0027] FIG. 5 is a flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0028] FIG. 6 is a flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0029] FIG. 7 is a flow schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0030] FIG. 8A is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0031] FIG. 8B is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0032] FIG. 9 is an interaction schematic diagram of an exemplary implementation of a model performance monitoring method according to an embodiment of the present disclosure.

[0033] FIG. 10 is a structural schematic diagram of a model performance monitoring apparatus according to an embodiment of the present disclosure.

[0034] FIG. 11A is a structural schematic diagram of a communication device according to an embodiment of the present disclosure.

[0035] FIG. 11B is a structural schematic diagram of a chip according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure provide a model performance monitoring method and apparatus, a communication device, a communication system, a storage medium and a computer program product.

[0037] In a first aspect, embodiments of the present disclosure provide a model performance monitoring method. The model performance monitoring method is performed by a first network element. The model performance monitoring method comprises: sending a first message to a second network element, wherein the first message is used to request first data from the second network element; receiving a second message sent by the second network element, wherein the second message comprises the first data; and wherein the first data is used to implement model performance monitoring on a first model.

[0038] Through the present embodiment, the first network element can send the first message to the second network element to request the first data, and the second network element sends the second message containing the first data to the first network element. In this way, the first network element can obtain the first data used for model performance monitoring of the first model. In this way, the first network element can obtain the first data used for model performance monitoring of the first model through interaction with the second network element, thereby effectively obtaining the first data used for model performance monitoring.

[0039] In some embodiments in combination with the first aspect, in some embodiments, the first model can comprise at least one of: an AI model; an ML model.

[0040] In some embodiments in combination with the first aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0041] Through the present embodiment, the first data obtained by the first network element can be used to implement model performance monitoring on the first model, and the first model is used to implement AI / ML-based positioning. In this way, based on the first data, model performance monitoring of the first model used for AI / ML-based positioning can be implemented.

[0042] In some embodiments of the first aspect, in some embodiments, the first message can comprise at least one of: indication information, used to indicate the model performance monitoring for the first model; identification information of the first model; region information, used to indicate a first region in which the first model implements positioning.

[0043] Through the embodiment, the first message can indicate the second network element to collect the first data for the model performance monitoring of the first model through at least one of the indication information, the identification information, and the region information. In this way, the first network element can obtain the required first data from the second network element.

[0044] In some embodiments of the first aspect, in some embodiments, the first data can comprise at least one of: measurement data; a calculated position based on the measurement data; an actual position.

[0045] In some embodiments of the first aspect, in some embodiments, the first data can be associated with a first region in which the first model implements positioning.

[0046] In some embodiments of the first aspect, in some embodiments, the first data can be authorized data.

[0047] Through the embodiment, the first data obtained by the first network element from the second network element is authorized data, and thus the first network element can directly use the first data to implement the model performance monitoring of the first model.

[0048] In some embodiments of the first aspect, in some embodiments, the first data can be data pre-stored in the second network element.

[0049] Through the embodiment, the first data can be pre-stored in the second network element. After receiving the first message from the first network element, the second network element can send the pre-stored first data to the first network element through the second message. As the first network element interacts with the second network element, the first network element can obtain the first data for the model performance monitoring of the first model. In this way, through a small amount of interaction, the first network element can obtain the first data, reducing the communication load of the communication system and improving the efficiency of obtaining the first data.

[0050] In some embodiments of the first aspect, in some embodiments, the second network element can comprise at least one of: an ADRF; a UDM function; a UDR function.

[0051] In some embodiments of the first aspect, in some embodiments, the method can further comprise: receiving a third message sent by a third network element, wherein the third message is used to trigger the first network element to perform the model performance monitoring.

[0052] In some embodiments combined with the first aspect, in some embodiments, the third message can include at least one of the following: identification information of the first model; and region information indicating a first region in which the first model implements positioning.

[0053] In some embodiments combined with the first aspect, in some embodiments, the method can further include: determining a model performance of the first model based on the first data; and performing a first operation related to the first model according to the model performance.

[0054] In some embodiments combined with the first aspect, in some embodiments, the first operation can include at least one of the following: sending a fourth message to a third network element; sending the fourth message to a fourth network element, wherein the fourth network element has the same function as the first network element; and training the first model.

[0055] In some embodiments combined with the first aspect, in some embodiments, the fourth message can be used for at least one of the following: indicating the model performance of the first model; triggering a change of a positioning method; and triggering training of the first model.

[0056] In a second aspect, the embodiments of the present disclosure provide a model performance monitoring method. The model performance monitoring method is performed by a second network element. The model performance monitoring method includes: receiving a first message sent by a first network element, wherein the first message is used to request first data from the second network element; and sending a second message to the first network element, wherein the second message includes the first data; and wherein the first data is used to implement model performance monitoring of a first model.

[0057] Through the present embodiment, the first network element can send the first message to the second network element to request the first data, and the second network element sends the second message including the first data to the first network element. In this way, the first network element can obtain the first data used for model performance monitoring of the first model. In this way, the first network element can obtain the first data used for model performance monitoring of the first model through interaction with the second network element, thereby effectively obtaining the first data used for model performance monitoring.

[0058] In some embodiments combined with the second aspect, in some embodiments, the first model can include at least one of the following: an AI model; and an ML model.

[0059] In some embodiments combined with the second aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0060] In some embodiments combined with the second aspect, in some embodiments, the first message can include at least one of the following: indication information indicating model performance monitoring of the first model; identification information of the first model; and region information indicating a first region in which the first model implements positioning.

[0061] In some embodiments combining with the second aspect, in some embodiments, the first data can comprise at least one of: measurement data; a calculated position based on the measurement data; an actual position.

[0062] In some embodiments combining with the second aspect, in some embodiments, the first data can be associated with a first area in which the first model implements positioning.

[0063] In some embodiments combining with the second aspect, in some embodiments, the first data can be authorized data.

[0064] In some embodiments combining with the second aspect, in some embodiments, the first data can be data pre-stored in the second network element.

[0065] In some embodiments combining with the second aspect, in some embodiments, the second network element can comprise at least one of: an ADRF; a UDM function; a UDR function.

[0066] In a third aspect, the embodiments of the present disclosure provide a model performance monitoring method. The model performance monitoring method is performed by a third network element. The above-mentioned model performance monitoring method comprises: sending a third message to a first network element, wherein the third message is used to trigger the first network element to perform model performance monitoring on a first model.

[0067] Through the present embodiment, the third network element can trigger the first network element to perform model performance monitoring on the first model by sending the third message. In this way, the first network element can perform model performance monitoring on the first model under the triggering of the third network element, so as to obtain the model performance of the first model.

[0068] In some embodiments combining with the third aspect, in some embodiments, the third message can comprise at least one of: identification information of the first model; area information used to indicate a first area in which the first model implements positioning.

[0069] In some embodiments combining with the third aspect, in some embodiments, the first model can comprise at least one of: an AI model; an ML model.

[0070] In some embodiments combining with the third aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0071] In some embodiments combining with the third aspect, in some embodiments, the model performance monitoring on the first model by the first network element can be implemented based on first data obtained from a second network element.

[0072] In some embodiments combining with the third aspect, in some embodiments, the first data can comprise at least one of: measurement data; a calculated position based on the measurement data; an actual position.

[0073] In some embodiments combining with the third aspect, in some embodiments, the first data can be associated with a first area in which the first model implements positioning.

[0074] In some embodiments combining with the third aspect, in some embodiments, the first data can be authorized data.

[0075] In some embodiments combining with the third aspect, in some embodiments, the first data can be data pre-stored in the second network element.

[0076] In some embodiments combining with the third aspect, in some embodiments, the second network element can include at least one of the following: ADRF; UDM function; UDR function.

[0077] In some embodiments combining with the third aspect, in some embodiments, the method can further include: receiving a fourth message sent by the first network element according to the model performance of the first model, wherein the fourth message is used for at least one of the following: indicating the model performance of the first model; triggering the third network element to change the positioning method; triggering the third network element to train the first model.

[0078] In a fourth aspect, the embodiments of the present disclosure provide a model performance monitoring apparatus. The model performance monitoring apparatus is arranged in a first network element. The model performance monitoring apparatus includes a transceiver module. The transceiver module is configured to: send a first message to a second network element, wherein the first message is used to request first data from the second network element; receive a second message sent by the second network element, wherein the second message includes the first data; and wherein the first data is used to implement model performance monitoring on a first model.

[0079] In some embodiments combining with the fourth aspect, in some embodiments, the first model can include at least one of the following: an AI model; an ML model.

[0080] In some embodiments combining with the fourth aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0081] In some embodiments combining with the fourth aspect, in some embodiments, the first message can include at least one of the following: indication information, used to indicate model performance monitoring on the first model; identification information of the first model; area information, used to indicate a first area in which the first model implements positioning.

[0082] In some embodiments combining with the fourth aspect, in some embodiments, the first data can include at least one of the following: measurement data; calculated position based on the measurement data; actual position.

[0083] In some embodiments of the fourth aspect, in some embodiments, the first data can be associated with a first area in which the first model is located.

[0084] In some embodiments of the fourth aspect, in some embodiments, the first data can be authorized data.

[0085] In some embodiments of the fourth aspect, in some embodiments, the first data can be data pre-stored in the second network element.

[0086] In some embodiments of the fourth aspect, in some embodiments, the second network element can include at least one of the following: an ADRF; a UDM function; a UDR function.

[0087] In some embodiments of the fourth aspect, in some embodiments, the transceiver module can be further configured to: receive a third message sent by a third network element, wherein the third message is used to trigger the first network element to perform model performance monitoring.

[0088] In some embodiments of the fourth aspect, in some embodiments, the third message can include at least one of the following: identification information of the first model; area information used to indicate a first area in which the first model is located.

[0089] In some embodiments of the fourth aspect, in some embodiments, the apparatus can further include a processing module. The processing module is configured to: determine a model performance of the first model based on the first data; and perform a first operation related to the first model according to the model performance.

[0090] In some embodiments of the fourth aspect, in some embodiments, the first operation can include at least one of the following: sending a fourth message to a third network element through the transceiver module; sending a fourth message to a fourth network element through the transceiver module, wherein the fourth network element has the same function as the first network element; and training the first model.

[0091] In some embodiments of the fourth aspect, in some embodiments, the fourth message can be used for at least one of the following: indicating the model performance of the first model; triggering a change in the positioning method; and triggering training of the first model.

[0092] In a fifth aspect, the embodiments of the present disclosure provide a model performance monitoring apparatus. The model performance monitoring apparatus is arranged in a second network element. The model performance monitoring apparatus includes a transceiver module. The transceiver module is configured to: receive a first message sent by a first network element, wherein the first message is used to request first data from the second network element; and send a second message to the first network element, wherein the second message includes the first data; and wherein the first data is used to implement model performance monitoring of a first model.

[0093] In some embodiments combining with the fifth aspect, in some embodiments, the first model can include at least one of: an AI model; an ML model.

[0094] In some embodiments combining with the fifth aspect, in some embodiments, the first model can be used to implement AI / ML based positioning.

[0095] In some embodiments combining with the fifth aspect, in some embodiments, the first message can include at least one of: indication information used to indicate model performance monitoring for the first model; identification information of the first model; area information used to indicate a first area in which the first model implements positioning.

[0096] In some embodiments combining with the fifth aspect, in some embodiments, the first data can include at least one of: measurement data; calculated position based on the measurement data; actual position.

[0097] In some embodiments combining with the fifth aspect, in some embodiments, the first data can be associated with a first area in which the first model implements positioning.

[0098] In some embodiments combining with the fifth aspect, in some embodiments, the first data can be authorized data.

[0099] In some embodiments combining with the fifth aspect, in some embodiments, the first data can be data pre-stored in the second network element.

[0100] In some embodiments combining with the fifth aspect, in some embodiments, the second network element can include at least one of: an ADRF; a UDM function; a UDR function.

[0101] In a sixth aspect, the embodiments of the present disclosure provide a model performance monitoring apparatus. The model performance monitoring apparatus is arranged in a third network element. The model performance monitoring apparatus includes a transceiver module. The transceiver module is configured to: send a third message to a first network element, wherein the third message is used to trigger the first network element to perform model performance monitoring on a first model.

[0102] In some embodiments combining with the sixth aspect, in some embodiments, the third message can include at least one of: identification information of the first model; area information used to indicate a first area in which the first model implements positioning.

[0103] In some embodiments combining with the sixth aspect, in some embodiments, the first model can include at least one of: an AI model; an ML model.

[0104] In some embodiments combining with the sixth aspect, in some embodiments, the first model can be used to implement AI / ML based positioning.

[0105] In some embodiments combining with the sixth aspect, in some embodiments, the model performance monitoring of the first model by the first network element can be implemented based on the first data obtained from the second network element.

[0106] In some embodiments combining with the sixth aspect, in some embodiments, the first data can comprise at least one of: measurement data; calculated position based on the measurement data; actual position.

[0107] In some embodiments combining with the sixth aspect, in some embodiments, the first data can be associated with a first area in which the first model implements positioning.

[0108] In some embodiments combining with the sixth aspect, in some embodiments, the first data can be authorized data.

[0109] In some embodiments combining with the sixth aspect, in some embodiments, the first data can be data pre-stored in the second network element.

[0110] In some embodiments combining with the sixth aspect, in some embodiments, the second network element can comprise at least one of: ADRF; UDM function; UDR function.

[0111] In some embodiments combining with the sixth aspect, in some embodiments, the transceiver module can be further configured to: receive a fourth message sent by the first network element according to the model performance of the first model, wherein the fourth message is used for at least one of: indicating the model performance of the first model; triggering the third network element to change the positioning method; triggering the third network element to train the first model.

[0112] In a seventh aspect, the embodiments of the present disclosure provide a communication device. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to any one of the first aspect and possible implementation manners thereof.

[0113] In an eighth aspect, the embodiments of the present disclosure provide a communication device. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to any one of the second aspect and possible implementation manners thereof.

[0114] In a ninth aspect, the embodiments of the present disclosure provide a communication device. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to any one of the third aspect and possible implementation manners thereof.

[0115] In a tenth aspect, an embodiment of the present disclosure provides a communication system. The communication system comprises at least: a first network element configured to implement the model performance monitoring method according to any one of the first aspect and possible implementation manners thereof; and a second network element configured to implement the model performance monitoring method according to any one of the second aspect and possible implementation manners thereof.

[0116] In some embodiments in combination with the tenth aspect, the communication system can further comprise: a third network element configured to implement the model performance monitoring method according to any one of the third aspect and possible implementation manners thereof.

[0117] In an eleventh aspect, an embodiment of the present disclosure provides a storage medium. The storage medium stores instructions. The instructions, when executed on a communication device, cause the communication device to perform the model performance monitoring method according to any one of the first aspect to the third aspect and possible implementation manners thereof.

[0118] In a twelfth aspect, an embodiment of the present disclosure provides a program product. The program product, when executed by a communication device, causes the communication device to perform the model performance monitoring method according to any one of the first aspect to the third aspect and possible implementation manners thereof.

[0119] In a thirteenth aspect, an embodiment of the present disclosure provides a computer program (product). The computer program, when executed on a computer, causes the computer to perform the model performance monitoring method according to any one of the first aspect to the third aspect and possible implementation manners thereof.

[0120] In a fourteenth aspect, an embodiment of the present disclosure provides a chip or chip system. The chip or chip system comprises processing circuitry. The processing circuitry is configured to perform the model performance monitoring method according to any one of the first aspect to the third aspect and possible implementation manners thereof.

[0121] It can be understood that the above model performance monitoring apparatus, communication device, communication system, storage medium, program product, computer program, chip, and chip system are all used to perform the model performance monitoring method provided by the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved thereby can refer to the beneficial effects in the corresponding method, which will not be described here again.

[0122] The embodiments of the present disclosure provide a model performance monitoring method and apparatus, a communication device, a communication system, a storage medium, and a computer program product. In some embodiments, the terms of model performance monitoring method, communication method, information processing method, and the like can be replaced with each other, and the terms of model performance monitoring apparatus, communication apparatus, communication device, information processing apparatus, and the like can be replaced with each other, and the terms of information processing system, communication system, and the like can be replaced with each other.

[0123] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily. In addition, the optional implementation manners in an embodiment can be combined arbitrarily. In addition, the embodiments can be combined arbitrarily. For example, part or all steps of different embodiments can be combined arbitrarily. For another example, an embodiment can be combined with optional implementation manners of other embodiments arbitrarily.

[0124] In each embodiment of the present disclosure, the terms and / or descriptions between the embodiments are consistent if there is no special description and logical conflict, and can be referred to each other, and the technical features in different embodiments can be combined to form a new embodiment according to the inherent logical relationship.

[0125] The terms used in the embodiments of the present disclosure are only for the purpose of describing the specific embodiments, and not as a limitation on the present disclosure.

[0126] In the embodiments of the present disclosure, unless otherwise specified, the elements expressed in singular form, such as "one", "a", "the", "above", "said", "preceding", "this" and the like, can represent "one and only one", and can also represent "one or more", "at least one" and the like. For example, in the case of using articles such as "a", "an", "the" and the like in English, the noun after the article can be understood as singular expression, and can also be understood as plural expression.

[0127] In the embodiments of the present disclosure, "a plurality of" means two or more.

[0128] In some embodiments, the terms "at least one of", "one or more", "a plurality of", "multiple" and the like can be replaced with each other.

[0129] In some embodiments, "at least one of A, B", "A and / or B", "in one case A, in another case B", "responsive to case A, responsive to case B" and the like, can be used to represent one or more of the following technical solutions: in some embodiments, A (A is executed regardless of B); in some embodiments, B (B is executed regardless of A); in some embodiments, A and B are selected from (A and B are selectively executed); in some embodiments, A and B (A and B are executed). When there are more branches such as A, B, C, and the like, the above is similar.

[0130] In some embodiments, "A or B" and the like can include the following technical solutions according to the case: in some embodiments, A (A is executed regardless of B); in some embodiments, B (B is executed regardless of A); in some embodiments, A and B are selected from (A and B are selectively executed). When there are more branches such as A, B, C, and the like, the above is similar.

[0131] In the embodiments of the present disclosure, the prefix words "first", "second" and the like are only used to distinguish different description objects, and do not constitute a limitation on the position, order, priority, quantity or content of the description objects. The description of the description objects should be referred to the description in the context of the claims or embodiments, and should not be limited by the prefix words. For example, the description object is "field", and the ordinal words before "field" in "first field" and "second field" do not limit the position or order between "fields", and "first" and "second" do not limit whether the "fields" modified thereby are in the same message or not, nor limit the order of "first field" and "second field". For another example, the description object is "level", and the ordinal words before "level" in "first level" and "second level" do not limit the priority between "levels". For another example, the quantity of the description object is not limited by the ordinal words, and can be one or more. For example, "first device", wherein the quantity of "device" can be one or more. In addition, the objects modified by different prefix words can be the same or different, for example, the description object is "device", and "first device" and "second device" can be the same device or different devices, and the types thereof can be the same or different; for another example, the description object is "information", and "first information" and "second information" can be the same information or different information, and the content thereof can be the same or different.

[0132] In some embodiments, "including A", "containing A", "for indicating A", "carrying A" can be interpreted as directly carrying A, or indirectly indicating A.

[0133] In some embodiments, the terms "in response to", "in response to determining", "in the case of", "when", "when", "if", "if" and the like can be replaced with each other.

[0134] In some embodiments, the terms "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not lower than", "above", and the like can be replaced with each other, and the terms "less than", "less than or equal to", "not greater than", "fewer than", "fewer than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", "below", and the like can be replaced with each other.

[0135] In some embodiments, an apparatus and the like can be interpreted as an entity, and can also be interpreted as virtual, and the name thereof is not limited to the name recited in the embodiments, and the terms "apparatus", "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject", and the like can be replaced with each other.

[0136] In some embodiments, "network" can be interpreted as an apparatus (for example, an access network device, a core network device, and the like) included in the network.

[0137] In some embodiments, the terms “access network device (AN device),” “radio access network device (RAN device),” “base station (BS),” “radio base station,” “fixed station,” “node,” “access point,” “transmission point (TP),” “reception point (RP),” “transmission / reception point (TRP),” “panel,” “antenna panel,” “antenna array,” “cell,” “macro cell,” “small cell,” “femto cell,” “pico cell,” “sector,” “cell group,” “serving cell,” “carrier,” “component carrier,” “bandwidth part (BWP),” and the like can be used interchangeably.

[0138] In some embodiments, the terms "terminal," "terminal device," "user equipment (UE)," "user terminal," "mobile station (MS)," "mobile terminal (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile terminal," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," and so on can be replaced with each other.

[0139] In some embodiments, the access network device, the core network device, or the network device can be replaced with a terminal. For example, the embodiments of the present disclosure can also be applied to a structure in which communication between the access network device, the core network device, or the network device and the terminal is replaced with communication between a plurality of terminals (e.g., device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, the terminal can also be configured to have all or part of the functions of the access network device. In addition, the terms "uplink," "downlink," and the like can also be replaced with terms corresponding to the inter-terminal communication (e.g., "side"). For example, the uplink channel, the downlink channel, and the like can be replaced with the side channel, and the uplink, the downlink, and the like can be replaced with the sidelink.

[0140] In some embodiments, the terminal can be replaced with the access network device, the core network device, or the network device. In this case, the access network device, the core network device, or the network device can also be configured to have all or part of the functions of the terminal.

[0141] In some embodiments, the data, information, etc. can be obtained in compliance with the laws and regulations of the country where the location is situated.

[0142] In some embodiments, the data, information, etc. can be obtained after obtaining the consent of the user.

[0143] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0144] FIG. 1 is a schematic diagram of an architecture of a communication system according to an embodiment of the present disclosure. As shown in FIG. 1, the communication system 100 includes a first network element 101, a second network element 102, a third network element 103, and a fourth network element 104.

[0145] In some embodiments, the first network element 101 can be configured to perceive and analyze the network based on network data, and improve user service experience.

[0146] In some embodiments, the first network element 101 can be a network data analytics function (NWDAF).

[0147] In some embodiments, the second network element 102 can be configured to implement storage of data.

[0148] In some embodiments, the second network element 102 can be an analytics data repository function (ADRF).

[0149] In some embodiments, the second network element 102 can be a unified data management (UDM) function.

[0150] In some embodiments, the second network element 102 can be a unified data repository (UDR) function.

[0151] In some embodiments, the third network element 103 can be configured to provide positioning services for terminals.

[0152] In some embodiments, the third network element 103 can be a location management function (LMF).

[0153] In some embodiments, the fourth network element 104 can be configured to perceive and analyze the network based on network data, and improve user service experience.

[0154] In some embodiments, the fourth network element 104 can be a NWDAF.

[0155] In some embodiments, the first network element 101 and the fourth network element 104 can have the same function, but the first network element 101 and the fourth network element 104 are different network elements.

[0156] In some embodiments, the communication system 100 can be a 5G communication system. In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be a network element in the 5G communication system. In some embodiments, the communication system 100 can be a 6G communication system. In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be a network element in the 6G communication system. It should be noted that the communication system 100 can also be other communication systems, for example, a 4G communication system, a 7G communication system, and the like, and the present embodiments do not make a specific limitation thereon.

[0157] In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be a core network device. In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be an access network device.

[0158] In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be a control plane network function. In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be a user plane network function. In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be a user plane network function. In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be a data plane network function. In some embodiments, at least one of the first network element 101, the second network element 102, the third network element 103, and the fourth network element 104 can be a computing plane network function.

[0159] In some embodiments, the terminal includes at least one of a mobile phone, a wearable device, an Internet of Things device, a communication-capable automobile, a smart automobile, a tablet (Pad), a wireless transceiver-equipped computer, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in a smart grid, a wireless terminal device in transportation safety, a wireless terminal device in a smart city, a wireless terminal device in a smart home, and the like, but is not limited thereto.

[0160] In some embodiments, the access network device is at least one of a node or a device that accesses a terminal to a wireless network, and can include an evolved NodeB (eNB) in a 5G communication system, a next generation eNB (ng-eNB), a next generation NodeB (gNB), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an Open RAN, a Cloud RAN, a base station in other communication systems, an access node in a Wi-Fi system, and the like, but is not limited thereto.

[0161] In some embodiments, the technical solutions of the present disclosure can be applied to an Open RAN architecture, in which case, the interfaces between or within the access network devices involved in the embodiments of the present disclosure can become internal interfaces of the Open RAN, and the processes and information interactions between these internal interfaces can be implemented through software or programs.

[0162] In some embodiments, the access network device can be composed of a central unit (CU) and a distributed unit (DU), where the CU can also be referred to as a control unit. The CU-DU structure can split the protocol layers of the access network device, with some protocol layer functions being centrally controlled by the CU and the remaining protocol layer functions being distributed in the DUs, which are centrally controlled by the CU, but the present disclosure is not limited thereto.

[0163] It can be understood that the communication system described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and does not constitute a limitation on the technical solutions proposed in the embodiments of the present disclosure. It can be known by those skilled in the art that, as the system architecture evolves and new service scenarios appear, the technical solutions proposed in the embodiments of the present disclosure are also applicable to similar technical problems.

[0164] The following embodiments of the present disclosure can be applied to the communication system 100 shown in FIG. 1 or part of the subjects in the communication system 100, but are not limited thereto. The subjects shown in FIG. 1 are exemplary, the communication system 100 can include all or part of the subjects in FIG. 1, or include other subjects other than FIG. 1, the number and form of each subject is arbitrary, each subject can be real or virtual, the connection relationship between each subject is exemplary, each subject can not be connected or can be connected, the connection can be in any way, can be direct connection or indirect connection, can be wired connection or wireless connection.

[0165] Embodiments of the present disclosure can be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G new radio (NR), 6th generation mobile communication system (6G), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New radio access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (Bluetooth (registered trademark)), Public Land Mobile Network (PLMN) network, Device-to-Device (D2D) system, Machine to Machine (M2M) system, Internet of Things (IoT) system, Vehicle-to-Everything (V2X), system using other communication methods, next-generation system expanded based thereon, and the like. Further, a plurality of systems can be applied in combination (for example, a combination of 5G and 6G, and the like).

[0166] In some embodiments, a communication system can support a location service (LCS). The communication system can provide location information of a terminal for a requestor of the location service. The requestor of the location service can be an application function (AF), a consumer network function (NF), a terminal, etc.

[0167] In some embodiments, AI technology can be integrated with communication technology, thereby enhancing the capability of a communication system. In particular, a communication system can implement a location service through AI technology. In some embodiments, an AI / machine learning (ML) model can be deployed in a communication system. In some embodiments, the AI / ML model can be deployed in an LMF of the communication system. The LMF can utilize the AI / ML model to provide a location service. In an example, the LMF can perform positioning inference based on the AI / ML model and provide an inference result to a requestor. In some embodiments, a model training logic function (MTLF) can also be deployed in the communication system. The MTLF can be responsible for training of the AI / ML model.

[0168] In some embodiments, in order to ensure the quality of service of a location service provided based on an AI / ML model (or AI / ML positioning for short), model performance monitoring can be performed on the AI / ML model. In other words, the LMF or the MTLF can perform model performance monitoring on the model-based positioning. In some embodiments, in order to implement model performance monitoring, the LMF can provide inference data related to the inference of the AI / ML model-based positioning to the MTLF for model performance monitoring by the MTLF.

[0169] In some embodiments, the MTLF can be located in a NWDAF. The NWDAF can have an analysis capability and / or an accurate verification capability for the AI / ML model. In an example, model performance monitoring can be performed by the NWDAF.

[0170] FIG. 2 is an interaction diagram of model performance monitoring in the related art. As shown in FIG. 2, the process of model performance monitoring can include steps S201 to S207.

[0171] In step S201, an AF / consumer NF subscribes to an LMF to request AI / ML positioning.

[0172] In step S202, the LMF performs UE location data collection to obtain location data from the UE, RAN, AMF, gateway mobile location center (GMLC), and the like.

[0173] In step S203, the LMF performs direct AI / ML positioning inference using the trained AI / ML model.

[0174] In step S204, including step S204a and step S204b. In step S204a, the LMF notifies the inference result of the AI / ML positioning inference to the AF / consumer NF; and in step S204b, the LMF can send related inference data of the AI / ML positioning inference to the MTLF for monitoring the model performance.

[0175] In step S205, the MTLF performs UE location data collection for model performance monitoring of the AI / ML positioning inference.

[0176] In step S206, the MTLF can obtain the model performance based on the collected UE location data, and can decide to retrain and / or re-provision the AI / ML model by the LMF.

[0177] In step S207, the MTLF can provide the model performance report to the LMF. After the newly generated or retrained AI / ML model is ready, the MTLF can send the AI / ML model to the LMF.

[0178] In order to realize model performance monitoring, the acquisition of UE location data is a key link. Therefore, how to effectively acquire UE location data is a problem to be solved.

[0179] FIG. 3A is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure. The model performance monitoring method related by the embodiment of the present disclosure can be applied to the communication system 100. As shown in FIG. 3A, the model performance monitoring method of the embodiment of the present disclosure includes steps S3101 to S3106.

[0180] In step S3101, the third network element 103 sends a third message to the first network element 101.

[0181] In some embodiments, the first network element 101 can receive the third message.

[0182] In some embodiments, the first network element 101 can be a NWDAF. In some embodiments, the first network element 101 can be a MTLF. In an example, the first network element 101 can be a MTLF located in a NWDAF. It can be understood that the first network element 101 can also be other network functions, which are not specifically limited in the embodiments of the present disclosure.

[0183] In some embodiments, the third network element 103 can be a LMF. It can be understood that the third network element 103 can also be other network functions, which are not specifically limited in the embodiments of the present disclosure.

[0184] In some embodiments, the third message can be used to trigger the first network element 101 to perform model performance monitoring.

[0185] In some embodiments, the model performance monitoring can be determining model performance of the first model. In some embodiments, the third message can be used to request to determine model performance of the first model.

[0186] In some embodiments, the first model can be used to implement a location service. In an example, the first model can be used to implement positioning for one or more terminals. In some embodiments, the location service implemented based on the first model can include at least one of the following: determining a current location of a terminal, predicting a location of a terminal.

[0187] In some embodiments, the first model can include at least one of the following: an AI model, a ML model. In some embodiments, the first model can be used to implement AI / ML based positioning. For example, the AI model can be used to implement AI model based positioning. For example, the ML model can be used to implement ML based positioning.

[0188] In some embodiments, the name of the third message is not limited, which can be, for example, a model performance monitoring request message, a model performance monitoring indication message, etc.

[0189] In some embodiments, the third message can include at least one of the following: identification information of the first model, area information.

[0190] In some embodiments, the identification information of the first model can be used to identify the first model. Through the identification information, the third network element 103 can indicate to perform model performance monitoring on the first model associated with the identification information.

[0191] In some embodiments, the identification information of the first model can include an identifier.

[0192] In some embodiments, the identifier can be used to identify each first model. In an example, one first model can be uniquely identified by an identifier. Different first models can be identified by different identifiers.

[0193] In some embodiments, the identifier can be used to identify a model group composed of one or more first models. In an example, one model group can be uniquely identified by the identifier. Different model groups can be identified by different identifiers.

[0194] In some embodiments, the area information can be used to indicate a first area in which the first model implements positioning.

[0195] In some embodiments, the first area can be an area to which the positioning based on the first model is directed. In some embodiments, the positioning based on the first model can be based on measurements within the first area. For example, the object to be positioned can be located within the first area. For example, the terminal and / or network device used to perform positioning can be located within the first area.

[0196] In some embodiments, the object to be positioned can be a UE.

[0197] In some embodiments, the terminal performing positioning can be a UE. In some embodiments, the terminal performing positioning can be a positioning reference unit (PRU).

[0198] In some embodiments, the network device performing positioning can be an access network device.

[0199] In some embodiments, after receiving the third message, the first network element 101 can determine, according to the third message, to perform model performance monitoring for the first model. In an example, the first network element 101 can determine, according to the identification information and / or the area information in the third message, that the model performance monitoring is for the first model.

[0200] In step S3102, the first network element 101 sends a first message to the second network element 102.

[0201] In some embodiments, in a case where it is determined to perform model performance monitoring for the first model, the first network element 101 can send the first message to the second network element 102.

[0202] In some embodiments, the second network element 102 can receive the first message.

[0203] In some embodiments, the second network element 102 can include at least one of the following: ADRF, UDM, UDR. It can be understood that the second network element 102 can also be other network functions, which are not specifically limited in the embodiments of the present disclosure.

[0204] In some embodiments, the first message can be used to request the second network element 102 for first data. The first data is used to implement performance monitoring of the first model.

[0205] In some embodiments, the first message can be used to request the second network element 102 to provide the first data.

[0206] In some embodiments, the first message can be used to request the second network element 102 to perform data collection for the first model.

[0207] In some embodiments, the name of the first message is not limited, which can be, for example, a data request message, a model performance monitoring indication message, a data collection request message, etc.

[0208] In some embodiments, the first message can comprise at least one of the following: indication information, identification information of the first model, area information.

[0209] In some embodiments, the indication information can be used to indicate model performance monitoring for the first model. In an example, the indication information can be used to indicate model performance monitoring for a model in the first network element 101. In an example, the indication information can be used to indicate model performance monitoring for a model in the third network element 103.

[0210] In some embodiments, there can be only one model, for example, the first model, in the first network element 101. In this case, the first message can only contain the indication information to indicate model performance monitoring. It can be understood that, because there is only one model in the first network element 101, the first message can request the first data for the first model by default.

[0211] In some embodiments, there can be only one model, for example, the first model, in the second network element 102. In this case, the first message can only contain the indication information to indicate model performance monitoring. It can be understood that, because there is only one model in the second network element 102, the first message can request the first data for the first model by default.

[0212] In some embodiments, the first message can further comprise identification information of the first model and / or area information. The identification information of the first model and / or the area information can be acquired by the first network element 101 from the third message and carried in the first message.

[0213] In some embodiments, the first message can contain the identification information of the first model if the third message contains the identification information of the first model. In some embodiments, the first message can not contain the identification information of the first model if the third message contains the identification information of the first model, and if there is only one first model in the first network element 101 and / or the second network element 102.

[0214] In some embodiments, the first message can contain the area information if the third message contains the area information.

[0215] In step S3103, the second network element 102 sends a second message to the first network element 101.

[0216] In some embodiments, the first network element 101 can receive the second message.

[0217] In some embodiments, the second message can be sent by the second network element 102 in response to the first message.

[0218] In some embodiments, the second message can be used to return data for the first model to the first network element 101.

[0219] In some embodiments, the name of the second message is not limited, which can be, for example, a data response message, a model performance monitoring data message, a data collection response message, etc.

[0220] In some embodiments, the second message can include first data.

[0221] In some embodiments, the first data can include at least one of the following: measurement data, a calculated position, an actual position.

[0222] In some embodiments, the measurement data can include measurement data provided by a terminal and / or a network device used to perform positioning.

[0223] In some embodiments, the calculated position can be a position calculated based on measurement data provided by a terminal and / or a network device.

[0224] In some embodiments, the actual position can be an actual position of an object to be positioned. In an example, the actual position can include a position of an object to be positioned determined by a global navigation satellite system (GNSS) or other means. The GNSS used to determine the actual position can include at least one of the following: a global positioning system (GPS), a beidou navigation satellite system (BDS), a Galileo satellite navigation system (Galileo), a global navigation satellite system (GLONASS), a quasi-zenith satellite system (QZSS).

[0225] In some embodiments, the first data can be associated with a first area in which the first model implements positioning. In an example, the first data can include data associated with the first model for an object to be positioned within the first area. In an example, the first data can include data associated with the first model for a terminal and / or a network device to perform positioning within the first area.

[0226] In some embodiments, the first data can be data pre-stored in the second network element 102. In some embodiments, the first data can include pre-stored data for model performance monitoring of the first model. In an example, the first data can be obtained in a process of performing positioning based on the first model and stored in the second network element 102.

[0227] In some embodiments, the first data in the second network element 102 can be stored for each first area. In some embodiments, the first data in the second network element 102 can be stored for the first model and each first area.

[0228] In some embodiments, the first data can be authorized data. In some embodiments, the first data can be data authorized for model performance monitoring. In some embodiments, the first data can be stored in the second network element 102 after being authorized. In some embodiments, the first data can be authorized after being stored in the second network element 102.

[0229] In some embodiments, the second message can further include identification information of the first model. In some embodiments, according to the identification information, the first second network element 102 can indicate that the first data is used for model performance monitoring of the first model.

[0230] In step S3104, the first network element 101 performs model performance monitoring.

[0231] In some embodiments, after receiving the second message, the first network element 101 can perform model performance monitoring based on the obtained first data.

[0232] In some embodiments, the model performance monitoring performed by the first network element 101 can include that the first network element 101 determines model performance of the first model based on the first data.

[0233] In some embodiments, the first network element 101 can evaluate model performance of the first model based on the first data. In this way, the first network element 101 can obtain an evaluation result. The evaluation result can indicate model performance of the first model.

[0234] In some embodiments, the first network element 101 can determine whether the model performance of the first model meets a threshold through model performance monitoring. In some embodiments, the first network element 101 can compare the model performance of the first model with the threshold. The comparison result can comprise one of the following: the model performance is lower than the threshold, the model performance is higher than the threshold.

[0235] In some embodiments, the first network element 101 determines whether the model performance of the first model is good or not. In an example, in a case where the performance result of the first model is higher than the threshold, the first network element 101 can determine that the model performance of the first model is good (or meets the requirement). In an example, in a case where the performance result of the first model is lower than the threshold, the first network element 101 can determine that the model performance of the first model is not good (or does not meet the requirement).

[0236] In step S3105, the first network element 101 performs model training.

[0237] In some embodiments, the first network element 101 can trigger training of the first model according to the model performance.

[0238] In some embodiments, in a case where the model performance of the first model is determined to be lower than the threshold, the first network element 101 can trigger training of the first model. In some embodiments, in a case where the model performance of the first model is determined to be lower than the threshold, the first network element 101 can perform training for the first model.

[0239] In some embodiments, in a case where the model performance of the first model is determined to be not good, the first network element 101 can trigger training of the first model. In some embodiments, in a case where the model performance of the first model is determined to be not good, the first network element 101 can perform training for the first model.

[0240] Through step S3105, training of the first model can be implemented by the first network element 101.

[0241] In step S3106, the first network element 101 sends a fourth message to the third network element 103.

[0242] In some embodiments, the third network element 103 can receive the fourth message.

[0243] In some embodiments, the fourth message can be sent according to the model performance.

[0244] In some embodiments, the fourth message can be sent in response to the third message.

[0245] In some embodiments, the fourth message can be used to inform the third network element 103 of the result of the model performance monitoring.

[0246] In some embodiments, the fourth message is not limited in name, which can be, for example, a model performance monitoring response message, a model performance monitoring notification message, etc.

[0247] In some embodiments, the fourth message can be used for at least one of the following: indicating the model performance of the first model, triggering a change of positioning method, triggering training of the first model.

[0248] In some embodiments, the fourth message can include performance information. The performance information can be used to indicate the model performance of the first model. In an example, the performance information can indicate a model performance value determined based on the model performance monitoring of the first model. In an example, the performance information can indicate whether the model performance of the first model meets a threshold. For example, the performance information can indicate that the model performance of the first model is higher than the threshold. For example, the performance information can indicate that the model performance of the first model is lower than the threshold. In an example, the performance information can indicate that the model performance of the first model is good or bad. For example, the performance information can indicate that the model performance of the first model is good. For example, the performance information can indicate that the model performance of the first model is bad.

[0249] In some embodiments, the fourth message can include first trigger information. The first trigger information can be used to trigger a change of positioning method.

[0250] In some embodiments, the positioning method can include at least one of the following: AI / ML based positioning, legacy positioning.

[0251] In some embodiments, the first trigger information can indicate a change from AI / ML based positioning to legacy positioning. In an example, in a case where the model performance of the first model is lower than a threshold, and / or in a case where the model performance of the first model is bad, the fourth message can include the first trigger information to indicate a change from AI / ML based positioning to legacy positioning. At this time, the first trigger information can indicate a change from positioning based on the first model to legacy positioning.

[0252] In some embodiments, the first trigger information can indicate a change from legacy positioning to AI / ML based positioning. In an example, in a case where the model performance of the first model is higher than a threshold, and / or in a case where the model performance of the first model is good, the fourth message can include the first trigger information to indicate a change from legacy positioning to AI / ML based positioning. At this time, the first trigger information can indicate a change from legacy positioning to positioning based on the first model.

[0253] In some embodiments, the fourth message can include second trigger information. The second trigger information can be used to trigger training of the first model.

[0254] In some embodiments, the second trigger information can indicate to train the first model. In an example, the fourth message can comprise the second trigger information to indicate to train the first model in a case that the model performance of the first model is below a threshold, and / or in a case that the model performance of the first model is poor.

[0255] In some embodiments, the fourth message can further comprise the trained first model. In an example, the fourth message can further comprise the model parameters of the trained first model in step S3105.

[0256] It is to be noted that the training of the first model can be implemented in the first network element 101 and / or the third network element 103. In an example, the training of the first model can be implemented in the first network element 101 or the third network element 103. For example, the training of the first model can be implemented in the first network element 101. At this time, step S3105 can be performed, and the fourth message can not be used to trigger the third network element 103 to train the first model. For example, the training of the first model can be implemented in the third network element 103. At this time, step S3105 can not be performed, and the fourth message can be used to trigger the third network element 103 to train the first model.

[0257] In some embodiments, the third network element 103 can obtain the model performance of the first model according to the fourth message. In an example, the third network element 103 can obtain the model performance of the first model according to the performance information in the fourth message. In an example, the third network element 103 can trigger to change the positioning method according to the first trigger information in the fourth message. In an example, the third network element 103 can trigger to train the first model according to the second trigger information in the fourth message.

[0258] In some embodiments, the third network element 103 can determine to trigger to change the positioning method according to the performance information carried in the fourth message. In some embodiments, the third network element 103 can determine to trigger to train the first model according to the performance information carried in the fourth message.

[0259] In some embodiments, the fourth message can be periodically sent or non-periodically sent. In an example, the first network element 101 can send the fourth message once according to the received third message. For example, the third message can be a request message, and the fourth message can be a response message for the request message. For example, the third message can be a query message, and the fourth message can be a feedback message for the query message. In an example, the first network element 101 can send the fourth message one or more times according to the received third message. For example, the third message can be a subscription message, and the fourth message can be a notification message for the subscription message. For example, the fourth message can be periodically sent. For example, the fourth message can be sent when a preset condition (e.g., a threshold) is met.

[0260] In some embodiments, the training of the first model can comprise retraining or retraining of the first model.

[0261] The model performance monitoring method according to the embodiments of the present disclosure can comprise at least one of steps S3101-S3106. For example, step S3102 can be implemented as an independent embodiment. For example, the combination of steps S3103 can be implemented as an independent embodiment. For example, the combination of steps S3102 and S3103 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S3101-S3106 are not limited thereto.

[0262] The model performance monitoring method according to the embodiments of the present disclosure can comprise at least one of steps S3101-S3106. For example, step S3102 can be implemented as an independent embodiment. For example, the combination of steps S3103 can be implemented as an independent embodiment. For example, the combination of steps S3102 and S3103 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S3101-S3106 are not limited thereto.

[0263] In some embodiments, at least two of steps S3101-S3106 can be exchanged in order or synchronously executed. For example, steps S3105 and S3106 can be exchanged in order or executed simultaneously.

[0264] In some embodiments, steps S3102, S3103, S3104, S3105, S3106 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0265] In some embodiments, steps S3101, S3102, S3104, S3105, S3106 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0266] FIG. 3B is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. The model performance monitoring method according to an embodiment of the present disclosure can be applied to the communication system 100. As shown in FIG. 3B, the model performance monitoring method according to an embodiment of the present disclosure includes steps S3201 to S3206.

[0267] In step S3201, the fourth network element 104 sends a third message to the first network element 101.

[0268] In some embodiments, the first network element 101 can receive the third message.

[0269] In some embodiments, the first network element 101 can be a NWDAF. In some embodiments, the first network element 101 can be a MTLF. In an example, the first network element 101 can be a MTLF located in a NWDAF. It can be understood that the first network element 101 can also be other network functions, which are not specifically limited in the present disclosure.

[0270] In some embodiments, the fourth network element 104 can be a NWDAF. In some embodiments, the fourth network element 104 can be a MTLF. In an example, the fourth network element 104 can be a MTLF located in a NWDAF. It can be understood that the fourth network element 104 can also be other network functions, which are not specifically limited in the present disclosure.

[0271] In some embodiments, the third message can be used to trigger the first network element 101 to perform model performance monitoring.

[0272] In some embodiments, the model performance monitoring can be determining the model performance of the first model. In some embodiments, the third message can be used to request to determine the model performance of the first model.

[0273] In some embodiments, the first model can be used to implement a location service. In an example, the first model can be used to implement positioning for one or more terminals. In some embodiments, the location service implemented based on the first model can include at least one of the following: determining a current location of a terminal, predicting a location of a terminal.

[0274] In some embodiments, the first model can include at least one of the following: an AI model, a ML model. In some embodiments, the first model can be used to implement AI / ML based positioning. For example, the AI model can be used to implement AI model based positioning. For example, the ML model can be used to implement ML based positioning.

[0275] In some embodiments, the name of the third message is not limited, which can be, for example, a model performance monitoring request message, a model performance monitoring indication message, etc.

[0276] In some embodiments, the third message can comprise at least one of the following: identification information of the first model, area information.

[0277] In some embodiments, the identification information of the first model can be used to identify the first model. Through the identification information, the fourth network element 104 can instruct to perform model performance monitoring on the first model associated with the identification information.

[0278] In some embodiments, the identification information of the first model can comprise an identifier.

[0279] In some embodiments, the identifier can be used to identify each first model. In an example, one first model can be uniquely identified by an identifier. Different first models can be identified by different identifiers.

[0280] In some embodiments, the identifier can be used to identify a model group composed of one or more first models. In an example, one model group can be uniquely identified by an identifier. Different model groups can be identified by different identifiers.

[0281] In some embodiments, the area information can be used to indicate a first area in which the first model implements positioning.

[0282] In some embodiments, the first area can be an area to which the positioning based on the first model is directed. In some embodiments, the positioning based on the first model can be based on measurements within the first area. For example, the object to be positioned can be located within the first area. For example, the terminal and / or network device used to perform positioning can be located within the first area.

[0283] In some embodiments, the object to be positioned can be a UE.

[0284] In some embodiments, the terminal performing positioning can be a UE. In some embodiments, the terminal performing positioning can be a PRU.

[0285] In some embodiments, the network device performing positioning can be an access network device.

[0286] In some embodiments, after receiving the third message, the first network element 101 can determine to perform model performance monitoring on the first model according to the third message. In an example, the first network element 101 can determine the model performance monitoring to be on the first model according to the identification information and / or the area information in the third message.

[0287] In some embodiments, step S3201 can not be performed. In some embodiments, the first network element 101 can not receive the third message from the fourth network element 104.

[0288] In some embodiments, the first network element 101 can determine to perform the model performance monitoring for the first model by itself. In an example, the first network element 101 can determine to perform the model performance monitoring for all models on the first network element 101 by itself, and the first model is included in the models. In an example, the first network element 101 can determine to perform the model performance monitoring for the first model by itself.

[0289] In step S3202, the first network element 101 sends a first message to the second network element 102.

[0290] Optional implementation of step S3202 can be referred to optional implementation of step S3102 of FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0291] In step S3203, the second network element 102 sends a second message to the first network element 101.

[0292] Optional implementation of step S3203 can be referred to optional implementation of step S3103 of FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0293] In step S3204, the first network element 101 performs the model performance monitoring.

[0294] Optional implementation of step S3204 can be referred to optional implementation of step S3104 of FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0295] In step S3205, the first network element 101 performs the model training.

[0296] In some embodiments, the first network element 101 can trigger the training of the first model according to the model performance.

[0297] In some embodiments, the first network element 101 can trigger the training of the first model in a case where the model performance of the first model is determined to be lower than a threshold. In some embodiments, the first network element 101 can perform the training of the first model in a case where the model performance of the first model is determined to be lower than a threshold.

[0298] In some embodiments, the first network element 101 can trigger the training of the first model in a case where the model performance of the first model is determined to be poor. In some embodiments, the first network element 101 can perform the training of the first model in a case where the model performance of the first model is determined to be poor.

[0299] The training of the first model can be implemented by the first network element 101 through step S3205. In an example, step S3205 can be performed in the case that the first network element 101 determines to perform the model performance monitoring by itself.

[0300] In step S3206, the first network element 101 sends a fourth message to the fourth network element 104.

[0301] In some embodiments, the fourth network element 104 can receive the fourth message.

[0302] In some embodiments, the fourth message can be sent according to the model performance.

[0303] In some embodiments, the fourth message can be sent in response to the third message.

[0304] In some embodiments, the fourth message can be used to trigger the fourth network element 104 to train the first model.

[0305] In some embodiments, the name of the fourth message is not limited, which can be, for example, a model performance monitoring response message, a model performance monitoring notification message, a training trigger message, etc.

[0306] In some embodiments, the fourth message can include second trigger information. The second trigger information can be used to trigger the training of the first model.

[0307] In some embodiments, the second trigger information can indicate the training of the first model. In an example, in the case that the model performance of the first model is lower than a threshold value, and / or in the case that the model performance of the first model is poor, the fourth message can include the second trigger information to indicate the training of the first model.

[0308] In some embodiments, step S3206 can be performed in the case that the first network element 101 receives the third message from the fourth network element 104.

[0309] In some embodiments, both step S3205 and step S3206 can be performed in the case that the first network element 101 receives the third message from the fourth network element 104.

[0310] In some embodiments, the fourth message can be periodically sent or non-periodically sent. In an example, the first network element 101 can send the fourth message once according to the received third message. For example, the third message can be a request message, and the fourth message can be a response message for the request message. For example, the third message can be a query message, and the fourth message can be a feedback message for the query message. In an example, the first network element 101 can send the fourth message one or more times according to the received third message. For example, the third message can be a subscription message, and the fourth message can be a notification message for the subscription message. For example, the fourth message can be periodically sent. For example, the fourth message can be sent when a preset condition (e.g., a threshold) is met.

[0311] In some embodiments, the training of the first model can comprise retraining or retraining of the first model.

[0312] The model performance monitoring method according to the embodiments of the present disclosure can comprise at least one of steps S3201 to S3206. For example, step S3202 can be implemented as an independent embodiment. For example, the combination of steps S3203 can be implemented as an independent embodiment. For example, the combination of steps S3202 and S3203 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S3201 to S3206 are not limited thereto.

[0313] The model performance monitoring method according to the embodiments of the present disclosure can comprise at least one of steps S3201 to S3206. For example, step S3202 can be implemented as an independent embodiment. For example, the combination of steps S3203 can be implemented as an independent embodiment. For example, the combination of steps S3202 and S3203 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S3201 to S3206 are not limited thereto.

[0314] In some embodiments, at least two of steps S3201 to S3206 can be exchanged in order or synchronously executed. For example, steps S3205 and S3206 can be exchanged in order or executed simultaneously.

[0315] In some embodiments, steps S3202, S3203, S3204, S3205, S3206 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0316] In some embodiments, steps S3201, S3202, S3204, S3205, S3206 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0317] In some embodiments, the names of information and the like are not limited to the names described in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "field", "symbol", "codebook", "codeword", "codepoint", "bit", "data", "program", "chip", and the like can be replaced with each other.

[0318] In some embodiments, terms such as "radio", "wireless", "radio access network (RAN)", "access network (AN)", "RAN-based", and the like can be replaced with each other.

[0319] In some embodiments, terms such as "acquire", "obtain", "get", "receive", "transmit", "bidirectional transmission", "transmit and / or receive" can be replaced with each other, and can be interpreted as receiving from another subject, acquiring from a protocol, obtaining from a higher layer, obtaining by processing oneself, autonomously implementing, and the like.

[0320] In some embodiments, terms such as "transmit", "emit", "report", "issue", "transmit", "bidirectional transmission", "transmit and / or receive" can be replaced with each other.

[0321] In some embodiments, terms such as "certain", "preset", "pre-set", "set", "indicated", "a certain", "arbitrary", "first", and the like can be replaced with each other, and "certain A", "preset A", "pre-set A", "set A", "indicated A", "a certain A", "arbitrary A", "first A" can be interpreted as A specified in advance in a protocol and the like, can be interpreted as A obtained by setting, configuring, or indicating, and the like, and can be interpreted as certain A, a certain A, arbitrary A, or first A, but are not limited thereto.

[0322] In some embodiments, the determining or judging can be performed by a value represented by 1 bit (0 or 1), a true or false value (Boolean value) represented by true or false, or a comparison of numerical values (for example, a comparison with a predetermined value), but is not limited thereto.

[0323] FIG. 4 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. The present embodiment relates to a model performance monitoring method. The model performance monitoring method is performed by the first network element 101. As shown in FIG. 4, the above method includes steps S401 to S406.

[0324] In step S401, a third message is acquired.

[0325] The optional implementation of step S401 can refer to the optional implementation of step S3101 in FIG. 3A, step S3201 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0326] In some embodiments, the first network element 101 can receive the third message sent by the third network element 103 or the fourth network element 104, but is not limited thereto, and can also receive the third message sent by other subjects.

[0327] In some embodiments, the first network element 101 can acquire the third message specified by a protocol.

[0328] In some embodiments, the first network element 101 can acquire the third message from an upper layer.

[0329] In some embodiments, the first network element 101 can process to obtain the third message.

[0330] In some embodiments, step S401 can be omitted, and the first network element 101 autonomously implements the function indicated by the third message, or the above function is default or default.

[0331] In step S402, a first message is sent.

[0332] The optional implementation of step S402 can refer to the optional implementation of step S3102 in FIG. 3A, step S3202 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0333] In some embodiments, the first network element 101 can send the first message to the second network element 102, but is not limited thereto, and can also send the first message to other subjects.

[0334] In step S403, a second message is acquired.

[0335] The optional implementation of step S403 can refer to the optional implementation of step S3103 in FIG. 3A, step S3203 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which are not described here again.

[0336] In some embodiments, the first network element 101 can receive the second message sent by the second network element 102, but is not limited thereto, and can also receive the second message sent by other subjects.

[0337] In step S404, model performance monitoring is performed.

[0338] The optional implementation of step S404 can refer to the optional implementation of step S3104 in FIG. 3A, step S3204 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A, which are not described here again.

[0339] In some embodiments, the model performance monitoring can be performed based on the first data in the second message.

[0340] In step S405, model training is performed.

[0341] The optional implementation of step S405 can refer to the optional implementation of step S3105 in FIG. 3A, step S3205 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which are not described here again.

[0342] In step S406, a fourth message is sent.

[0343] The optional implementation of step S406 can refer to the optional implementation of step S3106 in FIG. 3A, step S3206 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A, which are not described here again.

[0344] In some embodiments, the first network element 101 can send the fourth message to the third network element 103 or the fourth network element 104, but is not limited thereto, and can also send the fourth message to other subjects.

[0345] The model performance monitoring method related to the embodiments of the present disclosure can include at least one of steps S401 to S406. For example, step S401 can be implemented as an independent embodiment. For example, step S402 can be implemented as an independent embodiment. For example, step S403 can be implemented as an independent embodiment. For example, a combination of steps S401 and S402 can be implemented as an independent embodiment. For example, a combination of steps S402 and S403 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S401 to S406 are not limited to this.

[0346] In some embodiments, at least two of steps S401 to S406 can be exchanged in order or synchronously executed. For example, steps S405 and S406 can be exchanged in order or executed at the same time.

[0347] In some embodiments, steps S402, S403, S404, S405, S406 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0348] In some embodiments, steps S401, S402, S404, S405, S406 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0349] FIG. 5 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiments of the present disclosure relate to a model performance monitoring method. The model performance monitoring method is performed by the second network element 102. As shown in FIG. 5, the above method includes steps S501 to S502.

[0350] In step S501, a first message is obtained.

[0351] The optional implementation of step S501 can refer to the optional implementation of step S3102 in FIG. 3A, step S3202 in FIG. 3B, and other associated parts in the embodiments related to FIG. 3A and FIG. 3B, which will not be repeated here.

[0352] In some embodiments, the second network element 102 can receive the first message sent by the first network element 101, but is not limited thereto, and can also receive the first message sent by other subjects.

[0353] In step S502, a second message is sent.

[0354] The optional implementation of step S502 can refer to the optional implementation of step S3103 in FIG. 3A, step S3203 in FIG. 3B, and other associated parts in the embodiments related to FIG. 3A and FIG. 3B, which will not be repeated here.

[0355] In some embodiments, the second network element 102 can send the second message to the first network element 101, but is not limited thereto, and can send the second message to other subjects.

[0356] The model performance monitoring method related to the embodiments of the present disclosure can include at least one of steps S501 to S502. For example, step S502 can be implemented as an independent embodiment. For example, the combination of steps S501 and S502 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S501 to S502 are not limited thereto.

[0357] In some embodiments, step S501 is optional, and can be omitted or replaced in different embodiments.

[0358] FIG. 6 is a flowchart of a model performance monitoring method according to an embodiment of the present disclosure. The embodiments of the present disclosure relate to a model performance monitoring method. The model performance monitoring method is performed by a third network element 103. As shown in FIG. 6, the above method includes steps S601 to S602.

[0359] In step S601, a third message is sent.

[0360] The optional implementation of step S601 can refer to the optional implementation of step S3101 in FIG. 3A and other associated parts in the embodiments related to FIG. 3A, which will not be repeated here.

[0361] In some embodiments, the third network element 103 can send the third message to the first network element 101, but is not limited thereto, and can send the third message to other subjects.

[0362] In step S602, a fourth message is obtained.

[0363] The optional implementation of step S602 can refer to the optional implementation of step S3106 in FIG. 3A and other associated parts in the embodiments related to FIG. 3A, which will not be repeated here.

[0364] In some embodiments, the third network element 103 can receive the fourth message sent by the first network element 101, but is not limited thereto, and can receive the fourth message sent by other subjects.

[0365] The model performance monitoring method related to the embodiments of the present disclosure can include at least one of steps S601 to S602. For example, step S601 can be implemented as an independent embodiment. For example, step S602 can be implemented as an independent embodiment. For example, the combination of steps S601 and S602 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S601 to S602 are not limited to this.

[0366] In some embodiments, step S601 is optional, which can be omitted or replaced in different embodiments.

[0367] In some embodiments, step S602 is optional, which can be omitted or replaced in different embodiments.

[0368] FIG. 7 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiments of the present disclosure relate to a model performance monitoring method. The model performance monitoring method is performed by the fourth network element 104. As shown in FIG. 7, the above method includes steps S701 to S702.

[0369] In step S701, a third message is sent.

[0370] The optional implementation of step S701 can refer to the optional implementation of step S3201 in FIG. 3B and other associated parts in the embodiments related to FIG. 3B, which will not be repeated here.

[0371] In some embodiments, the fourth network element 104 can send the third message to the first network element 101, but is not limited to this, and can also send the third message to other subjects.

[0372] In step S702, a fourth message is obtained.

[0373] The optional implementation of step S702 can refer to the optional implementation of step S3206 in FIG. 3B and other associated parts in the embodiments related to FIG. 3B, which will not be repeated here.

[0374] In some embodiments, the fourth network element 104 can receive the fourth message sent by the first network element 101, but is not limited to this, and can also receive the fourth message sent by other subjects.

[0375] The model performance monitoring method according to the embodiments of the present disclosure can include at least one of steps S701 to S702. For example, step S701 can be implemented as an independent embodiment. For example, step S702 can be implemented as an independent embodiment. For example, a combination of steps S701 and S702 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S701 to S702 are not limited to this.

[0376] In some embodiments, step S701 is optional, which can be omitted or replaced in different embodiments.

[0377] In some embodiments, step S702 is optional, which can be omitted or replaced in different embodiments.

[0378] FIG. 8A is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiments of the present disclosure relate to a model performance monitoring method. As shown in FIG. 8A, the above method includes steps S8101 to S8102.

[0379] In step S8101, the first network element 101 sends a first message to the second network element 102.

[0380] The optional implementation of step S8101 can refer to the optional implementation of step S3102 in FIG. 3A, step S3202 in FIG. 3B, and other associated parts in the embodiments related to FIG. 3A and FIG. 3B, which will not be repeated here.

[0381] In step S8102, the second network element 102 sends a second message to the first network element 101.

[0382] The optional implementation of step S8102 can refer to the optional implementation of step S3103 in FIG. 3A, step S3203 in FIG. 3B, and other associated parts in the embodiments related to FIG. 3A and FIG. 3B, which will not be repeated here.

[0383] FIG. 8B is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiments of the present disclosure relate to a model performance monitoring method. As shown in FIG. 8B, the above method includes step S8201.

[0384] In step S8201, the third network element 103 sends a third message to the first network element 101.

[0385] The optional implementation of step S8201 can refer to the optional implementation of step S3101 in FIG. 3A, and other associated parts in the embodiments related to FIG. 3A, which will not be repeated here.

[0386] Hereinafter, the embodiments of the present disclosure are exemplarily described through specific implementations.

[0387] In some embodiments, the embodiments of the present disclosure propose an independent procedure for NWDAF (i.e., a first network element)-based model performance monitoring. Here, it is assumed that data (i.e., first data) for performing model performance monitoring is pre-stored in ADRF (i.e., a second network element). In an example, a request for model performance monitoring in a specific area (i.e., a first area) can be received from LMF (i.e., a third network element). In an example, the NWDAF requests the ADRF to collect pre-stored data for model performance monitoring. In an example, based on the collected data from the ADRF, the NWDAF starts to evaluate model performance and obtains a result. In an example, the result can be sent to the LMF to trigger model retraining in the NWDAF.

[0388] FIG. 9 is an interaction schematic diagram of an exemplary implementation of a model performance monitoring method according to the embodiments of the present disclosure. As shown in FIG. 9, the model performance monitoring method can include steps S901 to S905.

[0389] In some embodiments, it is assumed that pre-stored data for model performance monitoring can be stored in ADRF or 5G / 6G CN NF. The data at least includes measurement data from UE / gNB, or a calculated position based on measurement data from UE / gNB, and a corresponding known position (e.g., a position obtained through GPS).

[0390] In some embodiments, the pre-stored data for model performance monitoring can be stored for each area.

[0391] In some embodiments, it can be assumed that the pre-stored data for model performance monitoring is authorized.

[0392] In step S901, step S901a or step S901b can be performed.

[0393] In step S901a, the NWDAF receives a model performance monitoring request (i.e., a third message) from the LMF to trigger the NWDAF to evaluate model performance of a model adopted by AI-based positioning at the LMF. In an example, the model performance monitoring request can include a model ID and / or area information.

[0394] In step S901b, the NWDAF receives a trigger for model performance monitoring from an internal NF or NWDAF (i.e., a fourth network element).

[0395] In step S902, the NWDAF sends a data collection request (i.e., a first message) for model performance monitoring. In an example, the data collection request can include a model ID and / or area information.

[0396] In step S903, the ADRF sends a data collection response (i.e., a second message) to the NWDAF. In an example, the data collection response can include a data set for model performance monitoring.

[0397] In step S904, the NWDAF evaluates the model performance by using the data from the ADRF and obtains a result (i.e., an evaluation result).

[0398] In some embodiments, the result can indicate whether the model performance is below a threshold. In some embodiments, the result can indicate that the model performance is good or bad.

[0399] In step S905, step S905a or S905b can be performed.

[0400] In step S905a, the NWDAF sends the result to the LMF. The result can trigger a change of the positioning method in the LMF, or trigger the LMF to retrain the model.

[0401] In step S905b, the NWDAF can trigger to retrain the model.

[0402] In the embodiments of the present disclosure, part or all of the steps, and optional implementation manners thereof, can be combined with part or all of the steps in other embodiments, or can be combined with optional implementation manners of other embodiments.

[0403] The embodiments of the present disclosure also provide a communication apparatus for implementing any of the above methods. For example, the embodiments of the present disclosure provide a communication apparatus comprising units or modules for implementing the steps performed by the first network element in any of the above methods. For example, the embodiments of the present disclosure provide a communication apparatus comprising units or modules for implementing the steps performed by the second network element in any of the above methods. For example, the embodiments of the present disclosure provide a communication apparatus comprising units or modules for implementing the steps performed by the third network element in any of the above methods. For example, the embodiments of the present disclosure provide a communication apparatus comprising units or modules for implementing the steps performed by the fourth network element in any of the above methods.

[0404] It should be understood that the division of each unit or module in the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated in actual implementation. In addition, the units or modules in the apparatus can be implemented in the form of processor calling software: for example, the apparatus includes a processor, the processor is connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the units or modules of the above apparatus, wherein the processor is a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the apparatus or a memory outside the apparatus. Alternatively, the units or modules in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units or modules can be realized by the design of hardware circuit. The above hardware circuit can be understood as one or more processors; for example, in one implementation, the above hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the units or modules are realized by the design of the logical relationship of elements in the circuit; for another example, in another implementation, the above hardware circuit is a programmable logic device (PLD), and a field programmable gate array (FPGA) is taken as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the units or modules. All units or modules of the above apparatus can be implemented in the form of processor calling software, or all units or modules can be implemented in the form of hardware circuit, or part of the units or modules are implemented in the form of processor calling software, and the remaining part is implemented in the form of hardware circuit.

[0405] In the embodiments of the present disclosure, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit, a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), and the like. In another implementation, the processor can implement certain functions through a logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable. For example, the processor is a hardware circuit implemented by a special-purpose integrated circuit or a programmable logic device, such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the configuration of the hardware circuit. It can be understood that the processor loads an instruction to implement the functions of the above part or all units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), and the like.

[0406] FIG. 10 is a structural schematic diagram of a model performance monitoring apparatus provided by the embodiments of the present disclosure. As shown in FIG. 10, the model performance monitoring apparatus 1000 can include at least one of the following: a transceiver module 1001, a processing module 1002.

[0407] In some embodiments, the model performance monitoring apparatus 1000 can be the first network element 101. In some embodiments, the transceiver module 1001 can be configured to send a first message to the second network element, where the first message is used to request the first data from the second network element; receive a second message sent by the second network element, where the second message includes the first data; and the first data is used to implement the model performance monitoring on the first model. Optionally, the transceiver module 1001 can be configured to perform at least one of the communication steps (for example, steps S3101, S3102, S3103, S3106, S3201, S3202, S3203, S3206, but not limited to) of sending and / or receiving and the like performed by the first network element 101 in any of the above methods, and details are not described herein again. Optionally, the processing module 1002 can be configured to perform at least one of the other steps (for example, steps S3104, S3105, S3204, S3205, but not limited to) in addition to the communication steps of sending and / or receiving and the like performed by the first network element 101 in any of the above methods, and details are not described herein again.

[0408] In some embodiments, the model performance monitoring apparatus 1000 can be the second network element 102. In some embodiments, the transceiver module 1001 can be configured to receive a first message sent by the first network element, where the first message is used to request the first data from the second network element; and send a second message to the first network element, where the second message comprises the first data; and where the first data is used to implement the model performance monitoring on the first model. Optionally, the transceiver module 1001 can be configured to perform at least one of the communication steps (e.g., steps S3102, S3103, S3202, S3203, but not limited to) of sending and / or receiving performed by the second network element 102 in any of the above methods, which will not be described herein again.

[0409] In some embodiments, the model performance monitoring apparatus 1000 can be the third network element 103. In some embodiments, the transceiver module 1001 can be configured to send a third message to the first network element, where the third message is used to trigger the first network element to perform the model performance monitoring on the first model. Optionally, the transceiver module 1001 can be configured to perform at least one of the communication steps (e.g., steps S3101, S3106, but not limited to) of sending and / or receiving performed by the third network element 103 in any of the above methods, which will not be described herein again.

[0410] In some embodiments, the model performance monitoring apparatus 1000 can be the fourth network element 104. In some embodiments, the transceiver module 1001 can be configured to send a third message to the first network element, where the third message is used to trigger the first network element to perform the model performance monitoring on the first model. Optionally, the transceiver module 1001 can be configured to perform at least one of the communication steps (e.g., steps S3101, S3106, but not limited to) of sending and / or receiving performed by the fourth network element 104 in any of the above methods, which will not be described herein again.

[0411] In some embodiments, the transceiver module can comprise a sending module and / or a receiving module. The sending module and the receiving module can be separate or integrated together. Optionally, the transceiver module can be mutually replaced with a transceiver.

[0412] In some embodiments, the processing module can be one module or can comprise a plurality of sub-modules. Optionally, the plurality of sub-modules perform all or part of the steps required to be performed by the processing module respectively. Optionally, the processing module can be mutually replaced with a processor.

[0413] FIG. 11A is a structural schematic diagram of a communication device provided by an embodiment of the present disclosure. The communication device 11100 can be a network device (for example, an access network device, a core network device, etc.), a terminal (for example, a user equipment, etc.), a chip, a chip system, or a processor supporting the network device to implement any of the above methods, or a chip, a chip system, or a processor supporting the terminal to implement any of the above methods. The communication device 11100 can be used to implement the methods described in the above method embodiments, and details can be referred to the descriptions in the above method embodiments.

[0414] As shown in FIG. 11A, the communication device 11100 includes one or more processors 11101. The processor 11101 can be a general purpose processor or a special purpose processor, for example, a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, and the central processing unit can be used to control the communication device (for example, a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute programs, and process data of the programs. Optionally, the communication device 11100 is configured to perform any of the above methods. Optionally, the one or more processors 11101 are configured to invoke instructions to cause the communication device 11100 to perform any of the above methods.

[0415] In some embodiments, the communication device 11100 further includes one or more transceivers 11102. When the communication device 11100 includes the one or more transceivers 11102, the transceiver 11102 performs at least one of the communication steps (for example, steps S3101, S3102, S3103, S3106, S3201, S3202, S3203, S3206, but not limited to) in the above methods, and the processor 11101 performs at least one of the other steps (for example, steps S3104, S3105, S3204, S3205, but not limited to). In an optional embodiment, the transceiver can include a receiver and / or a transmitter, which can be separate or integrated together. Optionally, the terms of transceiver, transceiving unit, transceiver, transceiving circuit, interface circuit, interface, etc. can be replaced with each other, and the terms of transmitter, transmitting unit, transmitter, transmitting circuit, etc. can be replaced with each other, and the terms of receiver, receiving unit, receiver, receiving circuit, etc. can be replaced with each other.

[0416] In some embodiments, the communication device 11100 also includes one or more memories 11103 for storing data. Optionally, all or a portion of the memory 11103 can also reside in the communication device 11100. In some embodiments, the communication device 11100 can include one or more interface circuits 11104. Optionally, the interface circuit 11104 can be used to receive data from the memory 11103 or from another device or system, or to send data to the memory 11103 or to another device or system. For example, the interface circuit 11104 can receive data in packets, each packet having a header and a payload.

[0417] The communication device 11100 described in the above embodiments can be a network device or a terminal, but the scope of the communication device 11100 described in the present disclosure is not limited thereto, and the structure of the communication device 11100 can not be limited by FIG. 11A. The communication device can be a standalone device or can be part of a larger device. For example, the communication device can be: 1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally also include a storage component for storing data, programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, a smart terminal device, a cellular phone, a wireless device, a handset, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) other, etc.

[0418] FIG. 11B is a structural schematic diagram of a chip according to an embodiment of the present disclosure. For the case where the communication device 11100 can be a chip or a chip system, the structural schematic diagram of the chip 11200 shown in FIG. 11B can be referred to, but is not limited thereto.

[0419] The chip 11200 includes one or more processors 11201. The chip 11200 is configured to perform any of the above methods.

[0420] In some embodiments, the chip 11200 further includes one or more interface circuits 11202. Optionally, the terms interface circuit, interface, transceiver pin, etc. can be replaced by each other. In some embodiments, the chip 11200 further includes one or more memories 11203 for storing data. Optionally, all or part of the memory 11203 can be outside the chip 11200. Optionally, the interface circuit 11202 is connected with the memory 11203, the interface circuit 11202 can be used to receive data from the memory 11203 or other devices, the interface circuit 11202 can be used to send data to the memory 11203 or other devices. For example, the interface circuit 11202 can read the data stored in the memory 11203 and send the data to the processor 11201.

[0421] In some embodiments, the interface circuit 11202 performs at least one of the communication steps (for example, steps S3101, S3102, S3103, S3106, S3201, S3202, S3203, S3206, but not limited to) of transmitting and / or receiving in the above method. The interface circuit 11202 performing the communication steps such as transmitting and / or receiving in the above method means that the interface circuit 11202 performs data interaction between the processor 11201, the chip 11200, the memory 11203 or the transceiver device. In some embodiments, the processor 11201 performs at least one of the other steps (for example, steps S3104, S3105, S3204, S3205, but not limited to).

[0422] The modules and / or devices described in each of the embodiments of the virtual device, the physical device, the chip, etc. can be combined or separated as appropriate. Optionally, part or all of the steps can also be performed by multiple modules and / or devices, which are not limited here.

[0423] The embodiments of the present disclosure also propose a storage medium, and the storage medium stores instructions. When the instructions run on the communication device 11100, the communication device 11100 performs any one of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer readable storage medium, but is not limited to this, and it can also be a storage medium readable by other devices. Optionally, the storage medium can be a non-transitory storage medium, but is not limited to this, and it can also be a transitory storage medium.

[0424] The embodiments of the present disclosure also propose a program product, and the program product is executed by the communication device 11100, so that the communication device 11100 performs any one of the above methods. Optionally, the program product is a computer program product.

[0425] The embodiments of the present disclosure further provide a computer program, which, when running on a computer, enables the computer to perform any of the above methods.

[0426] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the following claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0427] It is to be understood that the application is not limited to particular details described herein and as illustrated in the figures and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.

Claims

A method of model performance monitoring, performed by a first network element, wherein, The method comprises: sending a first message to a second network element, wherein the first message is used to request first data from the second network element; receiving a second message sent by the second network element, wherein the second message comprises the first data; wherein the first data is used to implement model performance monitoring of a first model. The method of claim 1, wherein, The first model comprises at least one of: an artificial intelligence (AI) model; a machine learning (ML) model. The method according to claim 1 or 2, wherein The first model is used to implement AI / ML-based positioning. The method of any one of claims 1 to 3, wherein, The first message comprises at least one of: indication information used to indicate model performance monitoring of the first model; identification information of the first model; region information used to indicate a first region in which the first model implements positioning. The method of any one of claims 1 to 4, wherein, The first data comprises at least one of: measurement data; a calculated position obtained based on the measurement data; an actual position. The method of any one of claims 1 to 5, wherein, The first data is associated with a first region in which the first model implements positioning. The method of any one of claims 1 to 6, wherein, The first data is authorized data. The method of any one of claims 1 to 7, wherein, The first data is data pre-stored in the second network element. The method of any one of claims 1 to 8, wherein, The method further comprises: receiving a third message sent by a third network element, wherein the third message is used to trigger the first network element to perform the model performance monitoring. The method of claim 9, wherein, The third message comprises at least one of: identification information of the first model; region information used to indicate a first region in which the first model implements positioning. The method of any one of claims 1 to 10, wherein, The method further comprises: determining model performance of the first model based on the first data; performing a first operation related to the first model according to the model performance. The method of claim 11, wherein, The first operation comprises at least one of: sending a fourth message to a third network element; sending the fourth message to a fourth network element, wherein the fourth network element has the same function as the first network element; training the first model. The method of claim 12, wherein, The fourth message is used to at least one of: indicate model performance of the first model; trigger a change in a positioning method; trigger training of the first model. A model performance monitoring method, performed by a second network element, wherein, The method comprises: receiving a first message sent by a first network element, wherein the first message is used to request first data from the second network element; sending a second message to the first network element, wherein the second message comprises the first data; wherein the first data is used to implement model performance monitoring of a first model. The method of claim 14, wherein, The first model comprises at least one of: an artificial intelligence (AI) model; a machine learning (ML) model. The method according to claim 14 or 15, wherein The first model is used to implement AI / ML-based positioning. The method of any one of claims 14 to 16, wherein, The first message comprises at least one of: indication information used to indicate model performance monitoring of the first model; identification information of the first model; region information used to indicate a first region in which the first model implements positioning. The method of any one of claims 14 to 17, wherein, The first data comprises at least one of: measurement data; a calculated position obtained based on the measurement data; an actual position. The method of any one of claims 14 to 18, wherein, The first data is associated with a first region in which the first model implements positioning. The method of any one of claims 14 to 19, wherein, The first data is authorized data. The method of any one of claims 14 to 20, wherein, The first data is data pre-stored in the second network element. The method of any one of claims 14 to 21, wherein, The second network element comprises at least one of: an analytics data repository function (ADRF); a unified data management (UDM) function; a unified data repository (UDR) function. A method of model performance monitoring, performed by a third network element, wherein, The method comprises: sending a third message to a first network element, wherein the third message is used to trigger the first network element to perform model performance monitoring on a first model. The method of claim 23, wherein, The third message comprises at least one of: identification information of the first model; region information used to indicate a first region in which the first model implements positioning. The method according to claim 23 or 24, wherein The first model comprises at least one of: an artificial intelligence (AI) model; a machine learning (ML) model. The method of any one of claims 23 to 25, wherein, The first model is used to implement AI / ML-based positioning. The method of any one of claims 23 to 26, wherein, The model performance monitoring of the first model by the first network element is based on first data obtained from a second network element. The method of claim 27, wherein, The first data comprises at least one of: measurement data; a calculated position based on the measurement data; an actual position. The method of claim 27 or 28, wherein, The first data is associated with a first region in which the first model implements positioning. The method of any one of claims 27 to 29, wherein, The first data is authorized data. The method of any one of claims 27 to 30, wherein, The first data is data pre-stored in the second network element. The method of any one of claims 23-31, wherein The method further comprises: receiving a fourth message sent by the first network element according to the model performance of the first model, wherein the fourth message is used to at least one of: indicate the model performance of the first model; trigger the third network element to change a positioning method; trigger the third network element to train the first model. A model performance monitoring apparatus is arranged in a first network element, wherein, The apparatus comprises: a transceiver module configured to: send a first message to a second network element, wherein the first message is used to request first data from the second network element; receive a second message sent by the second network element, wherein the second message comprises the first data; wherein the first data is used to implement model performance monitoring on a first model. A model performance monitoring apparatus is arranged at a second network element, wherein The apparatus comprises: a transceiver module configured to: receive a first message sent by a first network element, wherein the first message is used to request first data from the second network element; send a second message to the first network element, wherein the second message comprises the first data; wherein the first data is used to implement model performance monitoring on a first model. A model performance monitoring apparatus is arranged at a third network element, wherein, The apparatus comprises: a transceiver module configured to send a third message to a first network element, wherein the third message is used to trigger the first network element to perform model performance monitoring on a first model. A communication device comprises: one or more processors; a memory storing instructions; wherein the instructions, when executed by the communication device, cause the communication device to implement at least one of: the model performance monitoring method according to any one of claims 1 to 13; the model performance monitoring method according to any one of claims 14 to 22; the model performance monitoring method according to any one of claims 23 to 32. A communication system comprises at least: a first network element configured to implement the model performance monitoring method according to any one of claims 1 to 13; a second network element configured to implement the model performance monitoring method according to any one of claims 14 to 22. A storage medium storing instructions, wherein, When the instructions run on the communication device, the communication device is caused to implement at least one of: the model performance monitoring method according to any one of claims 1 to 13; The model performance monitoring method of any one of claims 14 to 22; The model performance monitoring method of any one of claims 23 to 32. A computer program product comprising instructions which, when executed on a communications device, cause the communications device to at least one of: The model performance monitoring method of any one of claims 1 to 13; The model performance monitoring method of any one of claims 14 to 22; The model performance monitoring method of any one of claims 23 to 32.

Images