Model performance monitoring method and apparatus, and device, system and storage medium

Abstract

The present disclosure relates to a model performance monitoring method and apparatus, and a device, a system and a storage medium. The method comprises: sending a first message to a second network element, wherein the first message is configured to request the execution of data collection for at least one terminal device; and receiving a second message sent by the second network element, wherein the second message comprises first data, the first data is data collected by the second network element for the at least one terminal device, and the first data is configured to implement model performance monitoring of a first model. By means of the solution of the present disclosure, first data can be efficiently obtained, so as to implement model performance monitoring of a first model.

Description

Model performance monitoring method and apparatus, device, system, and storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and particularly relates to a model performance monitoring method and apparatus, device, system, and storage medium. BACKGROUND

[0002] With the development of communication technology, artificial intelligence technology can be combined with communication technology to improve the efficiency of communication networks and enhance the functions of communication networks.

[0003] SUMMARY

[0004] The present disclosure provides a model performance monitoring method and apparatus, a communication device, a communication system, a storage medium, and a computer program product.

[0005] According to a first aspect of the present disclosure, a model performance monitoring method is provided. The model performance monitoring method is performed by a first network element. The model performance monitoring method comprises: sending a first message to a second network element, wherein the first message is used to request data collection for at least one terminal device; and receiving a second message sent by the second network element, wherein the second message comprises first data; wherein the first data is data collected by the second network element for the at least one terminal device, and the first data is used to implement model performance monitoring of a first model.

[0006] According to a second aspect of the present disclosure, a model performance monitoring method is provided. The model performance monitoring method is performed by a second network element. The model performance monitoring method comprises: receiving a first message sent by a first network element, wherein the first message is used to request data collection for at least one terminal device; and sending a second message to the first network element, wherein the second message comprises first data; wherein the first data is data collected by the second network element for the at least one terminal device, and the first data is used to implement model performance monitoring of a first model.

[0007] According to a third aspect of the present disclosure, a model performance monitoring method is provided. The model performance monitoring method is performed by a third network element. The model performance monitoring method comprises: sending a third message to a first network element, wherein the third message is used to determine whether data collection for at least one terminal device is authorized; wherein the data collection for the at least one terminal device is used to obtain first data, and the first data is used to implement model performance monitoring of a first model.

[0008] According to a fourth aspect of embodiments of the present disclosure, a model performance monitoring apparatus is provided. The model performance monitoring apparatus is arranged at a first network element. The model performance monitoring apparatus comprises a transceiver. The transceiver is configured to: send, to a second network element, a first message, wherein the first message is used to request data collection for at least one terminal device; and receive a second message sent by the second network element, wherein the second message comprises first data, and wherein the first data is data collected by the second network element for the at least one terminal device, and the first data is used to implement model performance monitoring of a first model.

[0009] According to a fifth aspect of embodiments of the present disclosure, a model performance monitoring apparatus is provided. The model performance monitoring apparatus is arranged at a second network element. The model performance monitoring apparatus comprises a transceiver. The transceiver is configured to: receive a first message sent by a first network element, wherein the first message is used to request data collection for at least one terminal device; and send, to the first network element, a second message, wherein the second message comprises first data, and wherein the first data is data collected by the second network element for the at least one terminal device, and the first data is used to implement model performance monitoring of a first model.

[0010] According to a sixth aspect of embodiments of the present disclosure, a model performance monitoring apparatus is provided. The model performance monitoring apparatus is arranged at a third network element. The model performance monitoring apparatus comprises a transceiver. The transceiver is configured to: send, to a first network element, a third message, wherein the third message is used to determine whether data collection for at least one terminal device is authorized, and wherein the data collection for the at least one terminal device is used to obtain first data, and the first data is used to implement model performance monitoring of a first model.

[0011] According to a seventh aspect of embodiments of the present disclosure, a communication device is provided. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to the first aspect.

[0012] According to an eighth aspect of embodiments of the present disclosure, a communication device is provided. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to the second aspect.

[0013] According to a ninth aspect of embodiments of the present disclosure, a communication device is provided. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to the third aspect.

[0014] According to a tenth aspect of the embodiments of the present disclosure, a communication system is provided. The communication system comprises at least: a first network element configured to implement the model performance monitoring method according to the first aspect; a second network element configured to implement the model performance monitoring method according to the second aspect; and a third network element configured to implement the model performance monitoring method according to the third aspect.

[0015] According to an eleventh aspect of the embodiments of the present disclosure, a storage medium is provided. The storage medium stores instructions. The instructions, when executed on a communication device, cause the communication device to perform the model performance monitoring method according to the first aspect, the second aspect, or the third aspect.

[0016] According to a twelfth aspect of the embodiments of the present disclosure, a program product is provided. The program product, when executed by a communication device, causes the communication device to perform the model performance monitoring method according to the first aspect, the second aspect, or the third aspect.

[0017] According to a thirteenth aspect of the embodiments of the present disclosure, a computer program is provided. The computer program, when executed on a computer, causes the computer to perform the model performance monitoring method according to the first aspect, the second aspect, or the third aspect.

[0018] According to a fourteenth aspect of the embodiments of the present disclosure, a chip or chip system is provided. The chip or chip system comprises processing circuitry. The processing circuitry is configured to perform the model performance monitoring method according to the first aspect, the second aspect, or the third aspect.

[0019] According to the embodiments of the present disclosure, the first network element can obtain the relevant data for model performance monitoring of the first model from the second network element.

[0020] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not constitute a limitation on the embodiments of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following describes the drawings required for the embodiment description. The following drawings are only some embodiments of the present disclosure, and do not specifically limit the protection scope of the present disclosure.

[0022] FIG. 1 is an architecture schematic diagram of a communication system according to an embodiment of the present disclosure.

[0023] FIG. 2 is an interaction schematic diagram of model performance monitoring in the related art.

[0024] FIG. 3A is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0025] FIG. 3B is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0026] FIG. 4 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0027] FIG. 5 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0028] FIG. 6 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0029] FIG. 7 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0030] FIG. 8A is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0031] FIG. 8B is an interaction diagram of a model performance monitoring method according to an embodiment of the present disclosure.

[0032] FIG. 9 is an interaction diagram of an exemplary implementation of a model performance monitoring method according to an embodiment of the present disclosure.

[0033] FIG. 10 is a structural diagram of a model performance monitoring apparatus according to an embodiment of the present disclosure.

[0034] FIG. 11A is a structural diagram of a communication device according to an embodiment of the present disclosure.

[0035] FIG. 11B is a structural diagram of a chip according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure provide a model performance monitoring method and apparatus, a communication device, a communication system, a storage medium, and a computer program product.

[0037] In a first aspect, embodiments of the present disclosure provide a model performance monitoring method. The model performance monitoring method is performed by a first network element. The model performance monitoring method comprises: sending a first message to a second network element, wherein the first message is used to request to perform data collection on at least one terminal device; receiving a second message sent by the second network element, wherein the second message comprises first data; wherein the first data is data collected by the second network element on the at least one terminal device, and the first data is used to implement model performance monitoring on a first model.

[0038] Through the embodiment, the first network element can send the first message to the second network element to request the first data, and the second network element sends the second message containing the first data to the first network element. In this way, the first network element can obtain the first data of the at least one terminal device for model performance monitoring of the first model. In this way, the first network element can obtain the first data for model performance monitoring of the first model through interaction with the second network element, thereby effectively obtaining the first data for model performance monitoring of the first model for the at least one terminal device.

[0039] In combination with some embodiments of the first aspect, in some embodiments, the first message can include identification information of the at least one terminal device.

[0040] In combination with some embodiments of the first aspect, in some embodiments, the first model can include at least one of the following: an AI model; an ML model.

[0041] In combination with some embodiments of the first aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0042] Through the embodiment, the first data obtained by the first network element can be used to implement model performance monitoring of the first model, and the first model is used to implement AI / ML-based positioning. In this way, based on the first data, model performance monitoring of the first model for AI / ML-based positioning can be implemented.

[0043] In combination with some embodiments of the first aspect, in some embodiments, the first model can be applied to a first area.

[0044] In combination with some embodiments of the first aspect, in some embodiments, the first data can include at least one of the following: measurement data; a calculated position based on the measurement data; an actual position.

[0045] In combination with some embodiments of the first aspect, in some embodiments, the above method can further include determining the at least one terminal device, wherein the at least one terminal device supports data collection.

[0046] In combination with some embodiments of the first aspect, in some embodiments, the above method can further include receiving a third message sent by a third network element, wherein the third message is used to determine whether data collection for the at least one terminal device is authorized.

[0047] Through the embodiment, the first network element can receive the third message from the third network element. Through the third message, the first network element can determine whether data collection of the at least one terminal device is authorized. In this way, the first network element can trigger data collection for the authorized terminal device, thereby obtaining the first data.

[0048] In some embodiments combined with the first aspect, in some embodiments, the method can further include: receiving a fourth message sent by the second network element, wherein the fourth message is used to trigger the first network element to perform model performance monitoring.

[0049] In some embodiments combined with the first aspect, in some embodiments, the fourth message can include at least one of: identification information of the first model; and region information used to indicate a first region in which the first model implements positioning.

[0050] In some embodiments combined with the first aspect, in some embodiments, the method can further include: determining model performance of the first model based on the first data; and performing a first operation related to the first model according to the model performance.

[0051] In some embodiments combined with the first aspect, in some embodiments, the first operation can include at least one of: sending a fifth message to the second network element; sending the fifth message to a fourth network element, wherein the fourth network element has the same function as the first network element; and training the first model.

[0052] In some embodiments combined with the first aspect, in some embodiments, the fifth message can be used for at least one of: indicating the model performance of the first model; triggering a change in a positioning method; and triggering training of the first model.

[0053] In a second aspect, the embodiments of the present disclosure provide a model performance monitoring method. The model performance monitoring method is performed by a second network element. The model performance monitoring method includes: receiving a first message sent by a first network element, wherein the first message is used to request to perform data collection on at least one terminal device; and sending a second message to the first network element, wherein the second message includes first data; and wherein the first data is data collected by the second network element on the at least one terminal device, and the first data is used to implement model performance monitoring on a first model.

[0054] Through the present embodiment, the first network element can send a first message to the second network element to request first data, and the second network element sends a second message containing the first data to the first network element. In this way, the first network element can obtain the first data of the at least one terminal device for model performance monitoring of the first model. In this way, the first network element can obtain the first data for model performance monitoring of the first model through interaction with the second network element, thereby effectively obtaining the first data for model performance monitoring of the at least one terminal device.

[0055] In some embodiments combined with the second aspect, in some embodiments, the first message can include identification information of the at least one terminal device.

[0056] In some embodiments of the second aspect, the first model can include at least one of: an AI model; an ML model.

[0057] In some embodiments of the second aspect, the first model can be used to implement AI / ML-based positioning.

[0058] In some embodiments of the second aspect, the first model can be applied to the first area.

[0059] In some embodiments of the second aspect, the first data can include at least one of: measurement data; a calculated position based on the measurement data; an actual position.

[0060] In some embodiments of the second aspect, the method can further include: obtaining the first data according to the first message.

[0061] In some embodiments of the second aspect, the operation of obtaining the first data according to the first message can include at least one of: obtaining measurement data of at least one terminal device; determining a calculated position of the at least one terminal device based on the measurement data; and obtaining an actual position of the at least one terminal device.

[0062] In some embodiments of the second aspect, the method can further include: sending a fourth message to the first network element, wherein the fourth message is used to trigger the first network element to perform model performance monitoring.

[0063] In some embodiments of the second aspect, the fourth message can include at least one of: identification information of the first model; area information used to indicate the first area in which the first model implements positioning.

[0064] In some embodiments of the second aspect, the method can further include: receiving a fifth message sent by the first network element according to the model performance of the first model, wherein the fifth message is used for at least one of: indicating the model performance of the first model; triggering the second network element to change the positioning method; and triggering the second network element to train the first model.

[0065] In a third aspect, the embodiments of the present disclosure provide a model performance monitoring method. The model performance monitoring method is performed by a third network element. The model performance monitoring method includes: sending a third message to a first network element, wherein the third message is used to determine whether data collection for at least one terminal device is authorized; wherein the data collection for at least one device is used to obtain first data, and the first data is used to implement model performance monitoring on a first model.

[0066] Through the embodiment, the third network element can send a third message to the first network element. Through the third message, the first network element can determine whether data collection of the at least one terminal device is authorized. In this way, the first network element can trigger data collection of the authorized terminal device, thereby obtaining first data.

[0067] In some embodiments combining with the third aspect, in some embodiments, the third message can include at least one of the following: identification information of the first model; and region information, used to indicate a first region in which the first model implements positioning.

[0068] In some embodiments combining with the third aspect, in some embodiments, the first model can include at least one of the following: an AI model; and an ML model.

[0069] In some embodiments combining with the third aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0070] In some embodiments combining with the third aspect, in some embodiments, the first model can be applied to the first region.

[0071] In some embodiments combining with the third aspect, in some embodiments, the first data can include at least one of the following: measurement data; a calculated position based on the measurement data; and an actual position.

[0072] In some embodiments combining with the third aspect, in some embodiments, the method can further include: receiving a sixth message sent by the first network element, wherein the sixth message is used to request whether data collection of the at least one terminal device is authorized.

[0073] In a fourth aspect, the embodiments of the present disclosure provide a model performance monitoring apparatus. The model performance monitoring apparatus is arranged in a first network element. The model performance monitoring apparatus includes a transceiver module. The transceiver module is configured to: send a first message to a second network element, wherein the first message is used to request to perform data collection on at least one terminal device; and receive a second message sent by the second network element, wherein the second message includes first data; and wherein the first data is data collected by the second network element on the at least one terminal device, and the first data is used to implement model performance monitoring on a first model.

[0074] In some embodiments combining with the fourth aspect, in some embodiments, the first message can include identification information of the at least one terminal device.

[0075] In some embodiments combining with the fourth aspect, in some embodiments, the first model can include at least one of the following: an AI model; and an ML model.

[0076] In some embodiments combining with the fourth aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0077] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the first model can be applied to the first area.

[0078] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the first data can include at least one of: measurement data; calculated position based on the measurement data; actual position.

[0079] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the apparatus can further include a processing module. The processing module is configured to determine at least one terminal device, wherein the at least one terminal device supports data collection.

[0080] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the transceiver module can be further configured to receive a third message sent by a third network element, wherein the third message is used to determine whether data collection for the at least one terminal device is authorized.

[0081] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the transceiver module can be further configured to receive a fourth message sent by a second network element, wherein the fourth message is used to trigger the first network element to perform model performance monitoring.

[0082] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the fourth message can include at least one of: identification information of the first model; area information used to indicate the first area in which the first model implements positioning.

[0083] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the processing module can be further configured to determine model performance of the first model based on the first data; and perform a first operation related to the first model according to the model performance.

[0084] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the first operation can include at least one of: sending a fifth message to a second network element through the transceiver module; sending a fifth message to a fourth network element through the transceiver module, wherein the fourth network element has the same function as the first network element; training the first model.

[0085] In some embodiments combined with some embodiments of the fourth aspect, in some embodiments, the fifth message can be used for at least one of: indicating model performance of the first model; triggering a change in positioning method; triggering training of the first model.

[0086] In a fifth aspect, the embodiments of the present disclosure provide a model performance monitoring apparatus. The model performance monitoring apparatus is arranged in a second network element. The model performance monitoring apparatus comprises a transceiver module. The transceiver module is configured to: receive a first message sent by a first network element, wherein the first message is used to request to perform data collection on at least one terminal device; and send a second message to the first network element, wherein the second message comprises first data; and wherein the first data is data collected by the second network element on the at least one terminal device, and the first data is used to implement model performance monitoring on a first model.

[0087] In some embodiments in combination with the fifth aspect, in some embodiments, the first message can comprise identification information of the at least one terminal device.

[0088] In some embodiments in combination with the fifth aspect, in some embodiments, the first model can comprise at least one of: an AI model; and an ML model.

[0089] In some embodiments in combination with the fifth aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0090] In some embodiments in combination with the fifth aspect, in some embodiments, the first model can be applied to a first area.

[0091] In some embodiments in combination with the fifth aspect, in some embodiments, the first data can comprise at least one of: measurement data; a calculated position obtained based on the measurement data; and an actual position.

[0092] In some embodiments in combination with the fifth aspect, in some embodiments, the transceiver module can be further configured to: obtain the first data according to the first message.

[0093] In some embodiments in combination with the fifth aspect, in some embodiments, the processing module can be configured to perform at least one of: obtain measurement data of the at least one terminal device through the transceiver module; determine a calculated position of the at least one terminal device based on the measurement data; and obtain an actual position of the at least one terminal device through the transceiver module.

[0094] In some embodiments in combination with the fifth aspect, in some embodiments, the transceiver module can be further configured to: send a fourth message to the first network element, wherein the fourth message is used to trigger the first network element to perform model performance monitoring.

[0095] In some embodiments in combination with the fifth aspect, in some embodiments, the fourth message can comprise at least one of: identification information of the first model; and area information used to indicate a first area in which the first model implements positioning.

[0096] In some embodiments of the fifth aspect, in some embodiments, the transceiver module can be further configured to receive a fifth message sent by the first network element according to the model performance of the first model, wherein the fifth message is used for at least one of the following: indicating the model performance of the first model; triggering the second network element to change the positioning method; triggering the second network element to train the first model.

[0097] In a sixth aspect, the embodiments of the present disclosure provide a model performance monitoring apparatus. The model performance monitoring apparatus is arranged in a third network element. The model performance monitoring apparatus comprises a transceiver module. The transceiver module is configured to send a third message to a first network element, wherein the third message is used to determine whether data collection for at least one terminal device is authorized; wherein the data collection for at least one device is used to obtain first data, and the first data is used to implement model performance monitoring of a first model.

[0098] In some embodiments of the sixth aspect, in some embodiments, the third message can comprise at least one of the following: identification information of the first model; region information used to indicate a first region in which the first model implements positioning.

[0099] In some embodiments of the sixth aspect, in some embodiments, the first model can comprise at least one of the following: an AI model; an ML model.

[0100] In some embodiments of the sixth aspect, in some embodiments, the first model can be used to implement AI / ML-based positioning.

[0101] In some embodiments of the sixth aspect, in some embodiments, the first model can be applied to the first region.

[0102] In some embodiments of the sixth aspect, in some embodiments, the first data can comprise at least one of the following: measurement data; a calculated position obtained based on the measurement data; an actual position.

[0103] In some embodiments of the sixth aspect, in some embodiments, the transceiver module can be further configured to receive a sixth message sent by the first network element, wherein the sixth message is used to request whether data collection for at least one terminal device is authorized.

[0104] In a seventh aspect, the embodiments of the present disclosure provide a communication device. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to any one of the first aspect and possible implementation manners thereof.

[0105] In an eighth aspect, an embodiment of the present disclosure provides a communication device. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to any one of the second aspect and possible implementation manners thereof.

[0106] In a ninth aspect, an embodiment of the present disclosure provides a communication device. The communication device comprises one or more processors and a memory storing instructions. The instructions, when executed by the communication device, cause the communication device to implement the model performance monitoring method according to any one of the third aspect and possible implementation manners thereof.

[0107] In a tenth aspect, an embodiment of the present disclosure provides a communication system. The communication system comprises at least a first network element configured to implement the model performance monitoring method according to any one of the first aspect and possible implementation manners thereof, and a second network element configured to implement the model performance monitoring method according to any one of the second aspect and possible implementation manners thereof.

[0108] In an eleventh aspect, an embodiment of the present disclosure provides a storage medium. The storage medium stores instructions. The instructions, when executed on a communication device, cause the communication device to perform the model performance monitoring method according to any one of the first aspect to the third aspect and possible implementation manners thereof.

[0109] In a twelfth aspect, an embodiment of the present disclosure provides a program product. The program product, when executed by a communication device, causes the communication device to perform the model performance monitoring method according to any one of the first aspect to the third aspect and possible implementation manners thereof.

[0110] In a thirteenth aspect, an embodiment of the present disclosure provides a computer program (product). The computer program, when executed on a computer, causes the computer to perform the model performance monitoring method according to any one of the first aspect to the third aspect and possible implementation manners thereof.

[0111] In a fourteenth aspect, an embodiment of the present disclosure provides a chip or chip system. The chip or chip system comprises a processing circuitry. The processing circuitry is configured to perform the model performance monitoring method according to any one of the first aspect to the third aspect and possible implementation manners thereof.

[0112] It can be understood that the above model performance monitoring apparatus, communication device, communication system, storage medium, program product, computer program, chip, and chip system are all used to perform the model performance monitoring method provided by the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved thereby can refer to the beneficial effects in the corresponding method, which will not be described here again.

[0113] The embodiments of the present disclosure provide a model performance monitoring method and device, a communication device, a communication system, a storage medium and a computer program product. In some embodiments, the terms of the model performance monitoring method, the communication method, the information processing method, etc. can be replaced with each other, and the terms of the model performance monitoring device, the communication device, the communication equipment, the information processing device, etc. can be replaced with each other, and the terms of the information processing system, the communication system, etc. can be replaced with each other.

[0114] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily. In addition, the optional implementation manners in an embodiment can be combined arbitrarily. In addition, the embodiments can be combined arbitrarily. For example, part or all of the steps of different embodiments can be combined arbitrarily. For another example, an embodiment can be combined with the optional implementation manners of other embodiments.

[0115] In the embodiments of the present disclosure, the terms and / or descriptions between the embodiments are consistent if there is no special description and logical conflict, and can be referred to each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0116] The terms used in the embodiments of the present disclosure are only for the purpose of describing the specific embodiments, and not as a limitation on the present disclosure.

[0117] In the embodiments of the present disclosure, unless otherwise specified and logically contradictory, the elements expressed in singular form, such as "one", "a", "the", "above", "said", "preceding", "this" and the like, can represent "one and only one", or "one or more", "at least one" and the like. For example, in the case of using articles such as "a", "an", "the" in English, the noun after the article can be understood as singular expression, or as plural expression.

[0118] In the embodiments of the present disclosure, "plurality" means two or more.

[0119] In some embodiments, the terms "at least one of", "one or more", "a plurality of", "multiple", etc. can be replaced with each other.

[0120] In some embodiments, "at least one of A, B", "A and / or B", "in one case A, in another case B", "in response to case A, in response to case B", and the like, can include the following technical solutions: in some embodiments, A (A is executed regardless of B); in some embodiments, B (B is executed regardless of A); in some embodiments, A and B are selected from A and B (A and B are selectively executed); in some embodiments, A and B (A and B are executed). When there are more branches such as A, B, C, and the like, the above is similar.

[0121] In some embodiments, "A or B" and the like can include the following technical solutions according to the case: in some embodiments, A (A is executed regardless of B); in some embodiments, B (B is executed regardless of A); in some embodiments, A and B are selected from A and B (A and B are selectively executed). When there are more branches such as A, B, C, and the like, the above is similar.

[0122] The prefix words "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute a limitation on the position, order, priority, quantity or content of the description objects. The description of the description objects should refer to the description in the context of the claims or embodiments, and should not constitute an additional limitation because of the use of the prefix words. For example, the description object is "field", and the ordinal words before "field" in "first field" and "second field" do not limit the position or order between "fields", and "first" and "second" do not limit whether the "fields" modified thereby are in the same message or not, nor limit the order of "first field" and "second field". For another example, the description object is "level", and the ordinal words before "level" in "first level" and "second level" do not limit the priority between "levels". For another example, the quantity of the description object is not limited by the ordinal words, and can be one or more. For example, "first device", wherein the quantity of "device" can be one or more. In addition, the objects modified by different prefix words can be the same or different, for example, the description object is "device", and "first device" and "second device" can be the same device or different devices, and the types thereof can be the same or different; for another example, the description object is "information", and "first information" and "second information" can be the same information or different information, and the content thereof can be the same or different.

[0123] In some embodiments, "including A", "containing A", "for indicating A", "carrying A" can be interpreted as directly carrying A, or indirectly indicating A.

[0124] In some embodiments, the terms "in response to", "in response to determining", "in the case of", "when", "when", "if", and the like can be replaced with each other.

[0125] In some embodiments, an apparatus or the like can be interpreted as an entity, and can also be interpreted as virtual, and the name thereof is not limited to the name described in the embodiments. The terms "apparatus", "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject" and the like can be replaced with each other.

[0126] In some embodiments, a "network" can be interpreted as an apparatus (for example, an access network device, a core network device, and the like) included in the network.

[0127] In some embodiments, the terms "access network device (AN device)", "radio access network device (RAN device)", "base station (BS)", "radio base station", "fixed station", "node", "access point", "transmission point (TP)", "reception point (RP)", "transmission / reception point (TRP)", "panel", "antenna panel", "antenna array", "cell", "macro cell", "small cell", "femto cell", "pico cell", "sector", "cell group", "serving cell", "carrier", "component carrier", "bandwidth part (BWP)" and the like can be replaced with each other.

[0128] In some embodiments, the terms "terminal," "terminal device," "user equipment (UE)," "user terminal," "mobile station (MS)," "mobile terminal (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile terminal," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," and so on can be replaced with each other.

[0129] In some embodiments, the access network device, the core network device, or the network device can be replaced with a terminal. For example, the embodiments of the present disclosure can also be applied to a structure in which communication between the access network device, the core network device, or the network device and the terminal is replaced with communication between a plurality of terminals (e.g., device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, the terminal can also be configured to have all or part of the functions of the access network device. In addition, the terms "uplink," "downlink," and the like can also be replaced with terms corresponding to the inter-terminal communication (e.g., "side"). For example, the uplink channel, the downlink channel, and the like can be replaced with the side channel, and the uplink, the downlink, and the like can be replaced with the sidelink.

[0130] In some embodiments, the terminal can be replaced with the access network device, the core network device, or the network device. In this case, the access network device, the core network device, or the network device can also be configured to have all or part of the functions of the terminal.

[0131] In some embodiments, the data, information, etc. can be obtained in compliance with the laws and regulations of the country in which the location is situated.

[0132] In some embodiments, the data, information, etc. can be obtained after obtaining the consent of the user.

[0133] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0134] FIG. 1 is a schematic diagram of an architecture of a communication system according to an embodiment of the present disclosure. As shown in FIG. 1, the communication system 100 includes a terminal 101, an access network device 102, and a core network 103.

[0135] In some embodiments, the terminal 101 includes at least one of a mobile phone, a wearable device, an Internet of Things device, a car with communication function, a smart car, a Pad, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, a wireless terminal device in smart home, etc., but is not limited thereto.

[0136] In some embodiments, the access network device 102, for example, can be a node or device that accesses a terminal to a wireless network, and the access network device 102 can include at least one of an evolved node B (eNB) in a 5G communication system, a next generation eNB (ng-eNB), a next generation node B (gNB), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an open base station (Open RAN), a cloud base station (Cloud RAN), a satellite base station, a base station in other communication systems, an access node in a Wi-Fi system, but is not limited thereto.

[0137] In some embodiments, the terminal 101 and the core network device 103 can interact through the access network device 102. In some embodiments, the terminal 101 and the core network 103 can directly interact. In this regard, the embodiments of the present disclosure are not limited specifically.

[0138] In some embodiments, the technical solutions of the present disclosure can be applied to an Open RAN architecture, at this time, the interfaces between or within the access network devices involved in the embodiments of the present disclosure can become internal interfaces of the Open RAN, and the processes and information interactions between these internal interfaces can be realized through software or programs.

[0139] In some embodiments, the access network device 102 can be composed of a central unit (CU) and a distributed unit (DU), wherein the CU can also be referred to as a control unit (control unit). The CU-DU structure can split the protocol layers of the access network device, and part of the functions of the protocol layers are controlled by the CU, and the remaining part or all of the functions of the protocol layers are distributed in the DU and controlled by the CU, but the present disclosure is not limited thereto.

[0140] In some embodiments, the core network 103 can be one device including one or more network elements, or can be multiple devices or device groups each including all or part of the one or more network elements described above. The network elements can be virtual or physical. The core network includes, for example, at least one of an evolved packet core (EPC), a 5G core network (5GCN), a 6G core network (5GCN), and a next generation core (NGC).

[0141] As shown in FIG. 1, the core network 103 can include at least one of a first network element 1031, a second network element 1032, a third network element 1033, a fourth network element 1034, a fifth network element 1035, and a sixth network element 1036.

[0142] In some embodiments, the first network element 1031 can be configured to perceive and analyze the network based on network data, and improve user service experience.

[0143] In some embodiments, the first network element 1031 can be a network data analytics function (NWDAF).

[0144] In some embodiments, the second network element 1032 can be configured to provide positioning services for terminals.

[0145] In some embodiments, the second network element 1032 can be a location management function (LMF).

[0146] In some embodiments, the third network element 1033 can be configured to manage and store data related to users, and provide authentication, authorization, and user configuration, etc.

[0147] In some embodiments, the third network element 1033 can be a unified data management (UDM) function.

[0148] In some embodiments, the third network element 1033 can be a unified data repository (UDR) function.

[0149] In some embodiments, the fourth network element 1034 can be configured to perceive and analyze the network based on network data, and improve user service experience.

[0150] In some embodiments, the fourth network element 1034 can be a NWDAF.

[0151] In some embodiments, the fifth network element 1035 can be configured to implement storage of data.

[0152] In some embodiments, the fifth network element 1035 can be an analytics data repository function (ADRF).

[0153] In some embodiments, the fifth network element 1035 can be a UDM function.

[0154] In some embodiments, the fifth network element 1035 can be a UDR function.

[0155] In some embodiments, the sixth network element 1036 can be configured to be responsible for registration management, connection management, mobility management, without limitation.

[0156] In some embodiments, the sixth network element 1036 can be, for example, an access and mobility management function (AMF).

[0157] In some embodiments, the first network element 1031 and the fourth network element 1034 can have the same function, but the first network element 1031 and the fourth network element 1034 are different network elements.

[0158] In some embodiments, in the case that the third network element 1033 and the fifth network element 1035 are both UDMs, the third network element 1033 and the fifth network element 1035 can be the same UDM. In some embodiments, in the case that the third network element 1033 and the fifth network element 1035 are both UDMs, the third network element 1033 and the fifth network element 1035 can be different UDMs.

[0159] In some embodiments, in the case that the third network element 1033 and the fifth network element 1035 are both UDRs, the third network element 1033 and the fifth network element 1035 can be the same UDR. In some embodiments, in the case that the third network element 1033 and the fifth network element 1035 are both UDRs, the third network element 1033 and the fifth network element 1035 can be different UDRs.

[0160] In some embodiments, the communication system 100 can be a 5G communication system. In some embodiments, at least one of the first network element 1031, the second network element 1032, the third network element 1033, the fourth network element 1034, the fifth network element 1035, and the sixth network element 1036 can be a network element in the 5G communication system. In some embodiments, the communication system 100 can be a 6G communication system. In some embodiments, at least one of the first network element 1031, the second network element 1032, the third network element 1033, the fourth network element 1034, the fifth network element 1035, and the sixth network element 1036 can be a network element in the 6G communication system. It should be noted that the communication system 100 can also be other communication systems, for example, a 4G communication system, a 7G communication system, and the like, and the present embodiments are not limited thereto.

[0161] In some embodiments, at least one of the first network element 1031, the second network element 1032, the third network element 1033, the fourth network element 1034, the fifth network element 1035, and the sixth network element 1036 can be a network function of a control plane. In some embodiments, at least one of the first network element 1031, the second network element 1032, the third network element 1033, the fourth network element 1034, the fifth network element 1035, and the sixth network element 1036 can be a network function of a user plane. In some embodiments, at least one of the first network element 1031, the second network element 1032, the third network element 1033, the fourth network element 1034, the fifth network element 1035, and the sixth network element 1036 can be a network function of a user plane. In some embodiments, at least one of the first network element 1031, the second network element 1032, the third network element 1033, the fourth network element 1034, the fifth network element 1035, and the sixth network element 1036 can be a network function of a data plane. In some embodiments, at least one of the first network element 1031, the second network element 1032, the third network element 1033, the fourth network element 1034, the fifth network element 1035, and the sixth network element 1036 can be a network function of a computing plane.

[0162] It can be understood that the communication system described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the present disclosure, and does not constitute a limitation on the technical solutions proposed in the present disclosure. It can be known by those skilled in the art that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions proposed in the present disclosure are also applicable to similar technical problems.

[0163] The following embodiments of the present disclosure can be applied to the communication system 100 shown in FIG. 1, or part of the subjects in the communication system 100, but are not limited thereto. The subjects shown in FIG. 1 are illustrative, the communication system 100 can include all or part of the subjects in FIG. 1, or other subjects other than FIG. 1, the number and form of each subject is arbitrary, each subject can be real or virtual, the connection relationship between each subject is illustrative, each subject can not be connected or can be connected, and the connection can be in any way, can be direct connection or indirect connection, can be wired connection or wireless connection.

[0164] Embodiments of the present disclosure can be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G new radio (NR), 6th generation mobile communication system (6G), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New radio access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (Bluetooth (registered trademark)), Public Land Mobile Network (PLMN) network, Device-to-Device (D2D) system, Machine to Machine (M2M) system, Internet of Things (IoT) system, Vehicle-to-Everything (V2X), system using other communication methods, next-generation system expanded based thereon, and the like. Further, a plurality of systems can be applied in combination (for example, a combination of 5G and 6G, and the like).

[0165] In some embodiments, a communication system can support a location service (LCS). The communication system can provide location information of a terminal for a requestor of the location service. The requestor of the location service can be an application function (AF), a consumer network function (NF), a terminal, etc.

[0166] In some embodiments, AI technology can be integrated with communication technology, thereby enhancing the capability of a communication system. In particular, a communication system can implement a location service through AI technology. In some embodiments, an AI / machine learning (ML) model can be deployed in a communication system. In some embodiments, the AI / ML model can be deployed in an LMF of the communication system. The LMF can utilize the AI / ML model to provide a location service. In an example, the LMF can perform positioning inference based on the AI / ML model and provide an inference result to a requestor. In some embodiments, a model training logic function (MTLF) can also be deployed in the communication system. The MTLF can be responsible for training of the AI / ML model.

[0167] In some embodiments, in order to ensure the quality of service of a location service provided based on an AI / ML model (or simply referred to as AI / ML positioning), model performance monitoring can be performed on the AI / ML model. In other words, the LMF or the MTLF can perform model performance monitoring on the model-based positioning. In some embodiments, in order to implement model performance monitoring, the LMF can provide inference data related to the inference of the AI / ML model-based positioning to the MTLF for model performance monitoring by the MTLF.

[0168] In some embodiments, the MTLF can be located in a NWDAF. The NWDAF can have an analysis capability and / or an accurate verification capability for the AI / ML model. In an example, model performance monitoring can be performed by the NWDAF.

[0169] FIG. 2 is an interaction diagram of model performance monitoring in the related art. As shown in FIG. 2, the process of model performance monitoring can include steps S201 to S207.

[0170] In step S201, an AF / consumer NF subscribes to an LMF to request AI / ML positioning.

[0171] In step S202, the LMF performs UE location data collection to obtain location data from a UE, a RAN, an AMF, a gateway mobile location center (GMLC), and the like.

[0172] In step S203, the LMF performs direct AI / ML positioning inference using the trained AI / ML model.

[0173] In step S204, step S204a and step S204b can be included. In step S204a, the LMF notifies the inference result of the AI / ML positioning inference to an AF / consumer NF; and in step S204b, related inference data of the AI / ML positioning inference can be sent to the MTLF for monitoring model performance.

[0174] In step S205, the MTLF performs UE location data collection for model performance monitoring of the AI / ML positioning inference.

[0175] In step S206, the MTLF can obtain model performance based on the collected UE location data, and can decide to retrain and / or re-provision the LMF for the AI / ML model.

[0176] In step S207, the MTLF can provide a model performance report to the LMF. After the newly generated or retrained AI / ML model is ready, the MTLF can send the AI / ML model to the LMF.

[0177] In order to implement model performance monitoring, the acquisition of UE location data is a key link. Therefore, how to effectively acquire UE location data is a problem to be solved.

[0178] FIG. 3A is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure. The model performance monitoring method related by the embodiment of the present disclosure can be applied to the communication system 100. As shown in FIG. 3A, the model performance monitoring method of the embodiment of the present disclosure includes steps S3101 to S3110.

[0179] In step S3101, the second network element 1032 sends a fourth message to the first network element 1031.

[0180] In some embodiments, the first network element 1031 can receive the fourth message.

[0181] In some embodiments, the first network element 1031 can be a NWDAF. In some embodiments, the first network element 1031 can be a MTLF. In an example, the first network element 1031 can be a MTLF located in a NWDAF. It can be understood that the first network element 1031 can also be other network functions, which are not specifically limited in the embodiments of the present disclosure.

[0182] In some embodiments, the second network element 1032 can be a LMF. It can be understood that the second network element 1032 can also be other network functions, which are not specifically limited in the embodiments of the present disclosure.

[0183] In some embodiments, the fourth message can be used to trigger the first network element 1031 to perform model performance monitoring.

[0184] In some embodiments, the model performance monitoring can be determining model performance of the first model. In some embodiments, the fourth message can be used to request to determine model performance of the first model.

[0185] In some embodiments, the first model can be used to implement a location service. In an example, the first model can be used to implement positioning for one or more terminals. In some embodiments, the location service implemented based on the first model can include at least one of the following: determining a current location of a terminal, predicting a location of a terminal.

[0186] In some embodiments, the first model can include at least one of the following: an AI model, a ML model. In some embodiments, the first model can be used to implement AI / ML based positioning. For example, the AI model can be used to implement AI model based positioning. For example, the ML model can be used to implement ML based positioning.

[0187] In some embodiments, the name of the fourth message is not limited, which can be, for example, a model performance monitoring request message, a model performance monitoring indication message, etc.

[0188] In some embodiments, the fourth message can include at least one of the following: identification information of the first model, area information.

[0189] In some embodiments, the identification information of the first model can be used to identify the first model. Through the identification information, the second network element 1032 can indicate to perform model performance monitoring on the first model associated with the identification information.

[0190] In some embodiments, the identification information of the first model can include an identifier.

[0191] In some embodiments, the identifier can be used to identify each first model. In an example, one first model can be uniquely identified by an identifier. Different first models can be identified by different identifiers.

[0192] In some embodiments, the identifier can be used to identify a model group composed of one or more first models. In an example, one model group can be uniquely identified by the identifier. Different model groups can be identified by different identifiers.

[0193] In some embodiments, the area information can be used to indicate a first area in which the first model implements positioning.

[0194] In some embodiments, the first area can be an area in which the positioning based on the first model is directed. In some embodiments, the positioning based on the first model can be based on measurements within the first area. For example, the object to be positioned can be located within the first area. For example, the terminal and / or network device used to perform the positioning can be located within the first area.

[0195] In some embodiments, the object to be positioned can be a UE.

[0196] In some embodiments, the terminal performing the positioning can be a UE. In some embodiments, the terminal performing the positioning can be a positioning reference unit (PRU).

[0197] In some embodiments, the network device performing the positioning can be an access network device.

[0198] In some embodiments, after receiving the fourth message, the first network element 1031 can determine, according to the fourth message, that the model performance monitoring is performed for the first model. In an example, the first network element 1031 can determine, according to the identification information and / or the area information in the fourth message, that the model performance monitoring is for the first model.

[0199] In step S3102, the first network element 1031 determines at least one terminal 101.

[0200] In some embodiments, upon receiving the fourth message, the first network element 1031 can determine at least one terminal 101 related to the model performance monitoring.

[0201] In some embodiments, the at least one terminal 101 determined by the first network element 1031 can be a terminal 101 supporting model performance monitoring.

[0202] In some embodiments, step S3102 can include that the first network element 1031 interacts with the sixth network element 1036 to determine the terminal 101 supporting model performance monitoring.

[0203] In some embodiments, the terminal 101 supporting model performance can be regarded as a terminal supporting data collection.

[0204] In some embodiments, the terminal 101 supporting model performance monitoring can be a terminal 101 capable of providing data for inference of the first model identified by the identification information. For example, in the case that the data provided by the terminal 101 can be used for inference of the first model, the terminal 101 can support model performance monitoring. For example, the terminal 101 supporting model performance monitoring can provide at least one of the following: measurement data, a calculated position based on the measurement data, an actual position.

[0205] In some embodiments, the first network element 1031 can obtain the capability information of the terminal from the sixth network element 1036, and determine the terminal 101 supporting model performance monitoring according to the capability information.

[0206] In some embodiments, the terminal 101 supporting model performance monitoring can be a terminal located in the first area indicated by the area information.

[0207] In some embodiments, the first network element 1031 can obtain the location of the terminal from the sixth network element 1036, and determine the terminal 101 supporting model performance monitoring according to the location of the terminal. In some embodiments, the first network element 1031 can obtain the position that the terminal can sense from the sixth network element 1036, and determine the terminal 101 supporting model performance monitoring according to the position that the terminal can sense.

[0208] In some embodiments, the at least one terminal 101 determined by the first network element 1031 can include at least one of the following: a UE, a PRU.

[0209] In step S3103, the first network element 1031 determines whether data collection for model performance monitoring is authorized.

[0210] In some embodiments, after determining the terminal 101, the first network element 1031 can determine whether data collection for model performance monitoring of the terminal 101 is authorized.

[0211] In some embodiments, the first network element 1031 can interact with the third network element 1033 to determine whether data collection for model performance monitoring is authorized.

[0212] In some embodiments, step S3103 can include: the third network element 1033 sends a third message to the first network element 1031. The third message can be used to determine whether data collection for the terminal 101 is authorized.

[0213] In some embodiments, the third network element 1033 can hold subscription information of the terminal 101. The third network element 1033 can determine whether the terminal 101 is authorized to perform the data collection for the model performance monitoring according to the subscription information. For example, the subscription information can indicate that the terminal 101 is authorized to perform the data collection for the model performance monitoring. For example, the subscription information can indicate that the terminal 101 is not authorized to perform the data collection for the model performance monitoring.

[0214] In some embodiments, the subscription information of the terminal 101 can contain a permission of a user of the terminal 101 with respect to the model performance monitoring. In an example, the subscription information indicates that the user permits the monitoring, and the terminal 101 is authorized to perform the data collection for the model performance monitoring. In an example, the subscription information indicates that the user does not permit the monitoring or does not indicate whether the user permits the monitoring, and the terminal 101 is not authorized to perform the data collection for the model performance monitoring.

[0215] In some embodiments, the third message can indicate that the data collection of the terminal 101 is authorized. In some embodiments, the third message can indicate that the data collection of the terminal 101 is not authorized.

[0216] In some embodiments, the step S3103 can further include that the first network element 1031 sends a sixth message to the third network element 1033. The sixth message is used to request whether the data collection of the terminal 101 is authorized.

[0217] In some embodiments, the third network element 1033 can send the third message to the first network element 1031 according to the sixth message after receiving the sixth message. For example, the third message can be sent in response to the sixth message.

[0218] In some embodiments, the third network element 1033 can send the third message to the first network element 1031 without receiving the sixth message. For example, the third network element 1033 can be triggered to send the third message. For example, the third network element 1033 can send the third message in a case that the third network element 1033 obtains the subscription information of the terminal 101 or in a case that the subscription information of the terminal 101 in the third network element 1033 changes. For example, the third network element 1033 can periodically send the third message.

[0219] In the step S3104, the first network element 1031 sends the first message to the second network element 1032.

[0220] In some embodiments, the first network element 1031 can send the first message to the second network element 1032 in a case that it is determined to perform the model performance monitoring for the first model.

[0221] In some embodiments, the second network element 1032 can receive the first message.

[0222] In some embodiments, the first message can be used to request the first data from the second network element 1032. The first data is used to implement the performance monitoring of the first model.

[0223] In some embodiments, the first message can be used to request the second network element 1032 to provide the first data.

[0224] In some embodiments, the first message can be used to request the second network element 1032 to perform data collection for the first model.

[0225] In some embodiments, the name of the first message is not limited, which can be, for example, a data request message, a model performance monitoring indication message, a data collection request message, etc.

[0226] In some embodiments, the first message can comprise the identification information of the terminal 101.

[0227] In some embodiments, the first network element 1031 can carry the determined identification information of the terminal 101 in the first message.

[0228] In some embodiments, the identification information of the terminal 101 can be used to identify the terminal 101. In an example, the identification information of the terminal 101 can comprise an identifier of the terminal 101.

[0229] In some embodiments, the first message can comprise the identification information of each terminal 101. For example, the first message can comprise the identification information of each UE. For example, the first message can comprise the identification information of each PRU.

[0230] In some embodiments, the first message can further comprise at least one of the following: indication information, identification information of the first model, area information.

[0231] In some embodiments, the indication information can be used to indicate the model performance monitoring for the first model. In an example, the indication information can be used to indicate the model performance monitoring for the model in the first network element 1031. In an example, the indication information can be used to indicate the model performance monitoring for the model in the second network element 1032.

[0232] In some embodiments, there can be only one model, for example, the first model, in the first network element 1031. In this case, the first message can only contain the indication information to indicate the model performance monitoring. It can be understood that, because there is only one model in the first network element 1031, the first message can request the first data for the first model by default.

[0233] In some embodiments, there can be only one model, e.g., the first model, in the second network element 1032. In this case, the first message can only contain the indication information to indicate that the model performance monitoring is performed. It can be appreciated that, because there is only one model in the second network element 1032, the first message can request the first data for the first model by default.

[0234] In some embodiments, the first message can further comprise the identification information of the first model and / or the area information. The identification information of the first model and / or the area information can be acquired by the first network element 1031 from the fourth message and carried in the first message.

[0235] In some embodiments, the first message can comprise the identification information of the first model if the third message comprises the identification information of the first model. In some embodiments, the first message can not comprise the identification information of the first model if the third message comprises the identification information of the first model and there is only one first model in the first network element 1031 and / or the second network element 1032.

[0236] In some embodiments, the first message can comprise the area information if the fourth message comprises the area information.

[0237] In step S3105, the second network element 1032 triggers the positioning.

[0238] In some embodiments, the second network element 1032 can trigger the positioning based on the first model according to the first message.

[0239] In some embodiments, the positioning of the first model triggered by the second network element 1032 can be for the terminal 101. In an example, the second network element 1032 can trigger the positioning based on the first model for the terminal 101. At this time, the terminal 101 can comprise the object to be positioned.

[0240] In step S3106, the second network element 1032 can acquire the first data.

[0241] In some embodiments, the first data can comprise at least one of the following: measurement data, calculated position, actual position.

[0242] In some embodiments, the measurement data can comprise the measurement data provided by the terminal 101 and / or the access network device 102.

[0243] In some embodiments, the calculated position can be the position calculated based on the measurement data provided by the terminal 101 and / or the access network device 102.

[0244] In some embodiments, the actual position can be an actual position of the terminal 101. In an example, the actual position can include a position of the terminal 101 determined by a global navigation satellite system (GNSS) or other means. The GNSS used to determine the actual position can include at least one of the following: global positioning system (GPS), beidou navigation satellite system (BDS), Galileo satellite navigation system (Galileo), global navigation satellite system (GLONASS), quasi-zenith satellite system (QZSS).

[0245] In some embodiments, the first data can be associated with a first area in which the first model enables positioning. In an example, the first data can include data associated with the first model for terminals 101 within the first area. In an example, the first data can include data associated with the first model for terminals 101 and / or access network devices 102 performing positioning within the first area.

[0246] In some embodiments, the first data can be for each first area. In some embodiments, the first data can be acquired for the first model and each first area.

[0247] In some embodiments, the second network element 1032 acquires measurement data and / or computes a position sent by the terminal 101 and / or the access network device 102. In an example, the second network element 1032 acquires measurement data and computes a position sent by the terminal 101 and / or the access network device 102. At this time, the computed position can be determined by the terminal 101 and / or the access network device 102 according to the measurement data. In an example, the second network element 1032 acquires measurement data sent by the terminal 101 and / or the access network device 102. At this time, the computed position can be determined by the second network element 1032 according to the measurement data.

[0248] In some embodiments, the second network element 1032 acquires an actual position of the terminal 101. In an example, the second network element 1032 can receive an actual position of the terminal 101 determined by a GNSS.

[0249] It can be understood that the measurement data of the terminal 101, the calculated position, and the actual position can be sent at the same time or in sequence, and the disclosure embodiments do not make specific limitations.

[0250] In step S3107, the second network element 1032 sends a second message to the first network element 1031.

[0251] In some embodiments, the first network element 1031 can receive the second message.

[0252] In some embodiments, the second message can be sent by the second network element 1032 in response to the first message.

[0253] In some embodiments, the second message can be used to return data for the first model to the first network element 1031.

[0254] In some embodiments, the name of the second message is not limited, which can be, for example, a data response message, a model performance monitoring data message, a data collection response message, etc.

[0255] In some embodiments, the second message can include the first data.

[0256] In some embodiments, the second network element 1032 can send the first data obtained in step S3106 to the first network element 1031 through the second message.

[0257] In some embodiments, the second message can also include identification information of the first model. In some embodiments, according to the identification information, the second network element 1032 can indicate that the first data is used for model performance monitoring of the first model.

[0258] In step S3108, the first network element 1031 performs model performance monitoring.

[0259] In some embodiments, after receiving the second message, the first network element 1031 can perform model performance monitoring based on the obtained first data.

[0260] In some embodiments, the model performance monitoring performed by the first network element 1031 can include that the first network element 1031 determines the model performance of the first model based on the first data.

[0261] In some embodiments, the first network element 1031 can evaluate the model performance of the first model based on the first data. In this way, the first network element 1031 can obtain an evaluation result. The evaluation result can indicate the model performance of the first model.

[0262] In some embodiments, the first network element 1031 can determine whether the model performance of the first model meets a threshold through model performance monitoring. In some embodiments, the first network element 1031 can compare the model performance of the first model with the threshold. The comparison result can comprise one of the following: the model performance is lower than the threshold, the model performance is higher than the threshold.

[0263] In some embodiments, the first network element 1031 determines whether the model performance of the first model is good or not. In an example, in a case where the performance result of the first model is higher than the threshold, the first network element 1031 can determine that the model performance of the first model is good (or meets the requirement). In an example, in a case where the performance result of the first model is lower than the threshold, the first network element 1031 can determine that the model performance of the first model is not good (or does not meet the requirement).

[0264] In step S3109, the first network element 1031 performs model training.

[0265] In some embodiments, the first network element 1031 can trigger training of the first model according to the model performance.

[0266] In some embodiments, in a case where the model performance of the first model is determined to be lower than the threshold, the first network element 1031 can trigger training of the first model. In some embodiments, in a case where the model performance of the first model is determined to be lower than the threshold, the first network element 1031 can perform training for the first model.

[0267] In some embodiments, in a case where the model performance of the first model is determined to be not good, the first network element 1031 can trigger training of the first model. In some embodiments, in a case where the model performance of the first model is determined to be not good, the first network element 1031 can perform training for the first model.

[0268] Through step S3109, training of the first model can be implemented by the first network element 1031.

[0269] In step S3110, the first network element 1031 sends a fifth message to the second network element 1032.

[0270] In some embodiments, the second network element 1032 can receive the fifth message.

[0271] In some embodiments, the fifth message can be sent according to the model performance.

[0272] In some embodiments, the fifth message can be sent in response to the fourth message.

[0273] In some embodiments, the fifth message can be used to inform the second network element 1032 of the result of model performance monitoring.

[0274] In some embodiments, the name of the fifth message is not limited, which can be, for example, a model performance monitoring response message, a model performance monitoring notification message, etc.

[0275] In some embodiments, the fifth message can be used for at least one of the following: indicating the model performance of the first model, triggering a change of positioning method, triggering training of the first model.

[0276] In some embodiments, the fifth message can include performance information. The performance information can be used to indicate the model performance of the first model. In an example, the performance information can indicate a model performance value determined based on the model performance monitoring of the first model. In an example, the performance information can indicate whether the model performance of the first model meets a threshold. For example, the performance information can indicate that the model performance of the first model is higher than the threshold. For example, the performance information can indicate that the model performance of the first model is lower than the threshold. In an example, the performance information can indicate that the model performance of the first model is good or bad. For example, the performance information can indicate that the model performance of the first model is good. For example, the performance information can indicate that the model performance of the first model is bad.

[0277] In some embodiments, the fifth message can include first trigger information. The first trigger information can be used to trigger a change of positioning method.

[0278] In some embodiments, the positioning method can include at least one of the following: AI / ML based positioning, legacy positioning.

[0279] In some embodiments, the first trigger information can indicate a change from AI / ML based positioning to legacy positioning. In an example, in a case where the model performance of the first model is lower than a threshold, and / or in a case where the model performance of the first model is bad, the fifth message can include the first trigger information to indicate a change from AI / ML based positioning to legacy positioning. At this time, the first trigger information can indicate a change from positioning based on the first model to legacy positioning.

[0280] In some embodiments, the first trigger information can indicate a change from legacy positioning to AI / ML based positioning. In an example, in a case where the model performance of the first model is higher than a threshold, and / or in a case where the model performance of the first model is good, the fifth message can include the first trigger information to indicate a change from legacy positioning to AI / ML based positioning. At this time, the first trigger information can indicate a change from legacy positioning to positioning based on the first model.

[0281] In some embodiments, the fifth message can include second trigger information. The second trigger information can be used to trigger training of the first model.

[0282] In some embodiments, the second trigger information can indicate to train the first model. In an example, the fifth message can comprise the second trigger information to indicate to train the first model in a case that the model performance of the first model is below a threshold, and / or in a case that the model performance of the first model is poor.

[0283] In some embodiments, the fifth message can further comprise the trained first model. In an example, the fifth message can further comprise the model parameters of the trained first model in step S3109.

[0284] It is to be noted that the training of the first model can be implemented in the first network element 1031 and / or the second network element 1032. In an example, the training of the first model can be implemented in the first network element 1031 or the second network element 1032. For example, the training of the first model can be implemented in the first network element 1031. At this time, step S3109 can be performed, and the fifth message can not be used to trigger the second network element 1032 to train the first model. For example, the training of the first model can be implemented in the second network element 1032. At this time, step S3109 can not be performed, and the fifth message can be used to trigger the second network element 1032 to train the first model.

[0285] In some embodiments, the second network element 1032 can obtain the model performance of the first model according to the fifth message. In an example, the second network element 1032 can obtain the model performance of the first model according to the performance information in the fifth message. In an example, the second network element 1032 can trigger to change the positioning method according to the first trigger information in the fifth message. In an example, the second network element 1032 can trigger to train the first model according to the second trigger information in the fifth message.

[0286] In some embodiments, the second network element 1032 can determine to trigger to change the positioning method according to the performance information carried in the fifth message. In some embodiments, the second network element 1032 can determine to trigger to train the first model according to the performance information carried in the fifth message.

[0287] In some embodiments, the fifth message may be sent periodically or non-periodically. In one example, the first network element 1031 may send the fifth message once based on the received fourth message. For example, the fourth message may be a request message, and the fifth message may be a response message to the request message. For example, the fourth message may be a query message, and the fifth message may be a feedback message to the query message. In one example, the first network element 1031 may send the fifth message once or multiple times based on the received fourth message. For example, the fourth message may be a subscription message, and the fifth message may be a notification message for the subscription message. For example, the fifth message may be sent periodically. For example, the fifth message may be sent when a preset condition (e.g., a threshold) is met.

[0288] In some embodiments, training the first model may include retraining or refrigerating the first model.

[0289] The model performance monitoring method of this disclosure embodiment can be implemented through steps S3101 to S3110.

[0290] The model performance monitoring method disclosed in this embodiment may include at least one of steps S3101 to S3110. For example, step S3101 may be implemented as a standalone embodiment. For example, step S3103 may be implemented as a standalone embodiment. For example, a combination of steps S3104 may be implemented as a standalone embodiment. For example, step S3107 may be implemented as a standalone embodiment. For example, a combination of steps S3101 and S3103 may be implemented as a standalone embodiment. For example, a combination of steps S3101 and S3104 may be implemented as a standalone embodiment. For example, a combination of steps S3103 and S3104 may be implemented as a standalone embodiment. For example, a combination of steps S3104 and S3107 may be implemented as a standalone embodiment. For example, a combination of steps S3103, S3104, and S3107 may be implemented as a standalone embodiment. It should be noted that the possible standalone embodiments consisting of one or more steps S3101 to S3110 are not limited thereto.

[0291] In some embodiments, at least two of steps S3101 to S3110 may be executed in an interchangeable order or simultaneously. For example, steps S3109 and S3110 may be executed in an interchangeable order or simultaneously.

[0292] In some embodiments, steps S3102, S3103, S3104, S3105, S3106, S3107, S3108, S3109, S3110 are optional, and one or more of these steps can be omitted or replaced in different embodiments. In some embodiments, steps S3101, S3102, S3104, S3105, S3106, S3107, S3108, S3109, S3110 are optional, and one or more of these steps can be omitted or replaced in different embodiments. In some embodiments, steps S3101, S3102, S3103, S3105, S3106, S3107, S3108, S3109, S3110 are optional, and one or more of these steps can be omitted or replaced in different embodiments. In some embodiments, steps S3101, S3102, S3103, S3104, S3105, S3106, S3108, S3109, S3110 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0293] FIG. 3B is an interaction diagram of a model performance monitoring method according to embodiments of the present disclosure. The model performance monitoring method related by embodiments of the present disclosure can be applied to the communication system 100. As shown in FIG. 3B, the model performance monitoring method of embodiments of the present disclosure includes steps S3201 to S3210.

[0294] In step S3201, the fourth network element 1034 sends a fourth message to the first network element 1031.

[0295] In some embodiments, the first network element 1031 can receive the fourth message.

[0296] In some embodiments, the first network element 1031 can be a NWDAF. In some embodiments, the first network element 1031 can be a MTLF. In an example, the first network element 1031 can be a MTLF located in a NWDAF. It can be understood that the first network element 1031 can also be other network functions, which are not specifically limited in the present disclosure.

[0297] In some embodiments, the fourth network element 1034 can be a NWDAF. For example, the fourth network element 1034 can be another NWDAF different from the first network element 1031. In some embodiments, the first network element 1031 and the fourth network element 1034 can have the same function. It can be understood that the fourth network element 1034 can also be other network functions, which are not specifically limited in the present disclosure.

[0298] In some embodiments, the fourth message can be used to trigger the first network element 1031 to perform model performance monitoring.

[0299] In some embodiments, the model performance monitoring can be determining model performance of the first model. In some embodiments, the fourth message can be used to request determining model performance of the first model.

[0300] In some embodiments, the first model can be used to implement a location service. In an example, the first model can be used to implement positioning for one or more terminals. In some embodiments, the location service implemented based on the first model can comprise at least one of the following: determining a current location of a terminal, predicting a location of a terminal.

[0301] In some embodiments, the first model can comprise at least one of the following: an AI model, an ML model. In some embodiments, the first model can be used to implement AI / ML based positioning. For example, the AI model can be used to implement AI model based positioning. For example, the ML model can be used to implement ML based positioning.

[0302] In some embodiments, the name of the fourth message is not limited, which can be, for example, a model performance monitoring request message, a model performance monitoring indication message, etc.

[0303] In some embodiments, the fourth message can comprise at least one of the following: identification information of the first model, area information.

[0304] In some embodiments, the identification information of the first model can be used to identify the first model. Through the identification information, the fourth network element 1034 can indicate model performance monitoring on the first model associated with the identification information.

[0305] In some embodiments, the identification information of the first model can comprise an identifier.

[0306] In some embodiments, the identifier can be used to identify each first model. In an example, one first model can be uniquely identified by an identifier. Different first models can be identified by different identifiers.

[0307] In some embodiments, the identifier can be used to identify a model group consisting of one or more first models. In an example, one model group can be uniquely identified by an identifier. Different model groups can be identified by different identifiers.

[0308] In some embodiments, the area information can be used to indicate a first area in which the first model implements positioning.

[0309] In some embodiments, the first area can be an area for which the positioning based on the first model is targeted. In some embodiments, the positioning based on the first model can be based on measurements within the first area. For example, the object to be positioned can be located within the first area. For example, the terminal and / or the network equipment for performing the positioning can be located within the first area.

[0310] In some embodiments, the object to be positioned can be a UE.

[0311] In some embodiments, the terminal performing the positioning can be a UE. In some embodiments, the terminal performing the positioning can be a positioning reference unit (PRU).

[0312] In some embodiments, the network equipment performing the positioning can be an access network equipment.

[0313] In some embodiments, after receiving the fourth message, the first network element 1031 can determine, according to the fourth message, that the model performance monitoring is performed for the first model. In an example, the first network element 1031 can determine, according to the identification information and / or the area information in the fourth message, that the model performance monitoring is for the first model.

[0314] In some embodiments, step S3201 can not be performed. In some embodiments, the first network element 1031 can not receive the fourth message from the fourth network element 1034.

[0315] In some embodiments, the first network element 1031 can determine, by itself, that the model performance monitoring is performed for the first model. In an example, the first network element 1031 can determine, by itself, that the model performance monitoring is performed for all models on the first network element 1031, and the first model is included in the models. In an example, the first network element 1031 can determine, by itself, that the model performance monitoring is performed for the first model.

[0316] In step S3202, the first network element 1031 determines at least one terminal 101.

[0317] Optional implementation of step S3202 can refer to optional implementation of step S3102 in FIG. 3A and other associated parts in the embodiments involved in FIG. 3A, which will not be described here again.

[0318] In step S3203, the first network element 1031 determines whether the data collection for the model performance monitoring is authorized.

[0319] Optional implementation of step S3203 can refer to optional implementation of step S3103 in FIG. 3A and other associated parts in the embodiments involved in FIG. 3A, which will not be described here again.

[0320] In step S3204, the first network element 1031 sends a first message to the second network element 1032.

[0321] Optional implementation of step S3204 can be referred to optional implementation of step S3104 in FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0322] In step S3205, the second network element 1032 triggers positioning.

[0323] Optional implementation of step S3205 can be referred to optional implementation of step S3105 in FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0324] In step S3206, the second network element 1032 can acquire first data.

[0325] Optional implementation of step S3206 can be referred to optional implementation of step S3106 in FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0326] In step S3207, the second network element 1032 sends a second message to the first network element 1031.

[0327] Optional implementation of step S3207 can be referred to optional implementation of step S3107 in FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0328] In step S3208, the first network element 1031 performs model performance monitoring.

[0329] Optional implementation of step S3208 can be referred to optional implementation of step S3108 in FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0330] In step S3209, the first network element 1031 performs model training.

[0331] Optional implementation of step S3209 can be referred to optional implementation of step S3109 in FIG. 3A and other associated parts in embodiments involved in FIG. 3A, which will not be repeated here.

[0332] In step S3210, the first network element 1031 sends a fifth message to the fourth network element 1034.

[0333] In some embodiments, the fourth network element 1034 can receive the fifth message.

[0334] In some embodiments, the fifth message can be sent according to the model performance.

[0335] In some embodiments, the fifth message can be sent in response to the fourth message.

[0336] In some embodiments, the fifth message can be used to trigger the fourth network element 1034 to train the first model.

[0337] In some embodiments, the name of the fifth message is not limited, which can be, for example, a model performance monitoring response message, a model performance monitoring notification message, a training trigger message, etc.

[0338] In some embodiments, the fifth message can include second trigger information. The second trigger information can be used to trigger training of the first model.

[0339] In some embodiments, the second trigger information can indicate training of the first model. In an example, in a case where the model performance of the first model is lower than a threshold value, and / or in a case where the model performance of the first model is poor, the fifth message can include the second trigger information to indicate training of the first model.

[0340] In some embodiments, step S3210 can be performed in a case where the first network element 1031 receives the fourth message from the fourth network element 1034.

[0341] In some embodiments, both step S3209 and step S3210 can be performed in a case where the first network element 1031 receives the fourth message from the fourth network element 1034.

[0342] In some embodiments, the fifth message can be sent periodically or non-periodically. In an example, the first network element 1031 can send the fifth message once according to the received fourth message. For example, the fourth message can be a request message, and the fifth message can be a response message to the request message. For example, the fourth message can be a query message, and the fifth message can be a feedback message to the query message. In an example, the first network element 1031 can send the fifth message one or more times according to the received fourth message. For example, the fourth message can be a subscription message, and the fifth message can be a notification message to the subscription message. For example, the fifth message can be sent periodically. For example, the fifth message can be sent in a case where a preset condition (e.g., a threshold value) is met.

[0343] In some embodiments, the training of the first model can include retraining or retraining of the first model.

[0344] The model performance monitoring method of the embodiments of the present disclosure can be implemented through steps S3201 to S3210.

[0345] The model performance monitoring method related by the embodiments of the present disclosure can include at least one of steps S3201 to S3210. For example, step S3201 can be implemented as an independent embodiment. For example, step S3203 can be implemented as an independent embodiment. For example, the combination of step S3204 can be implemented as an independent embodiment. For example, step S3207 can be implemented as an independent embodiment. For example, the combination of steps S3201 and S3203 can be implemented as an independent embodiment. For example, the combination of steps S3201 and S3204 can be implemented as an independent embodiment. For example, the combination of steps S3203 and S3204 can be implemented as an independent embodiment. For example, the combination of steps S3204 and S3207 can be implemented as an independent embodiment. For example, the combination of steps S3203, S3204 and S3207 can be implemented as an independent embodiment. It should be noted that the possible independent embodiments composed of one or more of steps S3201 to S3210 are not limited to this.

[0346] In some embodiments, at least two of steps S3201 to S3210 can be exchanged in order or synchronously executed. For example, steps S3209 and S3210 can be exchanged in order or executed simultaneously.

[0347] In some embodiments, steps S3202, S3203, S3204, S3205, S3206, S3207, S3208, S3209, S3210 are optional, and one or more of these steps can be omitted or replaced in different embodiments. In some embodiments, steps S3201, S3202, S3204, S3205, S3206, S3207, S3208, S3209, S3210 are optional, and one or more of these steps can be omitted or replaced in different embodiments. In some embodiments, steps S3201, S3202, S3203, S3205, S3206, S3207, S3208, S3209, S3210 are optional, and one or more of these steps can be omitted or replaced in different embodiments. In some embodiments, steps S3201, S3202, S3203, S3204, S3205, S3206, S3208, S3209, S3210 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0348] In some embodiments, the names of information and the like are not limited to the names described in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "field", "symbol", "symbol", "codebook", "codeword", "codepoint", "bit", "data", "program", "chip", and the like can be replaced with each other.

[0349] In some embodiments, terms such as "radio", "wireless", "radio access network (RAN)", "access network (AN)", "RAN-based", and the like can be replaced with each other.

[0350] In some embodiments, "acquire", "obtain", "get", "receive", "transmit", "bidirectional transmission", "send and / or receive", and the like can be replaced with each other, and can be interpreted as receiving from another subject, acquiring from a protocol, acquiring from a higher layer, obtaining by processing oneself, implementing autonomously, and the like.

[0351] In some embodiments, terms such as "send", "transmit", "report", "issue", "transmit", "bidirectional transmission", "send and / or receive", and the like can be replaced with each other.

[0352] In some embodiments, terms such as "certain", "preset", "pre-set", "set", "indicated", "a certain", "arbitrary", "first", and the like can be replaced with each other, and "certain A", "preset A", "pre-set A", "set A", "indicated A", "a certain A", "arbitrary A", "first A" can be interpreted as A specified in advance in a protocol and the like, can be interpreted as A obtained by setting, configuring, or indicating, and the like, and can be interpreted as certain A, a certain A, arbitrary A, or first A, but are not limited thereto.

[0353] In some embodiments, the determining or judging can be performed by a value (0 or 1) represented by 1 bit, a true or false value (Boolean value) represented by true or false, or a comparison of numerical values (for example, a comparison with a predetermined value), but is not limited thereto.

[0354] FIG. 4 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. The present embodiment relates to a model performance monitoring method. The model performance monitoring method is performed by the first network element 1031. As shown in FIG. 4, the above method includes steps S401 to S408.

[0355] In step S401, a fourth message is acquired.

[0356] The optional implementation of step S401 can refer to the optional implementation of step S3101 in FIG. 3A, step S3201 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0357] In some embodiments, the first network element 1031 can receive the fourth message sent by the second network element 1032 or the fourth network element 1034, but is not limited thereto, and can also receive the fourth message sent by other subjects.

[0358] In some embodiments, the first network element 1031 can acquire the fourth message specified by a protocol.

[0359] In some embodiments, the first network element 1031 can acquire the fourth message from an upper layer.

[0360] In some embodiments, the first network element 1031 can process to obtain the fourth message.

[0361] In some embodiments, step S401 can be omitted, and the first network element 1031 autonomously implements the function indicated by the fourth message, or the above function is default or default.

[0362] In step S402, at least one terminal is determined.

[0363] The optional implementation of step S402 can refer to the optional implementation of step S3102 in FIG. 3A, step S3202 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0364] In step S403, it is determined whether data collection is authorized.

[0365] The optional implementation of step S403 can refer to the optional implementation of step S3103 in FIG. 3A, step S3203 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0366] In step S404, the first message is sent.

[0367] The optional implementation of step S404 can refer to the optional implementation of step S3104 in FIG. 3A, step S3204 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0368] In some embodiments, the first network element 1031 can send the first message to the second network element 1032, but is not limited thereto, and can also send the first message to other subjects.

[0369] In step S405, the second message is obtained.

[0370] The optional implementation of step S405 can refer to the optional implementation of step S3107 in FIG. 3A, step S3207 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0371] In some embodiments, the first network element 1031 can receive the second message sent by the second network element 1032, but is not limited thereto, and can also receive the second message sent by other subjects.

[0372] In step S406, the model performance monitoring is performed.

[0373] The optional implementation of step S406 can refer to the optional implementation of step S3108 in FIG. 3A, step S3208 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A, which will not be repeated here.

[0374] In some embodiments, the model performance monitoring can be performed based on the first data in the second message.

[0375] In step S407, the model training is performed.

[0376] The optional implementation of step S407 can refer to the optional implementation of step S3109 in FIG. 3A, step S3209 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0377] In step S408, the fifth message is sent.

[0378] The optional implementation of step S408 can refer to the optional implementation of step S3110 in FIG. 3A, the optional implementation of step S3210 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which are not described here again.

[0379] In some embodiments, the first network element 1031 can send the fifth message to the second network element 1032 or the fourth network element 1034, but is not limited thereto, and can send the fifth message to other subjects.

[0380] The model performance monitoring method involved in the embodiments of the present disclosure can include at least one of steps S401 to S408.

[0381] FIG. 5 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiments of the present disclosure relate to a model performance monitoring method. The model performance monitoring method is performed by the second network element 1032. As shown in FIG. 5, the above method includes steps S501 to S506.

[0382] In step S501, the fourth message is sent.

[0383] The optional implementation of step S501 can refer to the optional implementation of step S3101 in FIG. 3A, and other associated parts in the embodiments involved in FIG. 3A, which are not described here again.

[0384] In some embodiments, the second network element 1032 can send the fourth message to the first network element 1031, but is not limited thereto, and can send the fourth message to other subjects.

[0385] In step S502, the first message is acquired.

[0386] The optional implementation of step S502 can refer to the optional implementation of step S3104 in FIG. 3A, the optional implementation of step S3204 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which are not described here again.

[0387] In some embodiments, the second network element 1032 can receive the first message sent by the first network element 1031, but is not limited thereto, and can receive the first message sent by other subjects.

[0388] In step S503, positioning is triggered.

[0389] The optional implementation of step S503 can refer to the optional implementation of step S3105 in FIG. 3A, the optional implementation of step S3205 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which are not described here again.

[0390] In step S504, the first data is acquired.

[0391] The optional implementation of step S504 can refer to the optional implementation of step S3106 in FIG. 3A, the optional implementation of step S3206 in FIG. 3B, and other associated parts in the embodiments related to FIG. 3A and FIG. 3B, which are not described here again.

[0392] In some embodiments, the second network element 1032 can receive the first data sent by the terminal 101 and / or the access network device 102, but is not limited thereto, and can also receive the first data sent by other subjects.

[0393] In step S505, a second message is sent.

[0394] The optional implementation of step S505 can refer to the optional implementation of step S3107 in FIG. 3A, the optional implementation of step S3207 in FIG. 3B, and other associated parts in the embodiments related to FIG. 3A and FIG. 3B, which are not described here again.

[0395] In some embodiments, the second network element 1032 can send the second message to the first network element 1031, but is not limited thereto, and can also send the second message to other subjects.

[0396] In step S506, a fifth message is acquired.

[0397] The optional implementation of step S506 can refer to the optional implementation of step S3110 in FIG. 3A, and other associated parts in the embodiments related to FIG. 3A, which are not described here again.

[0398] In some embodiments, the second network element 1032 can receive the fifth message sent by the first network element 1031, but is not limited thereto, and can also receive the fifth message sent by other subjects.

[0399] The model performance monitoring method related to the embodiments of the present disclosure can include at least one of steps S501 to S506.

[0400] FIG. 6 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiments of the present disclosure relate to a model performance monitoring method. The model performance monitoring method is performed by a third network element 1033. As shown in FIG. 6, the above method includes steps S601 and S602.

[0401] In step S601, a sixth message is acquired.

[0402] The optional implementation of step S601 can refer to the optional implementation of step S3103 in FIG. 3A, the optional implementation of step S3203 in FIG. 3B, and other associated parts in the embodiments related to FIG. 3A and FIG. 3B, which are not described here again.

[0403] In some embodiments, the third network element 1033 can receive the sixth message sent by the first network element 1031, but is not limited thereto, and can also receive the sixth message sent by other subjects.

[0404] In step S602, a third message is sent.

[0405] Optional implementation of step S602 can refer to optional implementation of step S3103 in FIG. 3A, optional implementation of step S3203 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0406] In some embodiments, the third network element 1033 can send the third message to the first network element 1031, but is not limited thereto, and can also send the third message to other subjects.

[0407] The model performance monitoring method involved in the embodiments of the present disclosure can include at least one of steps S601 to S602.

[0408] FIG. 7 is a flow diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiments of the present disclosure relate to a model performance monitoring method. The model performance monitoring method is performed by a fourth network element 1034. As shown in FIG. 7, the above method includes steps S701 and S702.

[0409] In step S701, a fourth message is sent.

[0410] Optional implementation of step S701 can refer to optional implementation of step S3201 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3B, which will not be repeated here.

[0411] In some embodiments, the fourth network element 1034 can send the fourth message to the first network element 1031, but is not limited thereto, and can also send the fourth message to other subjects.

[0412] In step S702, a fifth message is obtained.

[0413] Optional implementation of step S702 can refer to optional implementation of step S3210 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3B, which will not be repeated here.

[0414] In some embodiments, the fourth network element 1034 can receive the fifth message sent by the first network element 1031, but is not limited thereto, and can also receive the fifth message sent by other subjects.

[0415] The model performance monitoring method involved in the embodiments of the present disclosure can include at least one of steps S701 to S702.

[0416] FIG. 8A is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiment of the present disclosure relates to a model performance monitoring method. As shown in FIG. 8A, the above method comprises steps S8101 to S8102.

[0417] In step S8101, the first network element 1031 sends a first message to the second network element 1032.

[0418] The optional implementation of step S8101 can refer to the optional implementation of step S3104 in FIG. 3A, step S3204 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0419] In step S8102, the second network element 1032 sends a second message to the first network element 1031.

[0420] The optional implementation of step S8102 can refer to the optional implementation of step S3107 in FIG. 3A, step S3207 in FIG. 3B, and other associated parts in the embodiments involved in FIG. 3A and FIG. 3B, which will not be repeated here.

[0421] FIG. 8B is an interaction schematic diagram of a model performance monitoring method according to an embodiment of the present disclosure. The embodiment of the present disclosure relates to a model performance monitoring method. As shown in FIG. 8B, the above method comprises step S8201.

[0422] In step S8201, the third network element 1033 sends a third message to the first network element 1031.

[0423] The optional implementation of step S8201 can refer to the optional implementation of step S3107 in FIG. 3A, and other associated parts in the embodiments involved in FIG. 3A, which will not be repeated here.

[0424] In the following, the embodiments of the present disclosure are exemplarily described through specific embodiments.

[0425] In some embodiments, the present disclosure provides an independent procedure for LMF-based model performance monitoring. That is, the execution of LMF-based model performance monitoring can be independent of the execution of LMF-based model inference. In this procedure, the operations on the NWDAF (i.e., the first network element) are defined. In an example, the NWDAF receives a trigger for model performance monitoring from the LMF (i.e., the second network element) or an internal NF or another NWDAF (i.e., the fourth network element). In an example, the NWDAF determines the PRU / UE (i.e., the terminal, also referred to as the terminal device) that supports model performance monitoring. In an example, the NWDAF triggers to initiate AI-based positioning for each PRU / UE, thereby collecting measurement data from the UE / gNB and calculating the PRU / UE position (i.e., the calculated position). In an example, the NWDAF collects the known position (i.e., the actual position) of the PRU / UE. In an example, the NWDAF evaluates the model performance by using the collected / calculated position data to determine whether to change the positioning method or retrain the AI model (i.e., the first model).

[0426] FIG. 9 is an interaction schematic diagram of an exemplary implementation of a model performance monitoring method according to embodiments of the present disclosure. As shown in FIG. 9, the model performance monitoring method can include steps S901 to S908.

[0427] In step S901, the NWDAF triggers model training monitoring, or receives a trigger for model training monitoring. In an example, the trigger includes a model ID, and area information.

[0428] In some embodiments, the model performance monitoring can be limited to the model applied in a specific area.

[0429] In step S902, the NWDAF determines the UE / PRU that supports model performance monitoring, and checks with the UDM (i.e., the third network element) whether the data collection for model performance monitoring is authorized.

[0430] In step S903, the NWDAF requests the LMF for data collection for model performance monitoring for each UE / PRU. In an example, the request can include a UE ID.

[0431] In step S904, the LMF triggers AI-based positioning for the UE.

[0432] In step S905, step S905a and / or step S905b can be performed.

[0433] In step S905a, the LMF starts collecting measurement data from the UE / gNB, and obtains a calculated position based on the measurement data.

[0434] In step S905b, meanwhile, the LMF also starts collecting known positions of the UE.

[0435] In step S906, the LMF sends the measurement data / computed position of the UE, and the known positions to the NWDAF.

[0436] In step S907, based on the data collected from the LMF, the NWDAF evaluates the model performance to obtain a result. The result is used to indicate that the model performance is good or bad, and meets a preset condition (i.e., a threshold).

[0437] In step S908, based on the result, the NWDAF can trigger retraining of the model, or can send the result to other network functions (e.g., the LMF).

[0438] In the embodiments of the present disclosure, part or all of the steps, and optional implementation manners thereof, can be combined with part or all of the steps in other embodiments, or can be combined with optional implementation manners of other embodiments.

[0439] The embodiments of the present disclosure also provide a communication apparatus for implementing any of the above methods. For example, the embodiments of the present disclosure provide a communication apparatus comprising units or modules for implementing the steps performed by the first network element in any of the above methods. For example, the embodiments of the present disclosure provide a communication apparatus comprising units or modules for implementing the steps performed by the second network element in any of the above methods. For example, the embodiments of the present disclosure provide a communication apparatus comprising units or modules for implementing the steps performed by the third network element in any of the above methods. For example, the embodiments of the present disclosure provide a communication apparatus comprising units or modules for implementing the steps performed by the fourth network element in any of the above methods.

[0440] It should be understood that the division of each unit or module in the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated in actual implementation. In addition, the units or modules in the apparatus can be implemented in the form of processor calling software: for example, the apparatus includes a processor connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit or module of the above apparatus, wherein the processor is, for example, a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the apparatus or a memory outside the apparatus. Alternatively, the units or modules in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units or modules can be realized by the design of hardware circuit. The above hardware circuit can be understood as one or more processors; for example, in one implementation, the above hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the above units or modules are realized by the design of the logical relationship of elements in the circuit; for example, in another implementation, the above hardware circuit is a programmable logic device (PLD), and a field programmable gate array (FPGA) is taken as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the above units or modules. All units or modules of the above apparatus can be implemented in the form of processor calling software, or all units or modules can be implemented in the form of hardware circuit, or part of the units or modules are implemented in the form of processor calling software, and the remaining part is implemented in the form of hardware circuit.

[0441] In the embodiments of the present disclosure, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit, a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), and the like. In another implementation, the processor can implement certain functions through a logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable. For example, the processor is a hardware circuit implemented by a dedicated integrated circuit or a programmable logic device, such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the hardware circuit configuration. It can be understood that the processor loads instructions to implement the functions of the above part or all units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), and the like.

[0442] FIG. 10 is a structural schematic diagram of a model performance monitoring apparatus provided by an embodiment of the present disclosure. As shown in FIG. 10, the model performance monitoring apparatus 1000 can include at least one of a transceiving module 1001 and a processing module 1002.

[0443] In some embodiments, the model performance monitoring apparatus 1000 can be the first network element 1031. In some embodiments, the transceiver module 1001 can be configured to send a first message to a second network element, where the first message is used to request performing data collection for at least one terminal device; and receive a second message sent by the second network element, where the second message comprises first data; and the first data is data collected by the second network element for the at least one terminal device, and the first data is used to implement model performance monitoring on a first model. Optionally, the transceiver module 1001 can be configured to perform at least one of the communication steps (e.g., steps S3101, S3102, S3103, S3104, S3107, S3110, S3201, S3202, S3203, S3204, S3207, S3210, but not limited to this) of sending and / or receiving performed by the first network element 1031 in any of the above methods, which will not be described here. Optionally, the processing module 1002 can be configured to perform at least one of the steps other than the communication steps of sending and / or receiving performed by the first network element 1031 in any of the above methods (e.g., steps S3108, S3109, S3208, S3209, but not limited to this), which will not be described here.

[0444] In some embodiments, the model performance monitoring apparatus 1000 can be the second network element 1032. In some embodiments, the transceiver module 1001 can be configured to receive a first message sent by a first network element, where the first message is used to request performing data collection for at least one terminal device; and send a second message to the first network element, where the second message comprises first data; and the first data is data collected by the second network element for the at least one terminal device, and the first data is used to implement model performance monitoring on a first model. Optionally, the transceiver module 1001 can be configured to perform at least one of the communication steps (e.g., steps S3101, S3104, S3106, S3107, S3108, S3204, S3206, S3207, but not limited to this) of sending and / or receiving performed by the second network element 1032 in any of the above methods, which will not be described here. Optionally, the processing module 1002 can be configured to perform at least one of the steps other than the communication steps of sending and / or receiving performed by the second network element 1032 in any of the above methods (e.g., steps S3105, S3205, but not limited to this), which will not be described here.

[0445] In some embodiments, the model performance monitoring apparatus 1000 can be the third network element 1033. In some embodiments, the transceiver module 1001 can be configured to send a third message to the first network element, where the third message is used to determine whether data collection for at least one terminal device is authorized; where the data collection for at least one device is used to obtain first data, and the first data is used to implement model performance monitoring on the first model. Optionally, the transceiver module 1001 can be configured to perform at least one of the communication steps (for example, steps S3103, S3203, but not limited to this) of sending and / or receiving performed by the third network element 1033 in any of the above methods, which will not be described here.

[0446] In some embodiments, the model performance monitoring apparatus 1000 can be the fourth network element 1034. In some embodiments, the transceiver module 1001 can be configured to send a fourth message to the first network element, where the fourth message is used to trigger the first network element to perform model performance monitoring. Optionally, the transceiver module 1001 can be configured to perform at least one of the communication steps (for example, steps S3201, S3210, but not limited to this) of sending and / or receiving performed by the fourth network element 1034 in any of the above methods, which will not be described here.

[0447] In some embodiments, the transceiver module can include a sending module and / or a receiving module. The sending module and the receiving module can be separate, or integrated together. Optionally, the transceiver module can be mutually replaced with the transceiver.

[0448] In some embodiments, the processing module can be one module, or can include multiple sub-modules. Optionally, the multiple sub-modules perform all or part of the steps required to be performed by the processing module. Optionally, the processing module can be mutually replaced with the processor.

[0449] FIG. 11A is a structural schematic diagram of a communication device provided by an embodiment of the present disclosure. The communication device 11100 can be a network device (for example, an access network device, a core network device, etc.), a terminal (for example, a user equipment, etc.), a chip, a chip system, or a processor supporting the network device to implement any of the above methods, or a chip, a chip system, or a processor supporting the terminal to implement any of the above methods. The communication device 11100 can be used to implement the methods described in the above method embodiments, and specific reference can be made to the descriptions in the above method embodiments.

[0450] As shown in FIG. 11A, the communication device 11100 includes one or more processors 11101. The processor 11101 can be a general processor or a special-purpose processor, etc., such as a baseband processor or a central processing unit. The baseband processor can be configured to process communication protocols and communication data, and the central processing unit can be configured to control a communication apparatus (e.g., a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute programs, and process data of the programs. Optionally, the communication device 11100 is configured to perform any of the above methods. Optionally, the one or more processors 11101 are configured to invoke instructions to cause the communication device 11100 to perform any of the above methods.

[0451] In some embodiments, the communication device 11100 further includes one or more transceivers 11102. When the communication device 11100 includes the one or more transceivers 11102, the transceiver 11102 performs at least one of the communication steps (e.g., steps S3101, S3102, S3103, S3104, S3106, S3107, S3110, S3201, S3202, S3203, S3204, S3206, S3207, S3210, but not limited to) in the above methods, and the processor 11101 performs at least one of the other steps (e.g., steps S3105, S3108, S3109, S3205, S3208, S3209, but not limited to). In optional embodiments, the transceiver can include a receiver and / or a transmitter, which can be separate or integrated together. Optionally, the terms transceiver, transceiving unit, transceiver, transceiving circuit, interface circuit, interface, etc. can be replaced by each other, and the terms transmitter, transmitting unit, transmitter, transmitting circuit, etc. can be replaced by each other, and the terms receiver, receiving unit, receiver, receiving circuit, etc. can be replaced by each other.

[0452] In some embodiments, the communication device 11100 further includes one or more memories 11103 for storing data. Optionally, all or part of the memory 11103 can also be outside the communication device 11100. In optional embodiments, the communication device 11100 can include one or more interface circuits 11104. Optionally, the interface circuit 11104 is connected to the memory 11103, and the interface circuit 11104 can be configured to receive data from the memory 11103 or other devices, and can be configured to send data to the memory 11103 or other devices. For example, the interface circuit 11104 can read data stored in the memory 11103 and send the data to the processor 11101.

[0453] The communication device 11100 described in the above embodiments can be a network device or a terminal, but the scope of the communication device 11100 described in the present disclosure is not limited thereto, and the structure of the communication device 11100 can not be limited by FIG. 11A. The communication device can be a standalone device or can be part of a larger device. For example, the communication device can be: 1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally also include storage components for storing data, programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, a smart terminal device, a cellular phone, a wireless device, a handset, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, and the like; (6) other devices, and the like.

[0454] FIG. 11B is a structural schematic diagram of a chip according to an embodiment of the present disclosure. For the case where the communication device 11100 can be a chip or a chip system, the structural schematic diagram of the chip 11200 shown in FIG. 11B can be referred to, but is not limited thereto.

[0455] The chip 11200 includes one or more processors 11201. The chip 11200 is configured to perform any of the above methods.

[0456] In some embodiments, the chip 11200 further includes one or more interface circuits 11202. Optionally, the terms interface circuit, interface, transceiver pin, and the like can be replaced with each other. In some embodiments, the chip 11200 further includes one or more memories 11203 for storing data. Optionally, all or part of the memory 11203 can be outside the chip 11200. Optionally, the interface circuit 11202 is connected to the memory 11203, and the interface circuit 11202 can be configured to receive data from the memory 11203 or other devices, and the interface circuit 11202 can be configured to send data to the memory 11203 or other devices. For example, the interface circuit 11202 can read data stored in the memory 11203 and send the data to the processor 11201.

[0457] In some embodiments, the interface circuit 11202 performs at least one of the communication steps (for example, steps S3101, S3102, S3103, S3104, S3106, S3107, S3110, S3201, S3202, S3203, S3204, S3206, S3207, S3210, but not limited to) of sending and / or receiving in the above method. The interface circuit 11202 performing the communication steps such as sending and / or receiving in the above method means that the interface circuit 11202 performs data interaction between the processor 11201, the chip 11200, the memory 11203 or the transceiver device. In some embodiments, the processor 11201 performs at least one of the other steps (for example, steps S3105, S3108, S3109, S3205, S3208, S3209, but not limited to).

[0458] The modules and / or devices described in each of the embodiments of the virtual device, the physical device, the chip, etc. can be combined or separated as the case may be. Alternatively, part or all of the steps can also be executed by a plurality of modules and / or devices in cooperation, which is not limited here.

[0459] The embodiments of the present disclosure also propose a storage medium, and the storage medium stores instructions. When the instructions run on the communication device 11100, the communication device 11100 executes any one of the above methods. Alternatively, the storage medium is an electronic storage medium. Alternatively, the storage medium is a computer readable storage medium, but is not limited to this, and it can also be a storage medium readable by other devices. Alternatively, the storage medium can be a non-transitory storage medium, but is not limited to this, and it can also be a transitory storage medium.

[0460] The embodiments of the present disclosure also propose a program product, and the program product is executed by the communication device 11100, so that the communication device 11100 executes any one of the above methods. Alternatively, the program product is a computer program product.

[0461] The embodiments of the present disclosure also propose a computer program, which makes the computer execute any one of the above methods when it runs on the computer.

[0462] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present disclosure is intended to cover any and all variations of the present application comprising adaptations, modifications, equivalents, and alternatives falling within the scope of the present application. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0463] It should be understood that the application is not limited to the precise construction which has been described above and which shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should be limited only by the appended claims.

Claims

A method of model performance monitoring, performed by a first network element, wherein, The method comprises: sending a first message to a second network element, the first message being used for requesting to perform data collection for at least one terminal device; receiving a second message sent by the second network element, the second message comprising first data; wherein the first data is data collected by the second network element for the at least one terminal device, and the first data is used for implementing model performance monitoring of a first model. The method of claim 1, wherein, The first message comprises identification information of the at least one terminal device. The method according to claim 1 or 2, wherein The first model comprises at least one of: an artificial intelligence (AI) model; a machine learning (ML) model. The method of any one of claims 1 to 3, wherein, The first model is used for implementing AI / ML-based positioning. The method of any one of claims 1 to 4, wherein, The first model is applied to a first area. The method of any one of claims 1 to 5, wherein, The first data comprises at least one of: measurement data; a calculated position obtained based on the measurement data; an actual position. The method of any one of claims 1 to 6, wherein, The method further comprises: determining the at least one terminal device, wherein the at least one terminal device supports data collection. The method of any one of claims 1 to 7, wherein, The method further comprises: receiving a third message sent by a third network element, the third message being used for determining whether data collection for the at least one terminal device is authorized. The method of any one of claims 1 to 8, wherein, The method further comprises: receiving a fourth message sent by the second network element, the fourth message being used for triggering the first network element to perform the model performance monitoring. The method of claim 9, wherein, The fourth message comprises at least one of: identification information of the first model; area information used for indicating a first area in which the first model implements positioning. The method of any one of claims 1 to 10, wherein, The method further comprises: determining model performance of the first model based on the first data; performing a first operation related to the first model according to the model performance. The method of claim 11, wherein, The first operation comprises at least one of: sending a fifth message to the second network element; sending the fifth message to a fourth network element, wherein the fourth network element has the same function as the first network element; training the first model. The method of claim 12, wherein, The fifth message is used for at least one of: indicating model performance of the first model; triggering a change of a positioning method; triggering training of the first model. A method of model performance monitoring, performed by a second network element, wherein, The method comprises: receiving a first message sent by a first network element, the first message being used for requesting to perform data collection for at least one terminal device; sending a second message to the first network element, the second message comprising first data; wherein the first data is data collected by the second network element for the at least one terminal device, and the first data is used for implementing model performance monitoring of a first model. The method of claim 14, wherein, The first message comprises identification information of the at least one terminal device. The method according to claim 14 or 15, wherein The first model comprises at least one of: an artificial intelligence (AI) model; a machine learning (ML) model. The method of any one of claims 14 to 16, wherein, The first model is used for implementing AI / ML-based positioning. The method of any one of claims 14 to 17, wherein, The first model is applied to a first area. The method of any one of claims 14 to 18, wherein, The first data comprises at least one of: measurement data; a calculated position obtained based on the measurement data; an actual position. The method of any one of claims 14 to 19, wherein, The method further comprises: obtaining the first data according to the first message. The method of claim 20, wherein, The obtaining the first data according to the first message comprises at least one of: obtaining measurement data of the at least one terminal device; determine a calculated position of the at least one terminal device based on the measurement data; obtain an actual position of the at least one terminal device. The method of any one of claims 14 to 21, wherein, The method further includes: sending, to the first network element, a fourth message, the fourth message being used to trigger the first network element to perform the model performance monitoring. The method of claim 22, wherein, The fourth message includes at least one of: identification information of the first model; region information used to indicate a first region in which the first model implements positioning. The method of any one of claims 14 to 23, wherein, The method further includes: receiving, from the first network element, a fifth message sent according to a model performance of the first model, the fifth message being used to at least one of: indicate the model performance of the first model; trigger the second network element to change a positioning method; trigger the second network element to train the first model. A method of model performance monitoring, performed by a third network element, wherein, The method includes: sending, to a first network element, a third message, the third message being used to determine whether data collection for at least one terminal device is authorized; wherein the data collection for the at least one device is used to obtain first data, the first data being used to implement model performance monitoring on a first model. The method of claim 25, wherein, The third message includes at least one of: identification information of the first model; region information used to indicate a first region in which the first model implements positioning. The method of claim 25 or 26, wherein, The first model includes at least one of: an artificial intelligence (AI) model; a machine learning (ML) model. The method of any one of claims 25 to 27, wherein, The first model is used to implement AI / ML-based positioning. The method of any one of claims 25 to 28, wherein, The first model is applied to a first region. The method of any one of claims 25 to 29, wherein, The first data includes at least one of: measurement data; a calculated position obtained based on the measurement data; an actual position. The method of any one of claims 25-30, wherein The method further includes: receiving, from the first network element, a sixth message, the sixth message being used to request whether the data collection for at least one terminal device is authorized. A model performance monitoring apparatus is arranged in a first network element, wherein, The apparatus includes: a transceiver module configured to: send, to a second network element, a first message, the first message being used to request data collection for at least one terminal device; receive, from the second network element, a second message, the second message including first data; wherein the first data is data collected by the second network element for the at least one terminal device, and the first data is used to implement model performance monitoring on a first model. A model performance monitoring apparatus is arranged at a second network element, wherein The apparatus includes: a transceiver module configured to: receive, from a first network element, a first message, the first message being used to request data collection for at least one terminal device; send, to the first network element, a second message, the second message including first data; wherein the first data is data collected by the second network element for the at least one terminal device, and the first data is used to implement model performance monitoring on a first model. A model performance monitoring apparatus is arranged at a third network element, wherein, The apparatus includes: a transceiver module configured to send, to a first network element, a third message, the third message being used to determine whether data collection for at least one terminal device is authorized; wherein the data collection for the at least one device is used to obtain first data, the first data being used to implement model performance monitoring on a first model. A communication device includes: one or more processors; a memory having instructions stored thereon; wherein the instructions, when executed on the communication device, cause the communication device to implement at least one of: the model performance monitoring method of any of claims 1-13; the model performance monitoring method of any of claims 14-24; the model performance monitoring method of any of claims 25-31. A communication system comprising: a first network element configured to implement the model performance monitoring method of any of claims 1-13; a second network element configured to implement the model performance monitoring method of any of claims 14-24; a third network element configured to implement the model performance monitoring method of any of claims 25-31. A storage medium storing instructions, wherein, when executed on a communication device, cause the communication device to implement at least one of: the model performance monitoring method of any of claims 1-13; the model performance monitoring method of any of claims 14-24; the model performance monitoring method of any of claims 25-31. A computer program product comprising instructions, wherein, when executed on a communication device, cause the communication device to implement at least one of: the model performance monitoring method of any of claims 1-13; the model performance monitoring method of any of claims 14-24; the model performance monitoring method of any of claims 25-31.

Images