Method and apparatus for acquiring performance information of positioning model, and network side device

Abstract

The present application relates to the technical field of wireless communications, and discloses a method and apparatus for acquiring performance information of a positioning model, and a network side device. The method for acquiring performance information of a positioning model of the embodiments of the present application comprises: a first network element sends a positioning message to at least one first access network device, wherein the positioning message is used for requesting reporting of one of the following: AI measurement information generated by reasoning by means of a target positioning model, AI measurement information and real measurement information, and difference information between AI measurement information and real measurement information; the first network element receives a positioning result message sent by the at least one first access network device, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information comprises one of the following: AI measurement information, AI measurement information and real measurement information, and difference information between AI measurement information and real measurement information; on the basis of the target positioning measurement information, the first network element acquires first positioning performance monitoring information of the target positioning model.

Description

Method, device and network-side equipment for obtaining positioning model performance information

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to a Chinese patent application filed with the Patent Office of China on November 22, 2023, with application number 202311567614.3 and invention name “Method, device and network-side equipment for obtaining positioning model performance information”. The entire contents of the Chinese patent application are incorporated herein by reference. Technical Field

[0003] The present application belongs to the field of wireless communication technology, and specifically relates to a method, apparatus, and network-side equipment for obtaining positioning model performance information. Background Art

[0004] Artificial Intelligence (AI) positioning aims to use AI positioning models to predict measurement information to provide a target location estimate, which can save a lot of measurement resources and reduce computing power.

[0005] In related technologies, an AI positioning model can be set in an access network device for model inference to obtain AI measurement information, and the AI ​​measurement information is reported to a positioning control function network element, such as a location management function (LMF). The positioning control function network element can perform position estimation based on the received AI measurement information.

[0006] However, in actual applications, the performance of the AI ​​positioning model is unknown, which makes it impossible to determine the accuracy and precision of the estimated results obtained by using the AI ​​measurement information inferred by the AI ​​positioning model for position estimation, and thus it is impossible to know whether the obtained estimation results are accurate. Summary of the Invention

[0007] The embodiments of the present application provide a method, apparatus, and network-side device for obtaining positioning model performance information, which can solve the problem of being unable to determine the accuracy and precision of the estimation results obtained by using AI measurement information inferred by the AI ​​positioning model for position estimation due to the unknown performance of the AI ​​positioning model.

[0008] In a first aspect, a method for obtaining positioning model performance information is provided, including: a first network element sends a positioning message to at least one first access network device, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; the first network element receives a positioning result message sent by the at least one first access network device, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; the first network element obtains first positioning performance monitoring information of the target positioning model based on the target positioning measurement information.

[0009] In a second aspect, a method for sending positioning monitoring information of a positioning model is provided, including: a first access network device receives a positioning message sent by a first network element, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; the first access network device obtains real measurement information and AI measurement information inferred using the target positioning model; the first access network device sends a positioning result message to the first network element based on the real measurement information and the AI ​​measurement information, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information.

[0010] In a third aspect, a device for obtaining positioning model performance information is provided, including: a first sending module, used to send a positioning message to at least one first access point and first access network device, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; a first receiving module, used to receive a positioning result message sent by the at least one first access network device, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; a first acquisition module, used to obtain first positioning performance monitoring information of the target positioning model based on the target positioning measurement information.

[0011] In a fourth aspect, a positioning monitoring information sending device for a positioning model is provided, including: a second receiving module, used to receive a positioning message sent by a first network element, wherein the positioning message is used to request a report of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; a second acquisition module, used to obtain real measurement information and AI measurement information inferred using the target positioning model; a second sending module, used to send a positioning result message to the first network element based on the real measurement information and the AI ​​measurement information, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information.

[0012] In the fifth aspect, a network side device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0013] In the sixth aspect, a network side device is provided, comprising a processor and a communication interface, wherein the processor is used to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect, and the communication interface is used to couple with the processor.

[0014] In the seventh aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0015] In an eighth aspect, a communication system is provided, comprising: a first network element and an access network device, wherein the first network element can be used to execute the steps of the method described in the first aspect, and the access network device can be used to execute the steps of the method described in the second aspect.

[0016] In the ninth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect, or to implement the method described in the second aspect.

[0017] In the tenth aspect, a computer program / program product is provided, which is stored in a storage medium and is executed by at least one processor to implement the method as described in the first aspect, or to implement the method as described in the second aspect.

[0018] In an embodiment of the present application, a first network element sends a positioning message to at least one first access network device, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and actual measurement information, or difference information between the AI ​​measurement information and the actual measurement information; the first network element receives a positioning result message sent by the at least one first access network device, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and actual measurement information, or difference information between the AI ​​measurement information and the actual measurement information; and the first network element obtains first positioning performance monitoring information of the target positioning model based on the target positioning measurement information. This enables the first network element to monitor the positioning performance of the target positioning model based on the target positioning measurement information reported by the first access network device, thereby resolving the problem of being unable to determine the accuracy and precision of the estimated result obtained by using the AI ​​measurement information inferred by the AI ​​positioning model due to the unknown performance of the AI ​​positioning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 shows a block diagram of a wireless communication system to which embodiments of the present application may be applied;

[0020] FIG2 shows a schematic diagram of a layout of an AI positioning model in an embodiment of the present application;

[0021] FIG3 shows a flow chart of a method for obtaining positioning model performance information according to an embodiment of the present application;

[0022] FIG4 shows a schematic diagram of a process for sending positioning monitoring information of a positioning model provided in an embodiment of the present application;

[0023] FIG5 is a schematic diagram showing another flow chart of a method for obtaining positioning model performance information provided in an embodiment of the present application;

[0024] FIG6 shows another flow chart of a method for obtaining positioning model performance information according to an embodiment of the present application;

[0025] FIG7 shows a schematic structural diagram of a device for acquiring positioning model performance information provided in an embodiment of the present application;

[0026] FIG8 shows a schematic structural diagram of a positioning monitoring information device of a positioning model provided in an embodiment of the present application;

[0027] FIG9 shows a schematic structural diagram of a communication device provided in an embodiment of the present application;

[0028] FIG10 is a schematic diagram showing the hardware structure of a network-side device provided in an embodiment of the present application;

[0029] FIG11 shows a schematic diagram of the hardware structure of another network-side device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0031] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0032] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0033] It is worth noting that the technology described in the embodiments of the present application is not limited to the Long Term Evolution (LTE) / LTE-Advanced (LTE-A) system, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA) or other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the described technology can be used for the systems and radio technologies mentioned above, as well as for other systems and radio technologies. The following description describes a New Radio (NR) system for example purposes, and NR terminology is used in most of the following description, but these technologies can also be applied to systems other than NR systems, such as 6th Generation (6G) communication systems.

[0034] FIG1 is a block diagram of a wireless communication system applicable to an embodiment of the present application. The wireless communication system includes a terminal 11 and a network-side device 12. The terminal 11 may be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a notebook computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile internet device (MID), an augmented reality (AR), a virtual reality (VR) device, a robot, a wearable device (Wearable Device), an aircraft (Flight Vehicle), a vehicle-mounted device (VUE), a ship-mounted device, a pedestrian user equipment (PUE), a smart home (home appliances with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), a game console, a personal computer (PC), an ATM, or a self-service machine, or other terminal-side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle-mounted device can also be called a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application. The network side device 12 may include an access network device or a core network device, wherein the access network device may also be called a radio access network (Radio Access Network, RAN) device, a radio access network function or a radio access network unit. The access network device may include a base station, a wireless local area network (WLAN) access point (AS) or a wireless fidelity (WiFi) node, etc.Among them, the base station can be referred to as Node B (NB), Evolved Node B (eNB), the next generation Node B (gNB), New Radio Node B (NR Node B), access point, Relay Base Station (RBS), Serving Base Station (SBS), Base Transceiver Station (BTS), radio base station, radio transceiver, Basic Service Set (BSS), Extended Service Set (ESS), Home Node B (HNB), Home Evolved Node B (home evolved Node B), Transmission Reception Point (TRP) or other appropriate terms in the field. As long as the same technical effect is achieved, the base station is not limited to specific technical vocabulary. It should be noted that in the embodiment of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited.

[0035] The core network equipment may include but is not limited to at least one of the following: core network node, core network function, mobility management entity (MME), access mobility management function (AMF), session management function (SMF), user plane function (UPF), policy control function (PCF), policy and charging rules function unit (PCRF), edge application service discovery function (EASDF), unified data management (UDM), unified data repository (UDR), home user server (HSS), centralized network configuration (CNC), network storage function (NRF), network exposure function (NEF), local NEF (L-NEF), binding support function (BSF), application function ( Function, AF), Location Management Function (LMF), Gateway Mobile Location Centre (GMLC), Network Data Analytics Function (NWDAF), etc. It should be noted that in the embodiment of the present application, only the core network equipment in the NR system is taken as an example to introduce, and the specific type of the core network equipment is not limited.

[0036] In related technologies, as shown in FIG2 , an AI positioning model is set up in a base station. The UE sends measurement feedback (e.g., a sounding reference signal (SRS)) to the base station. The base station uses the AI ​​positioning model to calculate AI measurement information, and then sends the AI ​​measurement information to the LMF to implement position calculation.

[0037] However, the model accuracy of the AI ​​positioning model in actual use may not be high, and the positioning accuracy may also be poor. The performance of the AI ​​positioning model is unknown, which makes it impossible to determine the accuracy and precision of the estimated results obtained by using the AI ​​measurement information inferred by the AI ​​positioning model for position estimation, and thus it is impossible to know whether the obtained estimation results are accurate.

[0038] To address this issue, an embodiment of the present application provides a method, apparatus, and network-side device for obtaining positioning model performance information. A triggerer can trigger the network to monitor information such as positioning accuracy and model accuracy of a specific AI model. If the reporting conditions are met, the AI ​​model performance result information can be reported to the triggerer.

[0039] The technical solutions provided by the embodiments of the present application are described in detail below through some embodiments and their application scenarios in conjunction with the accompanying drawings.

[0040] Figure 3 illustrates a flow chart of a method for obtaining positioning model performance information in an embodiment of the present application. Method 300 can be executed by a first network element. In other words, the method can be executed by software or hardware installed on a communication device. In this embodiment of the present application, the first network element can be a positioning control function network element, such as a Location Management Function (LMF).

[0041] As shown in FIG3 , the method may include the following steps.

[0042] S310: A first network element sends a positioning message to at least one first access network device.

[0043] In this embodiment of the present application, the positioning message is used to request reporting of one of the following:

[0044] The target positioning model is used for reasoning to generate AI measurement information; the first network element may request at least one first access network device to report the AI ​​measurement information generated by reasoning using the target positioning model through a positioning message. After receiving the positioning message, the first access network device may use the target positioning model for reasoning to obtain the AI ​​measurement information.

[0045] The AI ​​measurement information and the real measurement information; the first network element can request at least one first access network device to report the AI ​​measurement information and the real measurement information generated by reasoning using the target positioning model through a positioning message. After receiving the positioning message, the first access network device can use the target positioning model for reasoning to obtain the AI ​​measurement information, and adopt a non-AI method, for example, by measuring the positioning reference signal to obtain the real measurement information.

[0046] The difference information between the AI ​​measurement information and the actual measurement information. The first network element may request at least one first access network device to report, through a positioning message, the difference information between the AI ​​measurement information generated by reasoning using the target positioning model and the actual measurement information. After receiving the positioning message, the first access network device may use the target positioning model for reasoning to obtain the AI ​​measurement information, and adopt a non-AI method, for example, by measuring the positioning reference signal to obtain the actual measurement information, and then obtain the difference information between the AI ​​measurement information and the actual measurement information through comparison.

[0047] Optionally, the positioning message may carry at least one of the following:

[0048] Identification information of the target positioning model; the identification information is used to indicate the target positioning model to be monitored, wherein the target positioning model to be monitored can be one or more, which is not limited in the specific embodiments of the present application.

[0049] The first indication information is used to instruct the first access network device to use the AI ​​measurement information inferred by the target positioning model for positioning. The first indication information may also instruct the first access network device to report the AI ​​measurement information inferred by the first access network device using the target positioning model. For example, the first indication information may be an AI positioning indication (AI positioning ind).

[0050] The second indication information is used to indicate that the first network element will perform performance monitoring on the target positioning model. The second indication information indicates that the first network element will perform performance monitoring on the target positioning model. Based on the second indication information, the first access network device may determine that it is necessary to report the AI ​​measurement information, the AI ​​measurement information and the actual measurement information, and the difference between the AI ​​measurement information and the actual measurement information. For example, the second indication information may be a monitoring indication (Monitoring ind).

[0051] In an optional implementation, the first network element may be triggered by the target device to at least one first access network device. Therefore, in this optional implementation, before S310, the method may further include: the first network element receives a positioning performance monitoring message sent by the target device, wherein the positioning performance monitoring message is used to request performance measurement of the target positioning model, and the positioning performance monitoring message carries identification information of the target positioning model; the target device includes one of the following: the first access network device, the second access network device, and the second network element. That is, in this optional implementation, the first network element may be triggered by the first access network device, the second access network device, or the second network element to perform performance monitoring of the target positioning model.

[0052] For example, the first access network device can train and generate the above-mentioned target positioning model by itself, and then request the first network element to monitor the performance of the target positioning model; or the second access network device can train and generate the above-mentioned target positioning model by itself, and send it to the first access network device for use, and the second access network device can request the first network element to monitor the performance of the target positioning model; or the second network element trains and generates the above-mentioned target positioning model, and sends it to the first access network device for use, and the second network element can request the first network element to monitor the performance of the target positioning model.

[0053] Alternatively, the first network element may trigger performance monitoring of the target positioning model. For example, the first network element may train and generate the target positioning model and send the target positioning model to at least one first access network device for use. To obtain the positioning performance of the target positioning model, the first access network device may trigger performance monitoring of the target positioning model, for example, by executing S310 above.

[0054] In the above implementation, the second network element may be a network data analysis function (NWDAF) or another network element for training a target positioning model, which is not specifically limited in the embodiments of the present application.

[0055] In the above implementation, the identification information of the target positioning model carried in the positioning performance monitoring message can be the identification of the target positioning model, that is, Model ID, or the functional identification (functionality ID) of the target positioning model, or the analysis identification (analytics ID) of the target positioning model.

[0056] In an optional implementation, the positioning performance monitoring message may further carry at least one of the following:

[0057] (1) Identification information of the target device; for example, identification information of the first access network device, identification information of the second access network device, or identification information of the second network element, used to indicate the identity information of the target device to which the positioning performance monitoring message is sent. For example, if it is an access network device (including the first access network device and the second access network device), the identification information may be an identification of the access network device, such as a cell global identity (CGI), or the identification information may also be an Operation Administration and Maintenance (OAM) identification of the access network device (including the first access network device and the second access network device).

[0058] (2) Monitoring period information, which is used to indicate the time for monitoring the performance of the target positioning model; the monitoring period information can indicate the time for monitoring the performance of the model, for example, the next day, the next month, etc. Optionally, the monitoring period information may include at least one of: monitoring start time, monitoring end time, monitoring repetition mode, monitoring repetition start time, and monitoring repetition end time.

[0059] (3) A first threshold (notify threshold), which is used to indicate that the target device is notified when the performance parameter corresponding to the first positioning performance monitoring information reaches the first threshold. The first threshold can indicate that the positioning performance monitoring information of the target positioning model is reported to the target device when the performance parameter meets a certain condition. For example, if it is a scheduled report, the first threshold may include: one or more of: the reporting start time, the reporting interval, and the number of reports. If it is a conditional report, the first threshold may be at least one of the positioning accuracy threshold and the model accuracy threshold.

[0060] In the above optional implementation, the positioning performance monitoring message can be used to request the performance monitoring data of the target positioning model once, or to request the performance monitoring data of the target positioning model multiple times. For example, the positioning performance monitoring message can be a model performance monitoring request message, and the first network element can reply with the performance monitoring data of the target positioning model once in response to the request message. Alternatively, the positioning performance monitoring message can also be a model performance monitoring subscription message, and the first network element may need to reply with the performance monitoring data of the target positioning model multiple times in response to the subscription message, for example, replying with the performance monitoring data of the target positioning model to the target device once each time the performance monitoring data of the target positioning model changes, or periodically replying with the performance monitoring data of the target positioning model to the target device according to a certain period.

[0061] In the above optional implementation, optionally, after the first network element receives the positioning performance monitoring message sent by the target device, the method may further include: the first network element sending a positioning performance monitoring response message to the target device, wherein the positioning performance monitoring response message carries identification information of the target positioning model. The positioning performance monitoring response message can notify the target device that the first network element has received the positioning performance monitoring message.

[0062] In an optional implementation, after the first network element receives the above-mentioned positioning performance monitoring message, if the first network element has learned the second positioning performance monitoring information of the target positioning model and the learned second positioning performance monitoring information meets the second threshold, then the positioning performance monitoring response message may also carry performance indicators corresponding to the second positioning performance monitoring information, such as positioning accuracy information and model accuracy information. The second threshold may be the same as the first threshold, for example, when the first threshold is a positioning accuracy threshold or a model accuracy threshold, the second threshold may be the first threshold, or the second threshold may be different from the first threshold, for example, when the first threshold includes one or more of the reporting start time, the reporting interval, and the number of reports, the second threshold may be different from the first threshold, for example, the second threshold may be a preset value or a value indicated by a network-side device, etc. Through this optional implementation method, if the first network element has completed performance monitoring of the target positioning model and the performance meets the requirements (for example, the positioning performance monitoring message carries a first threshold, and the first threshold is a positioning accuracy threshold or a model accuracy threshold), the first network element can directly inform the target device of the performance indicators corresponding to the second positioning performance monitoring information in the response message, so there is no need to initiate the performance monitoring process again, thereby saving the process.

[0063] In the above optional implementation, the positioning performance monitoring response message may also carry at least one of the following:

[0064] The fourth indication information is used to indicate one of the following: accepting performance monitoring of the target positioning model or refusing performance monitoring of the target positioning model; for example, when the first network element has completed performance monitoring of the target positioning model, and the performance indicator corresponding to the second positioning performance monitoring information obtained by monitoring does not meet the requirements (for example, the performance indicator corresponding to the obtained second positioning performance monitoring information does not meet the second threshold), the fourth indication information carried in the positioning performance monitoring response message can indicate refusing performance monitoring of the target positioning model.

[0065] A reason value is used to indicate the reason for rejecting performance monitoring of the target positioning model. For example, the reason for rejecting performance monitoring of the target positioning model includes at least one of the following: network congestion; the load of the first network element is greater than a threshold; the model performance of the target positioning model does not meet the requirements; or positioning performance monitoring of the target positioning model is in progress.

[0066] After receiving the positioning performance monitoring response message, the target device can learn whether the first network element accepts the performance monitoring of the target positioning model. If not, the target device can further learn the reason for rejection.

[0067] In an optional implementation, after receiving the positioning performance monitoring message, the first network element may execute S310 above, and then execute S312 and S314 below to obtain first positioning performance monitoring information of the target positioning model, and then send a positioning performance monitoring notification message to the target device. The positioning performance monitoring notification message carries identification information of the target positioning model and a performance indicator corresponding to the first positioning performance monitoring information, for example, at least one of positioning accuracy information and model accuracy information corresponding to the first positioning performance monitoring information. This allows the target device to obtain the performance indicator of the target positioning model and further determine whether to use the target positioning model for inference based on the performance indicator.

[0068] S312: The first network element receives a positioning result message sent by the at least one first access network device.

[0069] The positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information.

[0070] After receiving the above-mentioned positioning message, the first access network device can send a positioning result message to the first network element, and report at least one of the AI ​​measurement information, the AI ​​measurement information and the actual measurement information, and the difference information between the AI ​​measurement information and the actual measurement information to the first network element through the positioning result message.

[0071] In an optional implementation, the positioning result message may also carry the following information:

[0072] The identification information of the target positioning model; through the identification information of the target positioning model, the first network element and the target positioning model associated with the target positioning measurement information can be indicated.

[0073] The third indication information is used to indicate that the positioning result message carries the target positioning measurement information. The third indication information can indicate that the positioning result message carries information related to AI measurement information inferred using the target positioning model. For example, the third indication information can be an AI information indication (AI info ind).

[0074] Optionally, the positioning result message may also carry a first timestamp indicating the time when the AI ​​measurement information was inferred using the target positioning model. The first timestamp information can be used to find the corresponding AI measurement information and conventional measurement information in subsequent steps, facilitating performance comparison.

[0075] S314: The first network element obtains first positioning performance monitoring information of the target positioning model based on the target positioning measurement information.

[0076] In an optional implementation, when the target positioning measurement information includes the AI ​​measurement information and the real measurement information, in S314, the first network element can compare the difference between the AI ​​measurement information and the real measurement information, and then obtain the first positioning performance monitoring information of the target positioning model based on the difference between the AI ​​measurement information and the real measurement information. For example, when there are multiple first access network devices, the first network element can obtain the difference between the AI ​​measurement information and the real measurement information corresponding to each first access network device based on the AI ​​measurement information and the real measurement information reported by each first access network device, and then calculate the difference corresponding to each first access network device based on a predetermined algorithm, thereby obtaining the first positioning performance monitoring information, wherein the predetermined algorithm can be variance, mean square error (MSE), mean absolute error (MAE), etc., and the specific algorithm is not limited in the embodiment of this application.

[0077] In an optional implementation, when the target positioning measurement information includes the difference information, in S314, the first network element may directly obtain the first positioning performance monitoring information of the target positioning model based on the difference information.

[0078] In an optional implementation, when the target positioning measurement information includes the AI ​​measurement information, in S314, the first network element may obtain terminal location information, wherein the terminal location information includes: terminal location estimation and second timestamp information; then, based on the AI ​​measurement information, obtain a model location estimation; and finally, based on the difference between the terminal location estimation and the model location estimation, obtain first positioning performance monitoring information of the target positioning model.

[0079] In the above-mentioned optional implementation method, the first network element may obtain the terminal location information from the UE, or the first network element may also obtain the terminal location information through calculation, for example, the terminal location information obtained based on real measurement information, or the terminal location information may also be obtained by the first network element based on another positioning task at the same time (the positioning task uses a non-AI positioning method) or a new positioning task initiated in the time period. The specific embodiment of the present application is not limited thereto.

[0080] In the above-mentioned optional implementation method, the first network element can match the second timestamp information in the acquired terminal location information with the time information of the AI ​​measurement information obtained by inferring the target positioning model, use the AI ​​measurement information obtained at the same time to perform position estimation, obtain a model position estimate, and compare the model position estimate with the terminal position estimate in the terminal location information to obtain the difference between the two position estimates.

[0081] In an optional implementation, the first network element may reply to the target device with a model performance detection reply message or a model performance detection notification message, where the message may carry at least one of the following model performance information:

[0082] Model identification information, model ID;

[0083] Performance Key Performance Indicators (KPIs) are used to characterize the performance parameters of the target positioning model inference. For example, they may include positioning accuracy information, model accuracy information, etc.

[0084] In the above implementation, optionally, the first network element may send the model performance information to the above target device if the local reporting policy or the above first threshold is met.

[0085] For example, the first network element may send the model performance information to the first access network device, and the first access network device may send the model performance monitoring information to the model training party (eg, LMF or NWDAF).

[0086] Through the above-mentioned technical solution provided by the embodiment of the present application, a first network element sends a positioning message to at least one first access network device, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and actual measurement information, or difference information between the AI ​​measurement information and the actual measurement information; the first network element receives a positioning result message sent by the at least one first access network device, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and actual measurement information, or difference information between the AI ​​measurement information and the actual measurement information; and the first network element obtains first positioning performance monitoring information of the target positioning model based on the target positioning measurement information. This enables the first network element to monitor the positioning performance of the target positioning model based on the target positioning measurement information reported by the first access network device, solving the problem of being unable to determine the accuracy and precision of the estimated result obtained by using the AI ​​measurement information inferred by the AI ​​positioning model due to the unknown performance of the AI ​​positioning model.

[0087] Based on the same technical concept, an embodiment of the present application also provides a method for sending positioning monitoring information of a positioning model.

[0088] FIG4 shows a flow chart of a method for sending positioning monitoring information of a positioning model provided in an embodiment of the present application. The method 400 can be executed by the first access network device mentioned above. As shown in FIG4 , the method mainly includes the following steps.

[0089] S410: A first access network device receives a positioning message sent by a first network element.

[0090] In an embodiment of the present application, the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information.

[0091] Among them, the positioning message is the same as the positioning message in method 300. The first network element can send the positioning message to the first access network device according to the relevant description in the above method 300. For details, please refer to the relevant description in method 300.

[0092] In an optional implementation, before S410, the method may further include: the first access network device sends a positioning performance monitoring message to the first network element, wherein the positioning performance monitoring message is used to request performance measurement of the target positioning model, and the positioning performance monitoring message carries identification information of the target positioning model. In this optional implementation, the first access network device can trigger the first network element to perform performance monitoring on the target positioning model. For example, the first access network device can trigger the first network element to perform performance monitoring on the target positioning model after receiving the target positioning model distributed by the network side device, or after the first access network device has trained and generated the target positioning model, so as to determine the positioning performance of the target positioning model.

[0093] In the above optional implementation, optionally, the above positioning performance monitoring message may further carry at least one of the following:

[0094] (1) Identification information of the first access network device; for example, the identification information may be an identification (ID) of the first access network device, such as a cell global identity (CGI), or the identification information may be an operation administration and maintenance (OAM) identifier of the first access network device (including the first access network device and the second access network device).

[0095] (2) Monitoring period information, which is used to indicate the time for monitoring the performance of the target positioning model; the monitoring period information can indicate the time for monitoring the performance of the model, for example, the next day, the next month, etc. Optionally, the monitoring period information may include at least one of: monitoring start time, monitoring end time, monitoring repetition mode, monitoring repetition start time, and monitoring repetition end time.

[0096] (3) A first threshold (notify threshold), which is used to indicate that the target device is notified when the performance parameter corresponding to the first positioning performance monitoring information reaches the first threshold. The first threshold can indicate that the positioning performance monitoring information of the target positioning model is reported to the target device when the performance parameter meets a certain condition. For example, if it is a scheduled report, the first threshold may include: one or more of: the reporting start time, the reporting interval, and the number of reports. If it is a conditional report, the first threshold may be at least one of the positioning accuracy threshold and the model accuracy threshold.

[0097] In the above optional implementation, the positioning performance monitoring message can be used to request the performance monitoring data of the target positioning model once, or to request the performance monitoring data of the target positioning model multiple times. For example, the positioning performance monitoring message can be a model performance monitoring request message, and in response to the request message, the first access network device can receive the performance monitoring data of the target positioning model once replied by the first network element. Alternatively, the positioning performance monitoring message can also be a model performance monitoring subscription message, and in response to the subscription message, the first access network device can receive the performance monitoring data of the target positioning model multiple times replied by the first network element. For example, each time the performance monitoring data of the target positioning model changes, the first network element can reply to the first access network device with the performance monitoring data of the target positioning model once, or, according to a certain period, periodically reply to the first access network device with the performance monitoring data of the target positioning model.

[0098] In an optional implementation, after the first access network device sends the positioning performance monitoring message to the first network element, the method may further include: the first access network device receiving a positioning performance monitoring response message sent by the first network element, wherein the positioning performance monitoring response message carries identification information of the target positioning model. Through the positioning performance monitoring response, the first access network device can be informed that the first network element has received the positioning performance monitoring message.

[0099] Optionally, the positioning performance monitoring response message may also carry performance indicators corresponding to the second positioning performance monitoring information, such as positioning accuracy information and model accuracy information. Through this optional implementation, if the first network element has completed performance monitoring of the target positioning model and the performance meets the requirements (for example, the performance indicator corresponding to the acquired second positioning performance monitoring information meets the second threshold), the first network element can directly inform the target device of the performance indicator corresponding to the second positioning performance monitoring information in the response message, so that there is no need to initiate the performance monitoring process again, thereby saving the process.

[0100] In the above optional implementation, the positioning performance monitoring response message may also carry at least one of the following:

[0101] The fourth indication information is used to indicate one of the following: accepting performance monitoring of the target positioning model or refusing performance monitoring of the target positioning model; for example, when the first network element has completed performance monitoring of the target positioning model and the performance does not meet the requirements (for example, the performance indicator corresponding to the obtained second positioning performance monitoring information does not meet the second threshold), the fourth indication information carried in the positioning performance monitoring response message can indicate refusing performance monitoring of the target positioning model.

[0102] A reason value is used to indicate the reason for rejecting performance monitoring of the target positioning model. For example, the reason for rejecting performance monitoring of the target positioning model includes at least one of the following: network congestion; the load of the first network element is greater than a threshold; the model performance of the target positioning model does not meet the requirements; or positioning performance monitoring of the target positioning model is in progress.

[0103] After receiving the positioning performance monitoring response message, the first access network device can learn whether the first network element accepts the performance monitoring of the target positioning model. If not, the first access network device can further learn the reason for rejection.

[0104] In an optional implementation, the positioning message may carry at least one of the following:

[0105] identification information of the target positioning model;

[0106] The first indication information is used to instruct the first access network device to perform positioning using the AI ​​measurement information inferred by the target positioning model;

[0107] The second indication information is used to instruct the first network element to perform performance monitoring on the target positioning model.

[0108] Through the positioning message, the first access network device may learn that the first network element will perform performance monitoring on the target positioning model, or the information that the first network element needs to obtain.

[0109] In a specific application, the positioning message may be a New Radio Positioning Protocol a (NRPPa) positioning message.

[0110] S412: The first access network device obtains real measurement information and AI measurement information inferred using the target positioning model.

[0111] After receiving the above-mentioned positioning message, the first access network device learns that it needs to report at least one of the AI ​​measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and the actual measurement information, and the difference information between the AI ​​measurement information and the actual measurement information. Therefore, the first access network device can use the target positioning model for reasoning to obtain the AI ​​measurement information. In addition, the first access network device can also use conventional positioning measurements, such as measurement of positioning reference signals, to obtain the actual measurement information.

[0112] S414: The first access network device sends a positioning result message to the first network element based on the real measurement information and the AI ​​measurement information.

[0113] In an embodiment of the present application, the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and the difference information between the AI ​​measurement information and the real measurement information.

[0114] The first access network device sends a positioning result message carrying target positioning measurement information to the first network element, so that the first network element can obtain positioning performance monitoring information of the target positioning model based on the target positioning measurement information, and realize positioning performance monitoring of the target positioning model.

[0115] In an optional implementation, the positioning result message may carry the following information:

[0116] (1) identification information of the target positioning model;

[0117] (2) Third indication information, used to indicate that the positioning result message carries the target positioning measurement information.

[0118] Optionally, the positioning result message further carries first timestamp information, which is used to indicate the time when the AI ​​measurement information is inferred using the target positioning model.

[0119] In an optional implementation, when the first access network device sends the aforementioned positioning performance monitoring message to the first network element, after S414, the method may further include: the first access network device receiving a positioning performance monitoring notification message sent by the first network element, the positioning performance monitoring notification message carrying identification information of the target positioning model and a performance indicator corresponding to the first positioning performance monitoring information, for example, at least one of positioning accuracy information and model accuracy information corresponding to the first positioning performance monitoring information. This allows the first access network device to obtain the performance indicator of the target positioning model and further determine whether to use the target positioning model for positioning inference based on the performance indicator.

[0120] Through the above-mentioned technical solution provided by the embodiment of the present application, the first access network device can send a positioning result message to the first network element after receiving the above-mentioned positioning message, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and the difference information between the AI ​​measurement information and the real measurement information, so that the first network element can obtain the first positioning performance monitoring information of the target positioning model based on the target positioning measurement information, thereby monitoring the positioning performance of the target positioning model, and solving the problem that the accuracy and precision of the estimation result obtained by using the AI ​​measurement information inferred by the AI ​​positioning model for position estimation cannot be determined due to the unknown performance of the AI ​​positioning model.

[0121] In an embodiment of the present application, after obtaining the performance parameters of the target positioning model, the above-mentioned target device can optimize the target positioning model to improve the positioning accuracy of the target positioning model and optimize the service experience of AI positioning when the model performance is poor.

[0122] The following describes the technical solution provided in the embodiment of the present application by taking the first network element as LMF and the second network element as NWDAF as an example.

[0123] FIG5 shows another flow chart of a method for obtaining positioning model performance information provided in an embodiment of the present application. As shown in FIG5 , the method mainly includes the following steps:

[0124] Step 501: Base station A obtains an AI positioning model.

[0125] In FIG5 , base station A is represented by access network equipment (RAN).

[0126] As shown in FIG5 , this step includes but is not limited to the following three implementations:

[0127] Step 501a: Base station A trains and generates an AI positioning model;

[0128] Step 501b: The AI ​​positioning model generated by LMF training is then sent to base station A;

[0129] Step 501c: After the NWDAF training generates a model, it is sent to base station A.

[0130] It should be noted that, before step 501b or step 501c, base station A may optionally send a request or subscription message to LMF or NWDAF, requesting LMF or NWDAF to provide base station A with an AI positioning model.

[0131] In step 502, base station A decides to perform performance monitoring on the target AI positioning model, triggering the performance monitoring process.

[0132] In step 503, base station A sends a model performance monitoring request or a signing message to the LMF, requesting performance monitoring of the target AI positioning model and feedback of performance monitoring information.

[0133] The model performance monitoring request or contract signing message may include at least one of the following:

[0134] Model identification information, which represents the target AI positioning model. For example, it can be one of: Model ID, functionality ID, and analytics ID.

[0135] Base station identification information, representing the identity information of base station A requesting model detection, for example, it can be CGI;

[0136] A monitoring period, which represents the time during which the model performance is monitored (e.g., the next day, the next month), and may include, for example, at least one of: a monitoring start time, a monitoring end time, a monitoring repetition pattern, a monitoring repetition start time, and a monitoring repetition end time;

[0137] The first threshold (notify threshold) indicates that the performance parameter is reported to base station A when it meets a certain condition. For example, it includes scheduled reporting: reporting start time, reporting interval, and number of reports; or it includes conditional reporting: positioning accuracy threshold and model accuracy threshold.

[0138] It should be noted that the above-mentioned model performance monitoring request or signing message can be encapsulated in the NRPPa message and sent to the LMF. NRPPa is a point-to-point protocol stack between NG-RAN and LMF, which is used to transmit positioning-related signaling. A new message can also be added to transmit the request or signing message.

[0139] It should be noted that the reply corresponding to the model performance monitoring request message is one-time, while the reply corresponding to the model performance monitoring contract message is one or more times.

[0140] It should be noted that the monitoring time period and the first threshold can be optional parameters. When one or both of them are not provided, LMF can detect and report according to the local policy or default monitoring parameters. For example, the local configuration defaults to a detection time period of one day in the future, and the first threshold is to report when the positioning accuracy is less than 1m.

[0141] It should be noted that the base station identification information may be a base station ID, or an OAM ID corresponding to the base station may be used to identify an OAM.

[0142] Step 504: LMF decides whether to perform this model detection based on the current business situation, load situation, and the known model detection performance.

[0143] If not, for example, the model ID requested for detection has been detected and the performance is poor or is being detected, then step 505 is executed and subsequent steps are ignored; if yes, then step 504 is executed and subsequent steps are executed.

[0144] In another implementation method, step 505 is not mandatory, and LMF does not need to reply. LMF executes steps 506 to 513 after deciding to perform monitoring. LMF directly executes step 513 after deciding not to perform monitoring.

[0145] Step 505: LMF sends a model performance detection reply message to base station A.

[0146] The model performance test reply message may include at least one of the following:

[0147] Model identification information, representing the model ID of the target model to be monitored;

[0148] Performance KPI, which represents the performance parameters of model inference, including positioning accuracy information and model accuracy information;

[0149] The fourth indication information represents acceptance or rejection of the performance monitoring of the target model, including acceptance and rejection;

[0150] Reason value, representing the reason for rejecting performance monitoring;

[0151] It should be noted that when the LMF has learned the monitoring performance of the target model or the known performance meets the first threshold in step 503, the model performance detection reply message carries the performance KPI and reports the model performance information to base station A. The fourth indication information is an optional parameter and can be implicit indication information, for example, carrying a reason value indicates rejection, and not carrying a reason value indicates acceptance, or it can be explicit indication information that clearly indicates acceptance or rejection of performance monitoring of the target model; the reason value is an optional parameter, including one of the following: network congestion, excessive load, model performance does not meet requirements, and monitoring in progress.

[0152] Step 506: In the case of a Mobile Original Location Request (MO-LR), Mobile Terminated Location Request (MT-LR), or Deferred MT-LR location request, the LMF determines that the UE needs to be located, selects a location method, and determines the base stations involved in the location.

[0153] In this step, the LMF can locate the UE by interacting with the Access and Mobility Management Function (AMF) and the Gateway Mobile Location Centre (GMLC).

[0154] It should be noted that the base station involved in positioning may not be the aforementioned base station A, but may be another base station. The illustrations in the subsequent steps do not distinguish for the sake of simplicity, and do not mean that the RAN here always has only one base station.

[0155] In step 507, the LMF sends a positioning message (an enhanced or newly added NRPPa message) to one or more base stations B (not limited to base station A, base station B can be base station A), requesting one or more base stations B to report the measurement information of the UE. The positioning message carries at least one of the following:

[0156] Model identification information, such as model ID and analytics ID;

[0157] The first indication information, i.e., the AI ​​positioning indication, indicates that the base station is required to report AI measurement information inferred using the target model;

[0158] It should be noted that in the related technology, LMF sends NRPPa positioning messages to one or more base stations B, and one or more base stations B return conventional measurement information (i.e., non-AI measurement information) to LMF. In the embodiment of the present application, the base station is required to report AI measurement information through the second indication information.

[0159] In step 508, one or more base stations B obtain conventional positioning measurement information (ie, the above-mentioned real measurement information) and AI measurement information.

[0160] Step 509: One or more base stations B reply with a measurement result to the LMF. The measurement result may include at least one of the following:

[0161] The third indication information, i.e., AI indication, indicates that this measurement result carries AI measurement information inferred using the target model;

[0162] Model identification information, such as model ID and analytics ID;

[0163] AI measurement information, representing AI measurement information inferred using the target model;

[0164] The difference information between AI measurement and conventional measurement, which represents the difference between AI measurement information and conventional measurement information, can be a matrix information;

[0165] Conventional measurement information (i.e. the above-mentioned real measurement information);

[0166] Timestamp.

[0167] It should be noted that, when the third indication is carried, one of the two parameters, AI measurement information and AI measurement and conventional measurement information difference information, can be carried.

[0168] In step 510, the LMF obtains UE location information from the UE, which is called a label positioning result or a UE position estimate calculated by non-AI positioning, or uses a positioning method different from the positioning method selected in S506 to locate the UE once and obtain a regular positioning result of the UE.

[0169] This step is optional.

[0170] One implementation method is to select the uplink positioning method in S506. This step uses the downlink positioning method to initiate a new positioning process to the UE, requiring the UE to report the UE location. It should be noted that in order for the LMF to compare the positioning estimates of the two positionings (AI positioning and conventional positioning), the UE can carry a timestamp when reporting the positioning estimate, and the LMF can bind it based on the timestamp; alternatively, the LMF can assign the same task identification information (e.g., LCS correlation ID) to the two positionings.

[0171] Step 511: LMF performs performance comparison and calculates performance KPI information such as positioning accuracy and model accuracy.

[0172] In specific applications, you can calculate positioning accuracy and model accuracy by selecting one of the following methods:

[0173] (1) Direct comparison: Compare the positioning estimates. Use the AI ​​measurements to calculate the AI ​​positioning estimate and compare it with the conventional positioning estimate obtained in step 510. Here, the comparison is between the AI ​​position estimate and the label positioning result.

[0174] (2) Indirect comparison 1: Comparison of measurement information, directly comparing AI measurement information with conventional measurement information;

[0175] (3) Indirect comparison 2: Directly judge the difference information between AI measurement and conventional measurement. LMF determines the model performance by judging whether the measurement difference meets the threshold.

[0176] Step 512: LMF responds to base station A with a model performance detection response or notification message.

[0177] The model performance test reply or notification message can include at least one of the following:

[0178] Model identification information, model ID;

[0179] Performance KPI, which represents the performance parameters of model inference, including positioning accuracy information and model accuracy information.

[0180] Optionally, when the LMF local reporting strategy or the first threshold condition in step 503 is met, the LMF replies to base station A with the above-mentioned model performance detection reply or notification message, and sends the model performance information to base station A.

[0181] Step 513: Base station A sends the model performance monitoring information to the model training party (LMF or NWDAF), including the information in the above step 512.

[0182] This step is optional.

[0183] FIG6 shows another flow chart of a method for obtaining positioning model performance information provided in an embodiment of the present application. The difference between this method and FIG6 is that this method is triggered by NWDAF to perform model performance monitoring. As shown in FIG6, this method mainly includes the following steps:

[0184] In step 601, NWDAF decides to perform performance monitoring on the target AI positioning model.

[0185] For example, NWDAF can generate a target AI positioning model during training, send the target AI positioning model to at least one base station A for use, and then NWDAF decides to perform performance monitoring on the target AI positioning model.

[0186] In step 602, NWDAF sends a model performance monitoring request or a contract message to LMF. It should be noted that this step can occur through a direct interface between NWDAF and LMF, or can be forwarded through GMLC. The same applies to the subsequent step 611.

[0187] Steps 603 to 610 are similar to steps 504 to 511 above. LMF performs performance monitoring of the target AI positioning model and calculates performance KPI information such as positioning accuracy and model accuracy.

[0188] Step 611: LMF sends a model performance detection response or notification message to NWDAF.

[0189] The model performance test reply or notification message can include at least one of the following:

[0190] Model identification information, model ID;

[0191] Performance KPI, which represents the performance parameters of model inference, including positioning accuracy information and model accuracy information.

[0192] The above-mentioned technical solution provided by the embodiment of the present application solves the problem of being unable to determine the accuracy and precision of the estimation results obtained by using the AI ​​measurement information inferred by the AI ​​positioning model for position estimation due to the unknown performance of the AI ​​positioning model. As a result, the model provider can optimize the AI ​​positioning model based on the monitoring results and improve the positioning accuracy of the AI ​​positioning model.

[0193] The method for obtaining positioning model performance information provided in the embodiment of the present application can be executed by a positioning model performance information obtaining device. In the embodiment of the present application, the positioning model performance information obtaining device executing the method for obtaining positioning model performance information is used as an example to illustrate the positioning model performance information obtaining device provided in the embodiment of the present application.

[0194] FIG7 shows a schematic structural diagram of a device for acquiring positioning model performance information provided in an embodiment of the present application. As shown in FIG7 , the device 700 includes: a first sending module 701 , a first receiving module 702 and a first acquiring module 703 .

[0195] In an embodiment of the present application, a first sending module 701 is used to send a positioning message to at least one first access point and a first access network device, wherein the positioning message is used to request a report of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and the actual measurement information, and the difference information between the AI ​​measurement information and the actual measurement information; a first receiving module 702 is used to receive a positioning result message sent by the at least one first access network device, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and the actual measurement information, and the difference information between the AI ​​measurement information and the actual measurement information; a first acquisition module 703 is used to obtain first positioning performance monitoring information of the target positioning model based on the target positioning measurement information.

[0196] In an optional implementation, the positioning message carries at least one of the following:

[0197] identification information of the target positioning model;

[0198] The first indication information is used to instruct the first access network device to perform positioning using the AI ​​measurement information inferred by the target positioning model;

[0199] The second indication information is used to instruct the first network element to perform performance monitoring on the target positioning model.

[0200] In an optional implementation, the positioning result message also carries the following information:

[0201] identification information of the target positioning model;

[0202] The third indication information is used to indicate that the positioning result message carries the target positioning measurement information.

[0203] In an optional implementation, the positioning result message further carries first timestamp information, which is used to indicate the time when the AI ​​measurement information is inferred using the target positioning model.

[0204] In an optional implementation, obtaining first positioning performance monitoring information of the target positioning model based on the target positioning measurement information includes one of the following:

[0205] When the target positioning measurement information includes the AI ​​measurement information and the real measurement information, comparing a difference between the AI ​​measurement information and the real measurement information, and obtaining first positioning performance monitoring information of the target positioning model based on the difference between the AI ​​measurement information and the real measurement information;

[0206] In a case where the target positioning measurement information includes the difference information, obtaining first positioning performance monitoring information of the target positioning model based on the difference information;

[0207] In a case where the target positioning measurement information includes the AI ​​measurement information, terminal position information is obtained, wherein the terminal position information includes: a terminal position estimate and second timestamp information; based on the AI ​​measurement information, a model position estimate is obtained; and based on a difference between the terminal position estimate and the model position estimate, first positioning performance monitoring information of the target positioning model is obtained.

[0208] In an optional implementation, the first receiving module 702 is also used to receive a positioning performance monitoring message sent by the target device, wherein the positioning performance monitoring message is used to request performance measurement of the target positioning model, and the positioning performance monitoring message carries identification information of the target positioning model; the target device includes one of the following: the first access network device, the second access network device, and the second network element.

[0209] In an optional implementation, the positioning performance monitoring message further carries at least one of the following:

[0210] identification information of the target device of the second network element;

[0211] Monitoring time period information, used to indicate the time for performance monitoring of the target positioning model;

[0212] The first threshold is used to indicate that when the performance parameter corresponding to the first positioning performance monitoring information reaches the first threshold, the target device is notified.

[0213] In an optional implementation, the identification information of the target device includes one of the following: an identification ID of the first access network device or the second access network device, and an OAM ID corresponding to the first access network device or the second access network device.

[0214] In an optional implementation, the first sending module 701 is further configured to send a positioning performance monitoring response message to the target device, wherein the positioning performance monitoring response message carries identification information of the target positioning model.

[0215] In an optional implementation, when the second positioning performance monitoring information of the target positioning model has been obtained and the obtained second positioning performance monitoring information meets the second threshold, the positioning performance monitoring response message also carries the performance indicator corresponding to the second positioning performance monitoring information.

[0216] In an optional implementation, the positioning performance monitoring response message further carries at least one of the following:

[0217] The fourth indication information is used to indicate one of the following: accepting the performance monitoring of the target positioning model, rejecting the performance monitoring of the target positioning model;

[0218] The reason value is used to indicate the reason for rejecting the performance monitoring of the target positioning model.

[0219] In an optional implementation, the reason for refusing to perform performance monitoring on the target positioning model includes at least one of the following:

[0220] Network congestion;

[0221] The load of the first network element is greater than a threshold;

[0222] The model performance of the target positioning model does not meet the requirements;

[0223] The positioning performance of the target positioning model is being monitored.

[0224] In an optional implementation, the positioning performance monitoring message includes one of the following:

[0225] Positioning performance monitoring request message, used to request to obtain positioning performance monitoring data of the target positioning model once;

[0226] The positioning performance monitoring signing message is used to request to obtain the positioning performance monitoring data of the target positioning model at least once.

[0227] In an optional implementation, the first sending module 701 is further used to send a positioning performance monitoring notification message to the target device, wherein the positioning performance monitoring notification message carries identification information of the target positioning model and performance indicators corresponding to the first positioning performance monitoring information.

[0228] In an optional implementation, the performance indicator includes at least one of the following:

[0229] Positioning accuracy information;

[0230] Model accuracy information.

[0231] The apparatus for obtaining positioning model performance information provided in the embodiment of the present application can implement each process implemented in the method embodiment of FIG3 and achieve the same technical effect. To avoid repetition, it will not be described here.

[0232] FIG8 shows a structural diagram of a positioning monitoring information sending device of a positioning model provided in an embodiment of the present application. As shown in FIG8 , the device 800 includes: a second receiving module 801 , a second acquiring module 802 and a second sending module 803 .

[0233] In an embodiment of the present application, a second receiving module 801 is used to receive a positioning message sent by a first network element, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; a second acquisition module 802 is used to obtain real measurement information and AI measurement information inferred using the target positioning model; a second sending module 803 is used to send a positioning result message to the first network element based on the real measurement information and the AI ​​measurement information, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information.

[0234] In an optional implementation, the positioning message carries at least one of the following:

[0235] identification information of the target positioning model;

[0236] The first indication information is used to instruct the first access network device to perform positioning using the AI ​​measurement information inferred by the target positioning model;

[0237] The second indication information is used to instruct the first network element to perform performance monitoring on the target positioning model.

[0238] In an optional implementation, the positioning result message carries the following information:

[0239] identification information of the target positioning model;

[0240] The third indication information is used to indicate that the positioning result message carries the target positioning measurement information.

[0241] In an optional implementation, the positioning result message further carries first timestamp information, which is used to indicate the time when the AI ​​measurement information is inferred using the target positioning model.

[0242] The positioning monitoring information sending device of the positioning model provided in the embodiment of the present application can implement each process implemented by the method embodiment of Figure 4 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0243] As shown in Figure 9, an embodiment of the present application further provides a communication device 900, including a processor 901 and a memory 902, wherein the memory 902 stores a program or instruction that can be run on the processor 901. For example, when the communication device 900 is a first network element, when the program or instruction is executed by the processor 901, each step of the embodiment of the method for obtaining positioning model performance information is implemented, and the same technical effect can be achieved. When the communication device 900 is an access network device, when the program or instruction is executed by the processor 901, each step of the embodiment of the method for sending positioning monitoring information of the positioning model is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0244] The present application also provides a network-side device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the method embodiment shown in Figure 3 or 4. This network-side device embodiment corresponds to the aforementioned network-side device method embodiment, and each implementation process and implementation method of the aforementioned method embodiment are applicable to this network-side device embodiment and can achieve the same technical effects.

[0245] Specifically, an embodiment of the present application also provides a network-side device. As shown in Figure 10, the network-side device 1000 includes: an antenna 1001, a radio frequency device 1002, a baseband device 1003, a processor 1004, and a memory 1005. Antenna 1001 is connected to radio frequency device 1002. In the uplink direction, radio frequency device 1002 receives information via antenna 1001 and sends the received information to baseband device 1003 for processing. In the downlink direction, baseband device 1003 processes the information to be transmitted and sends it to radio frequency device 1002. Radio frequency device 1002 processes the received information and sends it through antenna 1001.

[0246] The method executed by the network-side device in the above embodiment may be implemented in the baseband device 1003 , which includes a baseband processor.

[0247] The baseband device 1003 may, for example, include at least one baseband board, on which multiple chips are arranged, as shown in Figure 10, one of which is, for example, a baseband processor, which is connected to the memory 1005 through a bus interface to call the program in the memory 1005 and execute the network device operations shown in the above method embodiment.

[0248] The network side device may further include a network interface 1006, which is, for example, a Common Public Radio Interface (CPRI).

[0249] Specifically, the network side device 1000 of the embodiment of the present application also includes: instructions or programs stored in the memory 1005 and can be run on the processor 1004. The processor 1004 calls the instructions or programs in the memory 1005 to execute the method of execution of each module shown in Figure 8 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0250] Specifically, the embodiment of the present application further provides a network-side device. As shown in FIG11 , the network-side device 1100 includes a processor 1101, a network interface 1102, and a memory 1103. The network interface 1102 is, for example, a common public radio interface (CPRI).

[0251] Specifically, the network side device 1100 of the embodiment of the present application also includes: instructions or programs stored in the memory 1103 and executable on the processor 1101. The processor 1101 calls the instructions or programs in the memory 1103 to execute the method of execution of each module shown in Figure 7 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0252] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the embodiment of the method for obtaining the performance information of the positioning model described above are implemented, or the various processes of the embodiment of the method for sending positioning monitoring information of the positioning model described above are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0253] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0254] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions, the various processes of the above-mentioned positioning model performance information acquisition method embodiment, or the various processes of the positioning monitoring information sending method embodiment of the above-mentioned positioning model, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0255] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0256] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes of the embodiment of the method for obtaining the performance information of the positioning model, or to implement the various processes of the embodiment of the method for sending positioning monitoring information of the positioning model, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0257] An embodiment of the present application also provides a communication system, including: a first network element and a first access network device, wherein the first network element can be used to execute the steps of the method for obtaining positioning model performance information as described above, and the first access network device can be used to execute the steps of the method for sending positioning monitoring information of the positioning model as described above.

[0258] In one embodiment, the communication system further includes a target device, and the target device is configured to execute the steps executed by the target device in the above-mentioned method for acquiring positioning model performance information.

[0259] In one implementation, the target device includes one of the following: the first access network device, the second access network device, and the second network element.

[0260] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0261] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0262] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. A method for obtaining positioning model performance information, comprising: The first network element sends a positioning message to at least one first access network device, wherein the positioning message is used to request reporting of one of the following: artificial intelligence AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; The first network element receives a positioning result message sent by the at least one first access network device, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; The first network element obtains first positioning performance monitoring information of the target positioning model based on the target positioning measurement information.

2. The method according to claim 1, wherein: The positioning message carries at least one of the following: identification information of the target positioning model; The first indication information is used to instruct the first access network device to use the AI ​​measurement information inferred by the target positioning model to perform positioning; The second indication information is used to indicate that the first network element will perform performance monitoring on the target positioning model.

3. The method according to claim 1 or 2, wherein: The positioning result message also carries the following information: identification information of the target positioning model; The third indication information is used to indicate that the positioning result message carries the target positioning measurement information.

4. The method according to any one of claims 1 to 3, wherein: The positioning result message also carries first timestamp information, which is used to indicate the time when the AI ​​measurement information is inferred using the target positioning model.

5. The method according to any one of claims 1 to 4, wherein: The first network element acquires first positioning performance monitoring information of the target positioning model based on the target positioning measurement information, including one of the following: In a case where the target positioning measurement information includes the AI ​​measurement information and the real measurement information, the first network element compares a difference between the AI ​​measurement information and the real measurement information, and acquires first positioning performance monitoring information of the target positioning model based on the difference between the AI ​​measurement information and the real measurement information; In a case where the target positioning measurement information includes the difference information, the first network element acquires first positioning performance monitoring information of the target positioning model based on the difference information; In the case where the target positioning measurement information includes the AI ​​measurement information, the first network element obtains terminal location information, wherein the terminal location information includes: a terminal location estimate and second timestamp information; based on the AI ​​measurement information, a model location estimate is obtained; based on a difference between the terminal location estimate and the model location estimate, first positioning performance monitoring information of the target positioning model is obtained.

6. The method according to any one of claims 1 to 5, wherein: Before the first network element sends a positioning message to at least one first access network device, the method further includes: The first network element receives a positioning performance monitoring message sent by a target device, wherein the positioning performance monitoring message Used to request performance measurement of the target positioning model, the positioning performance monitoring message carries identification information of the target positioning model; the target device includes one of the following: the first access network device, the second access network device, and the second network element.

7. The method according to claim 6, wherein: The positioning performance monitoring message also carries at least one of the following: identification information of the target device; Monitoring time period information, used to indicate the time for performance monitoring of the target positioning model; The first threshold is used to indicate that when the performance parameter corresponding to the first positioning performance monitoring information reaches the first threshold, the target device is notified.

8. The method according to claim 7, wherein: The identification information of the target device includes one of the following: an identification ID of the first access network device or the second access network device, and an operation, administration and maintenance OAM ID corresponding to the first access network device or the second access network device.

9. The method according to claim 7, wherein: After the first network element receives the positioning performance monitoring message sent by the target device, the method further includes: The first network element sends a positioning performance monitoring response message to the target device, wherein the positioning performance monitoring response message carries identification information of the target positioning model.

10. The method according to claim 9, wherein: When the first network element has obtained the second positioning performance monitoring information of the target positioning model and the performance indicator corresponding to the obtained second positioning performance monitoring information meets the second threshold, the positioning performance monitoring response message also carries the performance indicator corresponding to the second positioning performance monitoring information.

11. The method according to claim 9 or 10, wherein: The positioning performance monitoring response message also carries at least one of the following: The fourth indication information is used to indicate one of the following: accepting the performance monitoring of the target positioning model or refusing the performance monitoring of the target positioning model; A reason value is used to indicate the reason for rejecting the performance monitoring of the target positioning model.

12. The method according to claim 11, wherein: The reason for refusing to perform performance monitoring on the target positioning model includes at least one of the following: Network congestion; The load of the first network element is greater than a threshold; The model performance of the target positioning model does not meet the requirements; The positioning performance of the target positioning model is being monitored.

13. The method according to any one of claims 6 to 12, wherein: The positioning performance monitoring message includes one of the following: A positioning performance monitoring request message, used to request to obtain positioning performance monitoring data of the target positioning model once; Positioning performance monitoring signing message, used to request to obtain at least one positioning performance monitoring data of the target positioning model according to.

14. The method according to any one of claims 6 to 12, wherein: After the first network element acquires first positioning performance monitoring information of the target positioning model based on the target positioning information, the method further includes: The first network element sends a positioning performance monitoring notification message to the target device, wherein the positioning performance monitoring notification message carries identification information of the target positioning model and a performance indicator corresponding to the first positioning performance monitoring information.

15. The method according to claim 10 or 14, wherein: The performance indicators include at least one of the following: Positioning accuracy information; Model accuracy information.

16. A method for sending positioning monitoring information of a positioning model, comprising: The first access network device receives a positioning message sent by the first network element, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; The first access network device obtains real measurement information and AI measurement information inferred using the target positioning model; The first access network device sends a positioning result message to the first network element based on the real measurement information and the AI ​​measurement information, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information.

17. The method according to claim 16, wherein: The positioning message carries at least one of the following: identification information of the target positioning model; The first indication information is used to instruct the first access network device to use the AI ​​measurement information inferred by the target positioning model to perform positioning; The second indication information is used to indicate that the first network element will perform performance monitoring on the target positioning model.

18. The method according to claim 17, wherein: The positioning result message carries the following information: identification information of the target positioning model; The third indication information is used to indicate that the positioning result message carries the target positioning measurement information.

19. The method according to any one of claims 16 to 18, wherein: The positioning result message also carries first timestamp information, which is used to indicate the time when the AI ​​measurement information is inferred using the target positioning model.

20. A device for acquiring positioning model performance information, comprising: A first sending module is configured to send a positioning message to at least one first access point and a first access network device, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; The first receiving module is configured to receive a positioning result message sent by the at least one first access network device, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; The first acquisition module is used to acquire first positioning performance monitoring information of the target positioning model based on the target positioning measurement information.

21. A positioning monitoring information sending device for a positioning model, comprising: A second receiving module is configured to receive a positioning message sent by the first network element, wherein the positioning message is used to request reporting of one of the following: AI measurement information generated by reasoning using the target positioning model, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information; A second acquisition module is used to acquire real measurement information and AI measurement information inferred using the target positioning model; A second sending module is used to send a positioning result message to the first network element based on the real measurement information and the AI ​​measurement information, wherein the positioning result message carries target positioning measurement information, and the target positioning measurement information includes one of the following: the AI ​​measurement information, the AI ​​measurement information and real measurement information, and difference information between the AI ​​measurement information and the real measurement information.

22. A network side device, comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 19 are implemented.

23. A readable storage medium storing a program or an instruction, wherein the program or the instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 19.

Images