Monitoring method and device of AI / ML model

Abstract

The embodiment of the invention provides an AI / ML model monitoring method and device. The method comprises the steps that a model monitoring device receives a model monitoring request sent by a model result output device, sends request information used for requesting model information to a model deployment device, and receives the model information fed back by the model deployment device.

Description

AI / ML model monitoring method and device Technical Field

[0001] The present application relates to the field of communication technology. Background Art

[0002] With the commercialization of fifth-generation (5G) communications, and particularly the large-scale deployment of the Industrial Internet, the demand for positioning terminal devices in wireless communications has increased significantly. Traditional wireless positioning relies on a variety of technologies. Among them, those directly relevant to 5G NR (New Radio) primarily utilize channel measurement results between network entities and terminals for estimation, such as TDOA (Time Difference of Arrival), E-CID (Enhanced Cell ID), and Multi-RTT (Multi-Round-Trip Time). These traditional positioning methods all have several inherent flaws, resulting in poor positioning accuracy for terminal devices in various wireless environments or scenarios. In particular, in environments with severe non-line-of-sight (NLOS) conditions, such as indoor factories (InF), traditional positioning methods can produce extremely large errors, often making them unacceptable. The fundamental reason for this is that positioning methods based on wireless channel measurement are only effective in line-of-sight (LOS) environments. Wireless channel measurements obtained in NLOS environments often deviate significantly from ideal values, and the accuracy of terminal positioning results directly depends on these measurements. Therefore, measurement errors lead to errors in the final terminal positioning results.

[0003] In recent years, artificial intelligence and machine learning (AI / ML) technologies, represented by deep learning, have developed rapidly and, due to their powerful nonlinear fitting capabilities, have been applied in various research and commercial fields. Similarly, the performance of AI-based wireless positioning evaluation has significantly improved compared to traditional methods.

[0004] However, due to the complex and volatile wireless communication environment and the inherent characteristics of big data-based AI / ML models used for wireless positioning, the generalization performance of AI / ML models when applied to wireless positioning (the consistency of inference operations using the same model in different environments) is poor. If the performance of the current AI / ML model no longer meets the positioning performance requirements, timely model monitoring is required, and the monitoring results can be used to make further adjustments to the AI / ML model, such as model reselection, model switching, or model rollback.

[0005] It should be noted that the above introduction to the technical background is merely intended to provide a clear and complete description of the technical solutions of this application and facilitate understanding by those skilled in the art. Simply because these solutions are described in the background technology section of this application, it should not be assumed that the above technical solutions are well known to those skilled in the art.

[0006] Summary of the Invention

[0007] However, the inventors discovered that, regardless of performance, the mathematical models and computational modules used by traditional positioning methods are fixed, lacking a corresponding supervision mechanism. Therefore, the real-time accuracy achieved using traditional positioning methods cannot be accurately measured. Generally speaking, because AI / ML models are data-driven and the training process is based on ground truth data (ground truth labels), as long as ground truth data can be obtained through other methods, the performance of the model can be measured by comparing the difference between the ground truth data and the output data of the AI / ML model using certain metrics.

[0008] However, AI / ML models used for positioning are quite specialized. The label data required for training can only be obtained through offline experimental equipment or simulation software. This data includes the known location of the UE, the time of arrival (TOA) between the UE and the gNB, or the loss of sight (LOS) / non-losing (NLOS) information between links. Therefore, this label data is only useful during model construction. Once deployed in a live communication network, this data cannot be obtained online, making conventional model detection methods unsuitable for wireless positioning.

[0009] In summary, a model monitoring mechanism is needed for wireless communication positioning. The wireless positioning process defined in the current 3GPP protocol does not involve the concepts related to AI / ML models, so the series of model monitoring processes are not clearly defined in the current protocol.

[0010] To address at least one of the above-mentioned problems, an embodiment of the present application provides a method and device for monitoring an AI / ML model, which enables data interaction and model monitoring between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model and obtaining more accurate positioning results.

[0011] According to one aspect of an embodiment of the present application, a method for monitoring an AI / ML model is provided, including:

[0012] The model monitoring device receives the model monitoring request sent by the model result output device;

[0013] The model monitoring device sends request information for requesting model information to the model deployment device;

[0014] The model monitoring device receives the model information fed back by the model deployment device.

[0015] According to another aspect of an embodiment of the present application, a model monitoring device is provided, comprising:

[0016] a first receiving unit, configured to receive a model monitoring request sent by the model result output device;

[0017] a first sending unit, configured to send request information for requesting model information to the model deployment device;

[0018] The first receiving unit also receives model information fed back by the model deployment device.

[0019] According to another aspect of an embodiment of the present application, a method for monitoring an AI / ML model is provided, including:

[0020] The model result output device sends a model monitoring request to the model monitoring device;

[0021] The model monitoring device sends request information for requesting model information to the model deployment device, and receives model information fed back by the model deployment device.

[0022] According to another aspect of an embodiment of the present application, a model result output device is provided, comprising:

[0023] a second sending unit, configured to send a model monitoring request to the model monitoring device;

[0024] The model monitoring device sends request information for requesting model information to the model deployment device, and receives model information fed back by the model deployment device.

[0025] According to another aspect of an embodiment of the present application, a method for monitoring an AI / ML model is provided, including:

[0026] The model deployment device receives request information for requesting model information from the model monitoring device;

[0027] The model deployment device feeds back model information to the model monitoring device.

[0028] According to another aspect of an embodiment of the present application, a model deployment device is provided, including:

[0029] a third receiving unit configured to receive request information for requesting model information from the model monitoring device;

[0030] A third sending unit is configured to feed back model information to the model monitoring device.

[0031] One of the beneficial effects of the embodiments of the present application is that the model monitoring device receives a model monitoring request sent by the model result output device, sends a request message for model information to the model deployment device, and receives the model information fed back by the model deployment device. As a result, real-time data collection and model monitoring can be performed between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model. The AI / ML model used for wireless positioning has better performance and / or better generalization, thereby achieving more accurate positioning results.

[0032] With reference to the following description and accompanying drawings, specific embodiments of the present application are disclosed in detail, indicating the manner in which the principles of the present application can be employed. It should be understood that the embodiments of the present application are not limited in scope. Within the spirit and scope of the appended claims, the embodiments of the present application include many variations, modifications and equivalents.

[0033] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0034] It should be emphasized that the term "include / comprising" when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The elements and features described in one figure or one embodiment of the present application can be combined with the elements and features shown in one or more other figures or embodiments. In addition, in the accompanying drawings, similar reference numerals represent corresponding parts in several figures and can be used to indicate corresponding parts used in more than one embodiment.

[0036] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0037] FIG1 is a schematic diagram of an application scenario of an embodiment of the present application;

[0038] FIG2 is a schematic diagram of a monitoring method for an AI / ML model according to an embodiment of the present application;

[0039] FIG3 is another schematic diagram of the monitoring method of the AI / ML model according to an embodiment of the present application;

[0040] FIG4 is another schematic diagram of the monitoring method of the AI / ML model according to an embodiment of the present application;

[0041] FIG5 is another schematic diagram of the monitoring method of the AI / ML model according to an embodiment of the present application;

[0042] FIG6 is another schematic diagram of the monitoring method of the AI / ML model according to an embodiment of the present application;

[0043] FIG7 is a schematic diagram of a model monitoring device according to an embodiment of the present application;

[0044] FIG8 is a schematic diagram of a model result output device according to an embodiment of the present application;

[0045] FIG9 is a schematic diagram of a model deployment device according to an embodiment of the present application;

[0046] FIG10 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] The above and other features of the present application will become apparent through the following description with reference to the accompanying drawings. In the description and the accompanying drawings, specific embodiments of the present application are disclosed in detail, which illustrate some embodiments in which the principles of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the present application includes all modifications, variations and equivalents that fall within the scope of the appended claims.

[0048] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements from the name, but do not indicate the spatial arrangement or temporal order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0049] In the embodiments of this application, the singular forms "a," "the," etc. include plural forms and should be broadly understood to mean "a" or "a type" rather than being limited to "one." Furthermore, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. Furthermore, the term "according to" should be understood to mean "at least in part based on...", and the term "based on" should be understood to mean "at least in part based on...", unless the context clearly indicates otherwise.

[0050] In the embodiments of the present application, the term "communication network" or "wireless communication network" may refer to a network that complies with any of the following communication standards, such as Long Term Evolution (LTE), enhanced Long Term Evolution (LTE-A, LTE-Advanced), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Packet Access (HSPA), etc.

[0051] Furthermore, communication between devices in the communication system may be carried out according to communication protocols of any stage, for example, including but not limited to the following communication protocols: 1G (generation), 2G, 2.5G, 2.75G, 3G, 4G, 4.5G and future 5G, New Radio (NR), etc., and / or other currently known or future communication protocols to be developed.

[0052] In the embodiments of the present application, the term "network device" refers to, for example, a device in a communication system that connects a terminal device to the communication network and provides services to the terminal device. Network devices may include, but are not limited to, the following devices: base station (BS), access point (AP), transmission reception point (TRP), broadcast transmitter, mobile management entity (MME), gateway, server, radio network controller (RNC), base station controller (BSC), etc.

[0053] Base stations may include, but are not limited to, NodeB (NB), evolved NodeB (eNodeB or eNB), 5G base stations (gNB), IAB hosts, and the like. They may also include remote radio heads (RRHs), remote radio units (RRUs), relays, or low-power nodes (e.g., femto, pico, etc.). The term "base station" may include some or all of their functions, and each base station may provide communication coverage for a specific geographic area. The term "cell" may refer to a base station and / or its coverage area, depending on the context in which the term is used.

[0054] In the embodiments of the present application, the term "user equipment" (UE) refers to, for example, a device that accesses a communication network through a network device and receives network services, and may also be referred to as "terminal equipment" (TE). Terminal equipment may be fixed or mobile, and may also be referred to as a mobile station (MS), terminal, user, subscriber station (SS), access terminal (AT), station, etc.

[0055] Terminal devices may include, but are not limited to, the following devices: cellular phones, personal digital assistants (PDAs), wireless modems, wireless communication devices, handheld devices, machine-type communication devices, laptop computers, cordless phones, smartphones, smart watches, digital cameras, etc.

[0056] For another example, in scenarios such as the Internet of Things (IoT), the terminal device can also be a machine or device for monitoring or measurement, including but not limited to: machine type communication (MTC) terminal, vehicle-mounted communication terminal, device-to-device (D2D) terminal, machine-to-machine (M2M) terminal, and so on.

[0057] The following describes the scenarios of the embodiments of the present application through examples, but the present application is not limited thereto.

[0058] FIG1 is a schematic diagram of a communication system according to an embodiment of the present application, schematically illustrating a situation using a terminal device and a network device as an example. As shown in FIG1 , a communication system 100 may include a network device 101, a terminal device 102, and a positioning server 103. For simplicity, FIG1 illustrates only one terminal device and one network device as an example, but the embodiments of the present application are not limited thereto.

[0059] In the embodiment of the present application, existing services or future services can be transmitted between the network device 101 and the terminal device 102. For example, these services may include, but are not limited to, enhanced mobile broadband (eMBB), massive machine type communication (mMTC), and ultra-reliable and low-latency communication (URLLC), etc.

[0060] It is worth noting that Figure 1 shows that the terminal device 102 is within the coverage of the network device 101, but the present application is not limited to this. The terminal device 102 may not be within the coverage of the network device 101. In addition, Figure 1 takes the deployment of the positioning server 103 as an example, and the AI ​​model can be run in the positioning server 103 to obtain the positioning result; however, the present application is not limited to this. The positioning server 103 can be deployed in the core network, in the network device 102 (such as a base station), or in the terminal device 103; the embodiments of the present application do not limit these situations.

[0061] In the embodiments of this application, the terminal device to be located may be referred to as the target device, and the function of the positioning server may be referred to as the Location Management Function (LMF). The LMF may be a network entity that locates and manages the terminal, and the location server with the location management function may be referred to as the LMF. To avoid confusion, the terms "LMF" and "location server" are interchangeable. For details on these concepts and positioning, please refer to the relevant art.

[0062] Based on current research, the monitoring (or supervision) methods of wireless communication positioning AI / ML models include: monitoring based on model output (OUTPUT) and monitoring based on model input (INPUT). After the model life cycle begins (marked as model activation), the model monitoring entity needs to select the monitoring method in real time based on model information and environmental information. The embodiments of this application focus on the interaction of this information. In addition, if INPUT is chosen as the model monitoring method, then for the metric calculation and confidence information of INPUT, there is also information that needs to be interacted between the entities.

[0063] In the embodiments of the present application, the model deployment device (also referred to as the model deployment module or model deployment entity) may be a UE, a gNB, or a LMF, or may be a partial function or entity of any of the above devices. The model monitoring device (also referred to as the model monitoring module or model monitoring entity) may be a UE, a gNB, a positioning reference unit (PRU), or a LMF, or may be a partial function or entity of any of the above devices. The model result output device (also referred to as the model result output module or model result output entity) may be a UE, a gNB, or a LMF, or may be a partial function or entity of any of the above devices.

[0064] Furthermore, the aforementioned devices may be combined. For example, a UE may include both a model deployment device and a model result output device, while a model monitoring device is provided in a gNB or LMF. For another example, a gNB may include both a model deployment device and a model monitoring device, while a model result output device is provided in an LMF. This application is not limited thereto.

[0065] Embodiments of the first aspect

[0066] The present application provides an AI / ML model monitoring method, which is described from the perspective of a model monitoring device. The model monitoring device can be a network device (such as a base station), a terminal device (such as a target device, a PRU, or other terminal), or a location server with LMF functionality.

[0067] FIG2 is a schematic diagram of a monitoring method for an AI / ML model according to an embodiment of the present application. As shown in FIG2 , the method includes:

[0068] 201, the model monitoring device receives a model monitoring request sent by the model result output device;

[0069] 202, the model monitoring device sends a request message for requesting model information to the model deployment device;

[0070] 203. The model monitoring device receives the model information fed back by the model deployment device.

[0071] It is worth noting that FIG2 above is merely a schematic illustration of an embodiment of the present application, and the present application is not limited thereto. For example, the execution order of the various operations may be appropriately adjusted, and other operations may be added or some operations may be reduced. Those skilled in the art may make appropriate modifications based on the above description, and are not limited to the description of FIG2 above.

[0072] Thus, the model monitoring device receives a model monitoring request from the model result output device, sends a request message for model information to the model deployment device, and receives the model information fed back by the model deployment device. This enables real-time data collection and model monitoring between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model. The AI / ML model used for wireless positioning has better performance and / or better generalization, thereby achieving more accurate positioning results.

[0073] In some embodiments, the model deployment device may report model information to the model supervision device proactively or upon request.

[0074] In some embodiments, the model monitoring device specifies a reporting period and a maximum reporting resource in the request information, and the model deployment device determines the model information to be fed back within the range of the maximum reporting resource according to the reporting period.

[0075] For example, the model deployment device can semi-autonomously report, with the report content determined by the model deployment module. Because model information is complex and cannot be updated entirely through model identification, the model monitoring device can specify the reporting period and the maximum reporting resources allowed by the entity, taking into account transmission resources. The model deployment device semi-autonomously determines the report content within the specified time and using the specified resources, and can select one or more report items from the IE.

[0076] In some embodiments, the model monitoring device specifies a reporting method and reporting content in the request information, and the model deployment device feeds back the model information according to the reporting method and the reporting content.

[0077] For example, the model deployment device can report passively, that is, the reporting method and reporting content are specified by the model supervision device, and the model deployment device cannot choose by itself and feedback model information according to the specified reporting method and reporting content.

[0078] In some embodiments, the model monitoring device sends the request information periodically, or the model monitoring device sends the request information aperiodically.

[0079] For example, the model monitoring device specifies a reporting period, and the model deployment device periodically reports at a specified time. For another example, the model deployment device reports irregularly and sends model information on demand after receiving the FEEDBACK (the above request information) from the model monitoring device, without specifying a period.

[0080] In some embodiments, the model information includes at least one of the following: data statistical information, delay distribution information, and receiving beam information. The present application is not limited thereto, and the above information can be arbitrarily combined or other information can be included.

[0081] In some embodiments, the data statistical information includes at least one of the following: statistical average data of channel impulse response (CIR), peak data of CIR, statistical average data of reference signal received power (RSRP), peak data of RSRP, statistical average data of RSRPP, peak data of RSRPP, cell identifier corresponding to the device providing input data, and distribution information of the receiving beam corresponding to the input data.

[0082] The above examples illustrate data statistical information, but the present application is not limited thereto and may include other information. Furthermore, the above information may be absolute or relative numerical information; for example, the change in the CIR average relative to the Mth cycle statistic may be used.

[0083] FIG3 is another schematic diagram of the monitoring method of the AI / ML model according to an embodiment of the present application. FIG3 can be executed alone or in combination with FIG2. As shown in FIG3, the method includes:

[0084] 301, the model monitoring device sends a request message for requesting environmental information to the model result output device;

[0085] 302. The model monitoring device receives environmental information fed back by the model result output device.

[0086] In some embodiments, the environmental information includes at least one of the following: signal measurement information and environmental statistical information; the environmental statistical information includes at least one of the following: positioning reference unit (PRU) information, PRU statistical information, non-radio access technology (NON-RAT) information, non-radio access technology (NON-RAT) positioning statistical information, and LOS / NLOS statistical information. The present application is not limited thereto, and the above information may be arbitrarily combined and may include other information.

[0087] Table 1 shows an example of PRU information.

[0088] Table 1

[0089]

[0090] As shown in Table 1, for example, the PRU information may include: PRU ID, PRU positioning information, PRU beam information, etc. The present application is not limited thereto, and for example, other specific content may also be included.

[0091] For another example, the statistical information of the positioning reference unit (PRU) includes at least one of the following: information on the number of available PRUs in the current environment, information on the types of available PRUs in the current environment, information on the measurement content supported by the available PRUs in the current environment, information on the beams corresponding to the available PRUs in the current environment, and information on measurement samples that can be provided by the available PRUs in the current environment.

[0092] Table 2 illustrates an example of NON-RAT information.

[0093] Table 2

[0094]

[0095] As shown in Table 1, for example, the NON-RAT information may include: NON-RAT method information, NON-RAT method delay information, etc. The present application is not limited thereto, and for example, other specific content may also be included.

[0096] For another example, the NON-RAT positioning statistics information includes at least one of the following: positioning method information of available NON-RAT in the current environment, positioning accuracy information of available NON-RAT in the current environment, and content information required to implement NON-RAT.

[0097] For example, the LOS / NLOS statistical information includes at least one of the following: LOS / NLOS ratio information for a cell (per cell), and LOS / NLOS ratio information for a transmit beam (per tx beam).

[0098] As shown in FIG3 , the method may further include:

[0099] 303. The model monitoring device feeds back model monitoring decision information to the model result output device.

[0100] In some embodiments, the model monitoring decision information includes at least one of the following: monitoring type information, metric calculation information corresponding to each monitoring method, input monitoring information, and output monitoring information. The input monitoring information includes at least one of the following: input type, confidence information of model input monitoring, accuracy information of model input monitoring, and metric information of model input monitoring. The present application is not limited to this, and the above information may be arbitrarily combined and may include other information.

[0101] Table 3 shows an example of monitoring type information.

[0102] Table 3

[0103]

[0104] As shown in Table 3, for example, the monitoring type information may include: input monitoring, output monitoring, and others. The present application is not limited thereto, and for example, other specific content may also be included.

[0105] For example, when the monitoring type information is input type, the metric calculation information includes at least one of the following: real-time reporting of model input statistical information, collection cycle information, quality threshold information, monitoring cycle, and monitoring metric.

[0106] Table 4 illustrates an example of metric calculation information.

[0107] Table 4

[0108]

[0109]

[0110] As shown in Table 4, for example, the input monitoring metric information may include: monitoring period, monitoring metric, and others. The present application is not limited thereto, and for example, other specific contents may also be included.

[0111] For another example, when the monitoring type information is of output type, the metric calculation information includes at least one of the following: information required for positioning reference unit (PRU) calculation, information required for NON-RAT positioning mode calculation, PRU information, and NON-RAT information.

[0112] Table 5 illustrates an example of metric calculation information.

[0113] Table 5

[0114]

[0115] As shown in Table 5, for example, the output monitoring metric information may include: PRU information, NON-RAT information, and others. The present application is not limited thereto, and for example, other specific content may also be included.

[0116] Table 6 shows an example of input monitoring information.

[0117] Table 6

[0118]

[0119] As shown in Table 6, for example, the input monitoring information may include: input type, monitoring confidence, monitoring accuracy, and monitoring measurement information. The present application is not limited thereto, and for example, other specific content may also be included.

[0120] For example, the input type (InputType) can be one of L1_RSRP, reference signal received path power (RSRPP), CIR, channel frequency response (CFR), TOA, and TDOA. Monitoring confidence (MonitoringConfidence) represents the confidence of the corresponding monitoring method; for example, the confidence value is in the range of 0 to 1, with a value interval of 0.1, corresponding to a confidence probability of 0%-100%, respectively. The specific statistical method can be implemented according to the actual situation. For another example, monitoring accuracy (MonitoringAccuracy) represents the accuracy corresponding to the monitoring method, with a value in the range of 0 to 10, with a value interval of 1, and the unit is meter.

[0121] It is worth noting that FIG3 above is merely a schematic illustration of an embodiment of the present application, and the present application is not limited thereto. For example, the execution order of the various operations may be appropriately adjusted, and other operations may be added or some operations may be reduced. Those skilled in the art may make appropriate modifications based on the above description, and are not limited to the description of FIG3 above.

[0122] FIG4 is another schematic diagram of the AI / ML model monitoring method according to an embodiment of the present application. FIG4 can be executed alone or in combination with FIG2 or FIG3. As shown in FIG4, the method includes:

[0123] 401. When the model monitoring device is unable to obtain statistical information, the model monitoring device sends model abnormality information to the model result output device.

[0124] The above embodiments are merely exemplary of the present invention, but the present invention is not limited thereto. Appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0125] According to the embodiments of the present application, real-time data collection and model monitoring can be performed between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model. The AI / ML model used for wireless positioning has better performance and / or better generalization, thereby obtaining more accurate positioning results.

[0126] Embodiments of the second aspect

[0127] The embodiment of the present application provides a monitoring method for an AI / ML model, which is described from the perspective of a model result output device. The embodiment of the second aspect corresponds to the embodiment of the first aspect, and the same content is not repeated here.

[0128] FIG5 is another schematic diagram of a method for monitoring an AI / ML model according to an embodiment of the present application. As shown in FIG5 , the method includes:

[0129] 501, the model result output device sends a model monitoring request to the model monitoring device;

[0130] The model monitoring device sends request information for requesting model information to the model deployment device, and receives model information fed back by the model deployment device.

[0131] For example, the model deployment and model result output devices can be located in the UE, while the model monitoring device can be located in the gNB or LMF. In other words, the AI / ML model is direct, with the UE outputting model results locally, while model monitoring is located in the gNB or LMF. The UE can request model identification from the network and initiate the model lifecycle, activating the AI / ML model for positioning. The UE then initiates a model monitoring request to the gNB / LMF, optionally attaching additional information.

[0132] In some embodiments, as shown in FIG5 , the method may further include:

[0133] 502, the model result output device receives request information for requesting environmental information from the model monitoring device; and

[0134] 503. The model result output device feeds back environmental information to the model monitoring device.

[0135] In some embodiments, the environmental information includes at least one of the following: signal measurement information, environmental statistical information;

[0136] The environmental statistical information includes at least one of the following: positioning reference unit (PRU) information, statistical information of positioning reference unit (PRU), NON-RAT information, positioning statistical information of NON-RAT, and LOS / NLOS statistical information.

[0137] In some embodiments, the positioning reference unit (PRU) statistical information includes at least one of the following: number information of available PRUs in the current environment, type information of available PRUs in the current environment, measurement content information supported by available PRUs in the current environment, beam information corresponding to available PRUs in the current environment, and measurement sample information that can be provided by available PRUs in the current environment;

[0138] The NON-RAT positioning statistical information includes at least one of the following: positioning mode information of available NON-RAT in the current environment, positioning accuracy information of available NON-RAT in the current environment, and content information required to implement NON-RAT;

[0139] The LOS / NLOS statistical information includes at least one of the following: LOS / NLOS ratio information for a cell, and LOS / NLOS ratio information for a transmit beam.

[0140] For example, the model deployment and monitoring devices can be located in the gNB, while the model result output device can be located in the LMF. This means that the AI / ML model is indirect, with the LMF outputting the model results. The gNB can initiate model monitoring and send an environmental information request signaling to the LMF. The LMF then sends environmental statistics to the gNB, including PRU statistics, non-RAT positioning mode information, or LOS / NLOS statistics.

[0141] In some embodiments, as shown in FIG5 , the method may further include:

[0142] 504. The model result output device receives the model monitoring decision information fed back by the model monitoring device.

[0143] In some embodiments, the model monitoring decision information includes at least one of the following: monitoring type information, metric calculation information corresponding to each monitoring method, input monitoring information, and output monitoring information.

[0144] In some embodiments, the input monitoring information includes at least one of the following: input type, confidence information of model input monitoring, accuracy information of model input monitoring, and metric information of model input monitoring.

[0145] In some embodiments, when the monitoring type information is of input type, the metric calculation information includes at least one of the following: statistical information of real-time reporting model input, collection cycle information, quality threshold information, monitoring cycle, and monitoring metric; when the monitoring type information is of output type, the metric calculation information includes at least one of the following: information required for positioning reference unit (PRU) calculation, information required for NON-RAT positioning method calculation, positioning reference unit (PRU) information, and NON-RAT information.

[0146] For example, the model deployment and monitoring devices can be located in the gNB, while the model result output device can be located in the LMF. This means that the AI / ML model is indirect, with the LMF outputting the model results. Based on the collected comprehensive information, the gNB can send monitoring mode information to the LMF. This information may include: supervision type (input, output, or other); and accompanying information required for metric calculations corresponding to the supervision type. For example, if the supervision type is input, real-time reporting of model input statistics, collection period, and quality thresholds; if the supervision type is output, additional information required for PRU or NON-RAT calculations can be reported.

[0147] For another example, the model deployment and monitoring devices can be located in the gNB, while the model result output device can be located in the LMF. That is, the AI / ML model is indirect, with the LMF outputting the model results. Based on the collected comprehensive information, the gNB can send monitoring method information to the LMF, which may include confidence information and / or accuracy information.

[0148] In some embodiments, the model result output device receives model anomaly information sent by the model monitoring device when statistical information cannot be obtained.

[0149] It is worth noting that FIG5 above is merely a schematic illustration of an embodiment of the present application, and the present application is not limited thereto. For example, the execution order of the various operations may be appropriately adjusted, and other operations may be added or some operations may be reduced. Those skilled in the art may make appropriate modifications based on the above description, and are not limited to the description of FIG5 above.

[0150] The above embodiments are merely exemplary of the present invention, but the present invention is not limited thereto. Appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0151] According to the embodiments of the present application, real-time data collection and model monitoring can be performed between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model. The AI / ML model used for wireless positioning has better performance and / or better generalization, thereby obtaining more accurate positioning results.

[0152] Embodiments of the third aspect

[0153] The embodiment of the present application provides a monitoring method for an AI / ML model, which is described from the perspective of a model deployment device. The embodiment of the third aspect corresponds to the embodiment of the first aspect, and the same content is not repeated here.

[0154] FIG6 is another schematic diagram of a method for monitoring an AI / ML model according to an embodiment of the present application. As shown in FIG6 , the method includes:

[0155] 601, the model deployment device receives request information for requesting model information from the model monitoring device;

[0156] 602. The model deployment device feeds back model information to the model monitoring device.

[0157] In some embodiments, the model monitoring device specifies a reporting period and a maximum reporting resource in the request information, and the model deployment device determines the model information to be fed back within the range of the maximum reporting resource according to the reporting period.

[0158] In some embodiments, the model monitoring device specifies a reporting method and reporting content in the request information, and the model deployment device feeds back the model information according to the reporting method and the reporting content.

[0159] In some embodiments, the model monitoring device sends the request information periodically, or the model monitoring device sends the request information aperiodically.

[0160] In some embodiments, the model information includes at least one of the following: data statistical information, delay distribution information, and receive beam information. The data statistical information includes at least one of the following: statistical average data of CIR, peak data of CIR, statistical average data of RSRP, peak data of RSRP, statistical average data of RSRPP, peak data of RSRPP, a cell identifier corresponding to a device providing input data, and distribution information of a receive beam corresponding to the input data.

[0161] For example, the model deployment and model result output devices can be located in the UE, while the model monitoring device can be located in the gNB or LMF. In other words, the AI / ML model is direct, with the UE outputting model results locally, while model monitoring is located in the gNB or LMF. The gNB / LMF can send a request message to the UE requesting model information, optionally specifying additional information. The UE can then send model information, including model training data statistics such as RSRP and RSRPP, and information about the UE's current received BEAM.

[0162] It is worth noting that FIG6 above is merely a schematic illustration of an embodiment of the present application, and the present application is not limited thereto. For example, the execution order of the various operations may be appropriately adjusted, and other operations may be added or some operations may be reduced. Those skilled in the art may make appropriate modifications based on the above description, and are not limited to the description of FIG6 above.

[0163] The above embodiments are merely exemplary of the present invention, but the present invention is not limited thereto. Appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0164] According to the embodiments of the present application, real-time data collection and model monitoring can be performed between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model. The AI / ML model used for wireless positioning has better performance and / or better generalization, thereby obtaining more accurate positioning results.

[0165] Embodiments of the fourth aspect

[0166] The embodiment of the present application provides a model monitoring device. Since the principle of solving the problem of the model monitoring device is the same as the method of the embodiment of the first aspect, its specific implementation can refer to the embodiment of the first aspect, and the same content will not be repeated.

[0167] FIG7 is a schematic diagram of a model monitoring device according to an embodiment of the present application. As shown in FIG7 , the model monitoring device 700 according to the embodiment of the present application includes:

[0168] A first receiving unit 701 receives a model monitoring request sent by a model result output device;

[0169] A first sending unit 702 sends request information for requesting model information to the model deployment device;

[0170] The first receiving unit 701 also receives model information fed back by the model deployment device.

[0171] In some embodiments, the first sending unit 702 specifies a reporting period and a maximum reporting resource in the request information, and the model deployment device determines the feedback model information within the range of the maximum reporting resource according to the reporting period.

[0172] In some embodiments, the first sending unit 702 specifies a reporting method and reporting content in the request information, and the model deployment device feeds back the model information according to the reporting method and the reporting content.

[0173] In some embodiments, the first sending unit 702 sends the request information periodically, or the first sending unit 702 sends the request information aperiodically.

[0174] In some embodiments, the model information includes at least one of the following: data statistical information, delay distribution information, and receiving beam information.

[0175] In some embodiments, the data statistical information includes at least one of the following: statistical average data of CIR, peak data of CIR, statistical average data of RSRP, peak data of RSRP, statistical average data of RSRPP, peak data of RSRPP, cell identifier corresponding to the device providing input data, and distribution information of the receiving beam corresponding to the input data.

[0176] In some embodiments, the first sending unit 702 further sends request information for requesting environmental information to the model result output device; and the first receiving unit 701 further receives environmental information fed back by the model result output device.

[0177] In some embodiments, the environmental information includes at least one of the following: signal measurement information, environmental statistical information.

[0178] In some embodiments, the environmental statistical information includes at least one of the following: positioning reference unit (PRU) information, statistical information of the positioning reference unit (PRU), NON-RAT information, positioning statistical information of NON-RAT, and LOS / NLOS statistical information.

[0179] In some embodiments, the statistical information of the positioning reference unit (PRU) includes at least one of the following: number information of available PRUs in the current environment, type information of available PRUs in the current environment, measurement content information supported by available PRUs in the current environment, beam information corresponding to available PRUs in the current environment, and measurement sample information that can be provided by available PRUs in the current environment;

[0180] The NON-RAT positioning statistical information includes at least one of the following: positioning method information of available NON-RAT in the current environment, positioning accuracy information of available NON-RAT in the current environment, and content information required to implement NON-RAT;

[0181] The LOS / NLOS statistical information includes at least one of the following: LOS / NLOS ratio information for a cell, and LOS / NLOS ratio information for a transmit beam.

[0182] In some embodiments, the first sending unit 702 further feeds back model monitoring decision information to the model result output device.

[0183] In some embodiments, the model monitoring decision information includes at least one of the following: monitoring type information, metric calculation information corresponding to each monitoring method, input monitoring information, and output monitoring information.

[0184] In some embodiments, the input monitoring information includes at least one of the following: input type, confidence information of model input monitoring, accuracy information of model input monitoring, and metric information of model input monitoring.

[0185] In some embodiments, when the monitoring type information is input type, the metric calculation information includes at least one of the following: real-time reporting of model input statistical information, collection cycle information, quality threshold information, monitoring cycle, and monitoring metric.

[0186] In some embodiments, when the monitoring type information is of output type, the metric calculation information includes at least one of the following: information required for positioning reference unit (PRU) calculation, information required for NON-RAT positioning mode calculation, positioning reference unit (PRU) information, and NON-RAT information.

[0187] In some embodiments, the first sending unit 702 further sends model anomaly information to the model result output device when statistical information cannot be obtained.

[0188] In addition, for the sake of simplicity, FIG7 only illustrates the connection relationship or signal direction between various components or modules. However, it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors, memories, transmitters, and receivers; the implementation of this application is not limited to this.

[0189] The above embodiments are merely exemplary of the present invention, but the present invention is not limited thereto. Appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0190] According to the embodiments of the present application, real-time data collection and model monitoring can be performed between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model. The AI / ML model used for wireless positioning has better performance and / or better generalization, thereby obtaining more accurate positioning results.

[0191] Embodiments of the fifth aspect

[0192] The embodiment of the present application provides a model result output device. Since the principle of solving the problem of the model result output device is the same as the method of the embodiment of the second aspect, its specific implementation can refer to the embodiments of the first and second aspects, and the same content will not be repeated.

[0193] FIG8 is a schematic diagram of a model result output device according to an embodiment of the present application. As shown in FIG8 , the model result output device 800 according to an embodiment of the present application includes:

[0194] A second sending unit 801 sends a model monitoring request to the model monitoring device;

[0195] The model monitoring device sends request information for requesting model information to the model deployment device, and receives model information fed back by the model deployment device.

[0196] In some embodiments, as shown in FIG8 , the model result output device 800 may further include:

[0197] A second receiving unit 802 receives request information for requesting environmental information from the model monitoring device;

[0198] The second sending unit 801 also feeds back environmental information to the model monitoring device.

[0199] In some embodiments, the second receiving unit 802 further receives model monitoring decision information fed back by the model monitoring device.

[0200] In some embodiments, the second receiving unit 802 further receives model anomaly information sent by the model monitoring device when statistical information cannot be obtained.

[0201] In addition, for the sake of simplicity, FIG8 only illustrates the connection relationship or signal direction between various components or modules. However, it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors, memories, transmitters, and receivers; the implementation of this application is not limited to this.

[0202] The above embodiments are merely exemplary of the present invention, but the present invention is not limited thereto. Appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0203] According to the embodiments of the present application, real-time data collection and model monitoring can be performed between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model. The AI / ML model used for wireless positioning has better performance and / or better generalization, thereby obtaining more accurate positioning results.

[0204] Embodiments of the sixth aspect

[0205] The present application provides a model deployment device. Since the principle of solving the problem of the model deployment device is the same as the method of the embodiment of the third aspect, its specific implementation can refer to the embodiments of the first to third aspects, and the same content will not be repeated here.

[0206] FIG9 is a schematic diagram of a model deployment apparatus according to an embodiment of the present application. As shown in FIG9 , the model deployment apparatus 900 according to an embodiment of the present application includes:

[0207] A third receiving unit 901 receives request information for requesting model information from the model monitoring device;

[0208] The third sending unit 902 is configured to feed back model information to the model monitoring device.

[0209] In addition, for the sake of simplicity, FIG9 only illustrates the connection relationship or signal direction between various components or modules. However, it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors, memories, transmitters, and receivers; the implementation of this application is not limited to this.

[0210] The above embodiments are merely exemplary of the present invention, but the present invention is not limited thereto. Appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0211] According to the embodiments of the present application, real-time data collection and model monitoring can be performed between network entities involved in positioning and / or between network entities and terminals, thereby optimizing the wireless positioning AI / ML model. The AI / ML model used for wireless positioning has better performance and / or better generalization, thereby obtaining more accurate positioning results.

[0212] Embodiments of the seventh aspect

[0213] An embodiment of the present application provides a communication system. Figure 1 is a schematic diagram of the communication system of the embodiment of the present application. As shown in Figure 1, the communication system 100 includes a network device 101, a terminal device 102 and a positioning server 103. For simplicity, Figure 1 only uses one network device and one terminal device as an example for illustration, but the embodiment of the present application is not limited to this.

[0214] In some embodiments, the communication system includes: a model monitoring device 700; a model result output device 800; and a model deployment device 900.

[0215] An embodiment of the present application also provides an electronic device, which is, for example, the aforementioned model monitoring device, or model result output device, or model deployment device.

[0216] Figure 10 is a schematic diagram of the electronic device according to an embodiment of the present application. As shown in Figure 10 , electronic device 1000 may include a processor 1010 (e.g., a central processing unit (CPU)) and a memory 1020; the memory 1020 is coupled to the processor 1010. The memory 1020 may store various data and may also store an information processing program 1030, which is executed under the control of the processor 1010.

[0217] For example, the processor 1010 may be configured to execute a program to implement the AI / ML model monitoring method as described in the embodiment of the first aspect. For example, the processor 1010 may be configured to perform the following control: receiving a model monitoring request sent by a model result output device; sending request information for requesting model information to a model deployment device; and receiving model information fed back by the model deployment device.

[0218] For another example, the processor 1010 may be configured to execute a program to implement the AI / ML model monitoring method as described in the embodiment of the second aspect. For example, the processor 1010 may be configured to perform the following control: sending a model monitoring request to a model monitoring device; wherein the model monitoring device sends a request message for requesting model information to a model deployment device, and receives model information fed back by the model deployment device.

[0219] For another example, the processor 1010 may be configured to execute a program to implement the AI / ML model monitoring method as described in the embodiment of the third aspect. For example, the processor 1010 may be configured to perform the following control: receiving request information for requesting model information from the model monitoring device; and feeding back the model information to the model monitoring device.

[0220] In addition, as shown in FIG10 , the electronic device 1000 may further include: a transceiver 1040 and an antenna 1050; wherein, the functions of the above components are similar to those in the prior art and are not described in detail here. It is worth noting that the electronic device 1000 does not necessarily include all the components shown in FIG10 ; in addition, the electronic device 1000 may also include components not shown in FIG10 , and reference may be made to the prior art for details.

[0221] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in a model monitoring device, the program causes a computer to execute the AI / ML model monitoring method described in the embodiment of the first aspect in the model monitoring device.

[0222] An embodiment of the present application also provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the AI / ML model monitoring method described in the embodiment of the first aspect in a model monitoring device.

[0223] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in a model result output device, the program causes the computer to execute the AI / ML model monitoring method described in the embodiment of the second aspect in the model result output device.

[0224] An embodiment of the present application also provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the AI / ML model monitoring method described in the embodiment of the second aspect in a model result output device.

[0225] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in a model deployment device, the program causes the computer to execute the AI / ML model monitoring method described in the embodiment of the third aspect in the model deployment device.

[0226] An embodiment of the present application also provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the AI / ML model monitoring method described in the embodiment of the third aspect in a model deployment device.

[0227] The above devices and methods of the present application can be implemented by hardware or by a combination of hardware and software. The present application relates to such a computer-readable program that, when executed by a logic component, enables the logic component to implement the devices or components described above, or enables the logic component to implement the various methods or steps described above. The logic component is, for example, a field programmable logic component, a microprocessor, a processor used in a computer, etc. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0228] The method / device described in conjunction with the embodiments of the present application can be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figure and / or one or more combinations of functional block diagrams can correspond to various software modules of the computer program flow or to various hardware modules. These software modules can respectively correspond to the various steps shown in the figure. These hardware modules can be implemented by solidifying these software modules, for example, using a field programmable gate array (FPGA).

[0229] The software module may be located in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in the memory of the mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the device (such as a mobile terminal) uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0230] One or more of the functional blocks and / or one or more combinations of functional blocks described in the accompanying drawings may be implemented as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any appropriate combination thereof for performing the functions described in this application. One or more of the functional blocks and / or one or more combinations of functional blocks described in the accompanying drawings may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0231] The present application has been described above in conjunction with specific embodiments. However, those skilled in the art should understand that these descriptions are merely illustrative and are not intended to limit the scope of protection of the present application. Those skilled in the art may make various modifications and variations to the present application based on the spirit and principles of the present application, and such modifications and variations are also within the scope of the present application.

[0232] Regarding the above implementation methods disclosed in this embodiment, the following additional notes are also disclosed:

[0233] 1. A method for monitoring an AI / ML model, comprising:

[0234] The model monitoring device receives the model monitoring request sent by the model result output device;

[0235] The model monitoring device sends request information for requesting model information to the model deployment device;

[0236] The model monitoring device receives the model information fed back by the model deployment device.

[0237] 2. The method according to Note 1, wherein the model monitoring device specifies a reporting period and a maximum reporting resource in the request information, and the model deployment device determines the feedback model information within the range of the maximum reporting resource according to the reporting period.

[0238] 3. The method according to Note 1, wherein the model monitoring device specifies a reporting method and reporting content in the request information, and the model deployment device feeds back the model information according to the reporting method and the reporting content.

[0239] 4. The method according to Note 1, wherein the model monitoring device sends the request information periodically, or the model monitoring device sends the request information aperiodically.

[0240] 5. The method according to any one of Notes 1 to 4, wherein the model information includes at least one of the following: data statistical information, delay distribution information, and receiving beam information.

[0241] 6. The method according to Note 5, wherein the data statistical information includes at least one of the following: statistical average data of CIR, peak data of CIR, statistical average data of RSRP, peak data of RSRP, statistical average data of RSRPP, peak data of RSRPP, and the cell identifier corresponding to the device providing input data and the distribution information of the receiving beam corresponding to the input data.

[0242] 7. The method according to any one of Notes 1 to 6, wherein the method further comprises:

[0243] The model monitoring device sends a request message for requesting environmental information to the model result output device;

[0244] The model monitoring device receives environmental information fed back by the model result output device.

[0245] 8. The method according to Note 7, wherein the environmental information includes at least one of the following: signal measurement information, environmental statistical information.

[0246] 9. The method according to Note 8, wherein the environmental statistical information includes at least one of the following: positioning reference unit (PRU) information, statistical information of the positioning reference unit (PRU), NON-RAT information, positioning statistical information of NON-RAT, and LOS / NLOS statistical information.

[0247] 10. The method according to Supplementary Note 9, wherein:

[0248] The statistical information of the positioning reference unit (PRU) includes at least one of the following: information on the number of available PRUs in the current environment, information on the types of available PRUs in the current environment, information on the measurement content supported by the available PRUs in the current environment, information on the beams corresponding to the available PRUs in the current environment, and information on the measurement samples that can be provided by the available PRUs in the current environment.

[0249] 11. The method according to Supplementary Note 9, wherein:

[0250] The NON-RAT positioning statistics information includes at least one of the following: positioning mode information of available NON-RAT in the current environment, positioning accuracy information of available NON-RAT in the current environment, and content information required to implement NON-RAT.

[0251] 12. The method according to Supplementary Note 9, wherein:

[0252] The LOS / NLOS statistical information includes at least one of the following: LOS / NLOS ratio information for a cell, and LOS / NLOS ratio information for a transmit beam.

[0253] 13. The method according to any one of Notes 1 to 12, wherein the method further comprises:

[0254] The model monitoring device feeds back model monitoring decision information to the model result output device.

[0255] 14. The method according to note 13, wherein the model monitoring decision information includes at least one of the following: monitoring type information, metric calculation information corresponding to each monitoring method, model input information, and model output information;

[0256] The model input information includes at least one of the following: confidence information of model input monitoring, accuracy information of model input monitoring, and measurement information of model input monitoring.

[0257] 15. The method according to Note 14, wherein, when the monitoring type information is input type, the metric calculation information includes at least one of the following: real-time reporting of statistical information of model input, collection cycle information, quality threshold information, monitoring cycle, and monitoring metric.

[0258] 16. The method according to Note 14, wherein, when the monitoring type information is of output type, the metric calculation information includes at least one of the following: information required for positioning reference unit (PRU) calculation, information required for NON-RAT positioning method calculation, positioning reference unit (PRU) information, and NON-RAT information.

[0259] 17. The method according to any one of Notes 1 to 16, wherein the method further comprises:

[0260] When the model monitoring device cannot obtain statistical information, it sends model abnormality information to the model result output device.

[0261] 18. A method for monitoring an AI / ML model, comprising:

[0262] The model result output device sends a model monitoring request to the model monitoring device;

[0263] The model monitoring device sends request information for requesting model information to the model deployment device, and receives model information fed back by the model deployment device.

[0264] 19. The method according to Supplementary Note 18, wherein the method further comprises:

[0265] The model result output device receives request information for requesting environmental information from the model monitoring device; and

[0266] The model result output device feeds back environmental information to the model monitoring device.

[0267] 20. The method according to Note 19, wherein the environmental information includes at least one of the following: signal measurement information, environmental statistical information.

[0268] 21. The method according to Note 20, wherein the environmental statistical information includes at least one of the following: positioning reference unit (PRU) information, statistical information of the positioning reference unit (PRU), NON-RAT information, positioning statistical information of NON-RAT, and LOS / NLOS statistical information.

[0269] 22. The method according to Supplementary Note 21, wherein:

[0270] The positioning reference unit (PRU) statistical information includes at least one of the following: information on the number of available PRUs in the current environment, information on the types of available PRUs in the current environment, information on the measurement content supported by the available PRUs in the current environment, information on the beams corresponding to the available PRUs in the current environment, and information on measurement samples that can be provided by the available PRUs in the current environment.

[0271] 23. The method according to Supplementary Note 21, wherein:

[0272] The NON-RAT positioning statistical information includes at least one of the following: positioning mode information of available NON-RAT in the current environment, positioning accuracy information of available NON-RAT in the current environment, and content information required to implement NON-RAT.

[0273] 24. The method according to Supplement 21, wherein:

[0274] The LOS / NLOS statistical information includes at least one of the following: LOS / NLOS ratio information for a cell, and LOS / NLOS ratio information for a transmit beam.

[0275] 25. The method according to any one of Notes 18 to 24, wherein the method further comprises:

[0276] The model result output device receives the model monitoring decision information fed back by the model monitoring device.

[0277] 26. The method according to note 25, wherein the model monitoring decision information includes at least one of the following: monitoring type information, metric calculation information corresponding to each monitoring method, model input information, and model output information;

[0278] The model input information includes at least one of the following: confidence information of model input monitoring, accuracy information of model input monitoring, and measurement information of model input monitoring.

[0279] 27. The method according to Note 26, wherein, when the monitoring type information is input type, the metric calculation information includes at least one of the following: real-time reporting of statistical information of model input, collection cycle information, quality threshold information, monitoring cycle, and monitoring metric.

[0280] 28. The method according to Note 26, wherein, when the monitoring type information is of output type, the metric calculation information includes at least one of the following: information required for positioning reference unit (PRU) calculation, information required for NON-RAT positioning method calculation, positioning reference unit (PRU) information, and NON-RAT information.

[0281] 29. The method according to any one of Notes 18 to 28, wherein the method further comprises:

[0282] The model result output device receives the model abnormality information sent by the model monitoring device when the statistical information cannot be obtained.

[0283] 30. A method for monitoring an AI / ML model, comprising:

[0284] The model deployment device receives request information for requesting model information from the model monitoring device;

[0285] The model deployment device feeds back model information to the model monitoring device.

[0286] 31. The method according to Note 30, wherein the model monitoring device specifies a reporting period and a maximum reporting resource in the request information, and the model deployment device determines the feedback model information within the range of the maximum reporting resource based on the reporting period.

[0287] 32. The method according to Note 30, wherein the model monitoring device specifies a reporting method and reporting content in the request information, and the model deployment device feeds back the model information according to the reporting method and the reporting content.

[0288] 33. The method according to Note 30, wherein the model monitoring device sends the request information periodically, or the model monitoring device sends the request information non-periodically.

[0289] 34. The method according to any one of Notes 30 to 33, wherein the model information includes at least one of the following: data statistical information, delay distribution information, and receiving beam information.

[0290] 35. The method according to Note 34, wherein the data statistical information includes at least one of the following: statistical average data of CIR, peak data of CIR, statistical average data of RSRP, peak data of RSRP, statistical average data of RSRPP, peak data of RSRPP, and the cell identifier corresponding to the device providing input data and the distribution information of the receiving beam corresponding to the input data.

[0291] 36. A model monitoring device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the AI / ML model monitoring method as described in any one of Notes 1 to 17.

[0292] 37. A model result output device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the AI / ML model monitoring method as described in any one of Notes 18 to 29.

[0293] 38. A model deployment device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the AI / ML model monitoring method as described in any one of Notes 30 to 35.

[0294] 39. A communication system comprising:

[0295] A model monitoring device as described in Supplementary Note 36;

[0296] The model result output device as described in Supplementary Note 37; and

[0297] A model deployment device as described in Note 38.

Claims

1. A model monitoring device, comprising: A first receiving unit, which receives a model monitoring request sent by the model result output device; A first sending unit, which sends request information for requesting model information to the model deployment device; The first receiving unit also receives model information fed back by the model deployment device.

2. The device according to claim 1, wherein: The first sending unit specifies a reporting period and a maximum reporting resource in the request information, and the model deployment device determines the feedback model information within the range of the maximum reporting resource according to the reporting period.

3. The device according to claim 1, wherein: The first sending unit specifies a reporting method and reporting content in the request information, and the model deployment device feeds back the model information according to the reporting method and the reporting content. 4 . The device according to claim 1 , wherein the first sending unit sends the request information periodically, or the first sending unit sends the request information aperiodically.

5. The device according to claim 1, wherein: The model information includes at least one of the following: data statistical information, delay distribution information, and receiving beam information.

6. The device according to claim 5, wherein: The data statistical information includes at least one of the following: statistical average data of channel impulse response, peak data of channel impulse response, statistical average data of reference signal received power, peak data of reference signal received power, statistical average data of reference signal received path power, peak data of reference signal received path power, cell identifier corresponding to the device providing input data, and distribution information of the receiving beam corresponding to the input data.

7. The device according to claim 1, wherein: The first sending unit also sends request information for requesting environmental information to the model result output device; The first receiving unit also receives environmental information fed back by the model result output device.

8. The device according to claim 7, wherein: The environmental information includes at least one of the following: signal measurement information and environmental statistical information.

9. The device according to claim 8, wherein the environmental statistical information comprises at least one of the following: positioning reference unit information, statistical information of the positioning reference unit, non-radio intervention technology information, positioning statistical information of non-radio intervention technology, and line-of-sight / non-line-of-sight statistical information.

10. The device according to claim 9, wherein: The statistical information of the positioning reference unit includes at least one of the following: number information of available positioning reference units in the current environment, type information of available positioning reference units in the current environment, measurement content information supported by available positioning reference units in the current environment, beam information corresponding to available positioning reference units in the current environment, and measurement sample information that can be provided by available positioning reference units in the current environment; The positioning statistics information of the non-radio intervention technology includes at least one of the following: positioning mode information of the non-radio intervention technology available in the current environment, positioning accuracy information of the non-radio intervention technology available in the current environment, and content information required to implement the non-radio intervention technology; The line-of-sight / non-line-of-sight statistical information includes at least one of the following: line-of-sight / non-line-of-sight ratio information for a cell, and line-of-sight / non-line-of-sight ratio information for a transmit beam.

11. The device according to claim 1, wherein: The first sending unit also feeds back model monitoring decision information to the model result output device.

12. The device according to claim 11, wherein The model monitoring decision information includes at least one of the following: monitoring type information, metric calculation information corresponding to each monitoring method, model input information, and model output information; The model input information includes at least one of the following: confidence information of model input monitoring, accuracy information of model input monitoring, and measurement information of model input monitoring.

13. The device according to claim 12, wherein: In the case where the monitoring type information is of input type, the metric calculation information includes at least one of the following: real-time reporting of statistical information of model input, collection cycle information, quality threshold information, monitoring cycle, and monitoring metric.

14. The device according to claim 12, wherein: When the monitoring type information is of output type, the metric calculation information includes at least one of the following: information required for positioning reference unit calculation, information required for non-radio intervention technology positioning method calculation, positioning reference unit information, and non-radio intervention technology information.

15. The device according to claim 1, wherein: The first sending unit also sends model abnormality information to the model result output device when the statistical information cannot be obtained.

16. A model result output device, comprising: a second sending unit, which sends a model monitoring request to the model monitoring device; The model monitoring device sends request information for requesting model information to the model deployment device, and receives model information fed back by the model deployment device.

17. The device according to claim 16, wherein: The device also includes: a second receiving unit, which receives request information for requesting environmental information from the model monitoring device; The second sending unit also feeds back environmental information to the model monitoring device.

18. The device according to claim 17, wherein: The second receiving unit also receives model monitoring decision information fed back by the model monitoring device.

19. The device according to claim 17, wherein: The second receiving unit also receives model abnormality information sent by the model monitoring device when statistical information cannot be obtained.

20. A model deployment device, comprising: a third receiving unit, which receives request information for requesting model information from the model monitoring device; A third sending unit is configured to feed back model information to the model monitoring device.

Images