AI model supervision method and device, equipment, storage medium and program product

Abstract

The invention discloses an AI model supervision method and device, equipment, a storage medium and a program product, and relates to the field of communication, the method comprises the steps that first information is acquired, the first information is used for assisting supervision of a first AI model, and the first AI model is deployed on first equipment; and executing a model supervision decision based on the first information. According to the scheme of the invention, the supervision decision is executed based on the obtained first information used for assisting the supervision of the first AI model, a set of complete and specific model supervision scheme is provided, the supervision of the first AI model is realized, and thus the problem that the communication process is affected by the abnormal performance of the first AI model is solved.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to an AI model supervision method, apparatus, device, storage medium, and program product. Background Technology

[0002] In research on the integration of Artificial Intelligence (AI) / Machine Learning (ML) with air interface technology, AI technology can solve some complex problems in communication by deploying AI models with corresponding functions in terminal or network-side devices. However, the applicability of AI models is limited. For example, a model may achieve good performance in scenario A, but its performance may drop sharply in scenario B, which differs significantly from scenario A, thus affecting the communication process. Therefore, supervising AI models is a crucial aspect of AI / ML integration with air interface technology; however, existing AI-air interface integration technologies lack a complete and specific model supervision solution. Summary of the Invention

[0003] This invention provides an AI model supervision method, apparatus, device, storage medium, and program product, which solves the problem that the performance differences of AI models affect the communication process.

[0004] In a first aspect, embodiments of this application provide a method for supervising an artificial intelligence (AI) model, applied to a first device, comprising:

[0005] Obtain first information, which is used to assist in the supervision of the first AI model, which is deployed on the first device;

[0006] Based on the first information, a model-supervised decision is executed.

[0007] Secondly, embodiments of this application provide a method for supervising an AI model, applied to a second device, comprising:

[0008] Send first information, which is used to assist in the supervision of the first AI model, which is deployed on the first device.

[0009] Thirdly, embodiments of this application provide a supervision device for an artificial intelligence (AI) model, applied to a first device, the device comprising:

[0010] An acquisition module is used to acquire first information, which is used to assist in the supervision of the first AI model, which is deployed on the first device.

[0011] The execution module is used to perform model-supervised decisions based on the first information.

[0012] Fourthly, embodiments of this application provide a supervision device for an AI model, applied to a second device, the device comprising:

[0013] The first sending module is used to send first information, which is used to assist in the supervision of the first AI model, which is deployed on the first device.

[0014] Fifthly, embodiments of this application provide an AI model supervision device, including a transceiver, a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the AI ​​model supervision method as described in the first aspect, or implements the AI ​​model supervision method as described in the second aspect.

[0015] Sixthly, embodiments of this application provide a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the AI ​​model supervision method as described in the first aspect, or implements the AI ​​model supervision method as described in the second aspect.

[0016] In a seventh aspect, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the AI ​​model supervision method as described in the first aspect, or implement the AI ​​model supervision method as described in the second aspect.

[0017] The beneficial effects of the above-mentioned technical solution of this application are:

[0018] In the scheme of this application embodiment, firstly, first information is obtained, which is used to assist in the supervision of a first AI model deployed on the first device; secondly, based on the first information, a model supervision decision is executed. Thus, the embodiments of this application provide a complete and specific model supervision scheme. Furthermore, by supervising the first AI model, a supervision decision can be obtained and executed, thereby solving the problem of communication processes being affected by abnormal performance of the first AI model. Attached Figure Description

[0019] Figure 1 This is one of the flowcharts illustrating the AI ​​model supervision method according to an embodiment of this application;

[0020] Figure 2 This is a second flowchart illustrating the AI ​​model supervision method according to an embodiment of this application;

[0021] Figure 3This is a schematic diagram illustrating the use of two localization methods to supervise the AI ​​model in the embodiments of this application;

[0022] Figure 4 This is the third flowchart illustrating the AI ​​model supervision method according to an embodiment of this application;

[0023] Figure 5 This is the fourth flowchart illustrating the AI ​​model supervision method according to an embodiment of this application;

[0024] Figure 6 This is one of the structural schematic diagrams of the AI ​​model supervision device according to an embodiment of this application;

[0025] Figure 7 This is a second schematic diagram of the structure of the AI ​​model supervision device according to an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of the structure of the AI ​​model supervision device according to an embodiment of this application. Detailed Implementation

[0027] To make the technical problems, technical solutions, and advantages of this invention clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as particular configurations and components are provided merely to aid in a comprehensive understanding of the embodiments of this invention. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this invention. Furthermore, for clarity and brevity, descriptions of known functions and structures have been omitted.

[0028] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0029] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0030] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0031] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0032] Embodiments of this application provide an AI model supervision method, which is applied to a first device, such as a user equipment (UE), such as a terminal, on-board unit (OBU), or roadside unit (RSU), or a network-side device, such as a base station (specifically, a 5G base station, gNB), a transmission and access point (TRP), etc. Figure 1 As shown, the method includes:

[0033] Step 101: Obtain first information, which is used to assist in supervising the first AI model, which is deployed on the first device.

[0034] In step 101 above, the first information can specifically be referred to as supervisory auxiliary information. Obtaining the first information in step 101 can specifically be either the first device obtaining first information generated / calculated by itself or receiving first information from other devices.

[0035] Step 102: Based on the first information, perform model-supervised decision-making.

[0036] Here, it should be noted that, as shown above, the first information is used to assist in the supervision of the first AI model. Therefore, one scenario is that step 102 can obtain the performance of the first AI model based on the first information, and then execute the corresponding model supervision decision based on the performance of the first AI model. For example, the first information can carry parameters characterizing the performance of the first AI model (such as reliability), and then execute the model supervision decision corresponding to the parameter based on the parameter; or, the first information carries relevant information for calculating the parameters characterizing the performance of the first AI model, and then calculates the parameters characterizing the performance of the first AI model based on this information, so as to execute the model supervision decision corresponding to the parameter based on the parameter; another scenario is that the first information carries a supervision decision, and step 102 specifically obtains the supervision decision from the first information and executes the obtained supervision decision.

[0037] Here, it should also be noted that, as an example, the supervised decision may include at least one of the following, but is not limited to: switching to a non-AI mode (non-AI localization method, i.e., traditional localization method), retraining or updating the model, fine-tuning the model, switching models (such as switching from model A (corresponding to the aforementioned first AI model) to model B (deployed on the first device side and capable of replacing model A)), and not performing any operation (e.g., when the supervision results indicate that the model performance is good).

[0038] The AI ​​model supervision method of this application embodiment firstly involves a first device acquiring first information, which is used to assist in the supervision of a first AI model deployed on the first device. Secondly, the first device executes model supervision decisions based on the first information. Thus, this application embodiment, by executing supervision decisions based on the acquired first information used to assist in the supervision of the first AI model, provides a complete and specific model supervision scheme, achieving supervision of the first AI model and thereby solving the problem of communication processes being affected by abnormal performance of the first AI model.

[0039] As an optional implementation, the first information is obtained based on a second AI model, which has the same function as the first AI model.

[0040] Specifically, the functions of the first AI model and the second AI model in this optional implementation include: AI positioning (e.g., positioning in complex scenarios), AI beam management, AI channel state information (CSI) feedback or AI CSI prediction, etc., wherein AI positioning includes, for example, direct AI positioning and / or AI-assisted positioning.

[0041] In other words, the AI ​​model supervision method in this application embodiment can use a verified, high-performance AI model (the second AI model) to supervise a first AI model deployed on a first device. The second AI model can be deployed on the first device or on other devices. When the second AI model is deployed on the first device, the first device can supervise the first AI model based on its internal implementation. When the second AI model is deployed on other devices, the first device can supervise the first AI model based on the interaction between the first device and the other device.

[0042] Subsequently, taking the first AI model as an example for localization, the implementation process of the AI ​​model supervision method in this application embodiment will be described in detail.

[0043] As an optional implementation, step 101 involves obtaining the first information, including:

[0044] The system receives the first information sent by the second device, which can specifically be a core network device, such as a Location Management Function (LMF).

[0045] The first information includes at least one of the following:

[0046] The first location information is obtained by the second device based on a second AI model or a non-AI positioning method, wherein the second AI model is deployed on the second device; wherein, when the first device is a UE, the first location information is the location information of the first device obtained based on the second AI model or a non-AI positioning method, and when the first device is a network-side device, the first location information is the location information of the UE that sends a reference signal to the network-side device, obtained based on the second AI model or a non-AI positioning method.

[0047] The first indication information is used to indicate whether the first location information is obtained based on the second AI model inference; that is, if the first location information is obtained based on the second AI model, then the first indication information indicates that the first location information is obtained based on the second AI model inference; if the second location information is obtained based on a non-AI positioning method, then the first indication information indicates that the first location information is not obtained based on the second AI model inference, or the first indication information indicates that the first location information is obtained based on a non-AI positioning method.

[0048] The second indication information is used to indicate the reliability of the first location information and / or the second AI model; here, the second indication information can also be called quality indication information; the reliability of the first location information and / or the second AI model is specifically, for example, reliability, accuracy or confidence; for example, when the second device obtains the first location information based on the second AI model, the second indication information can indicate the reliability of the second AI model, and when the second device obtains the first location information in a non-AI manner, the second indication information can indicate the reliability of the first location information;

[0049] Information related to the reference signal (RS) is used to identify a unique RS; here, the RS is the RS received by the first device; specifically, the information related to the RS includes, for example, the identification (ID) information of the RS, the ID information of the resource carrying the RS, or the ID information of the resource set carrying the RS; in addition, the information related to the RS may also include reference signal configuration information; here, the reference signal is, for example, a downlink positioning reference signal (DL PRS), an uplink sounding reference signal (UL SRS), a sidelink positioning reference signal (SL PRS), and a channel sounding signal SRS-pos used for positioning, etc.

[0050] Supervisory indicator information; as an example, supervisory indicator information can specifically be obtained by comparing the location information of the same UE obtained from different positioning methods (when the first device is a UE, the same UE is the first device; when the first device is a network-side device, the same UE is the UE that sends RS to the network-side device); supervisory indicator information can also be obtained based on intermediate positioning information related to the same UE obtained using different methods, but is not limited thereto. For example, supervisory indicator information is the comparison result of the location information obtained by the first device based on the first AI model and the location information obtained by the second device based on the second AI model. Supervisory indicator information can be represented by mean square error (MSE), similarity, or difference, but is not limited thereto.

[0051] Supervisory decisions; as previously described, the supervisory decisions include at least one of switching to non-AI mode, re-training, fine-tuning, model switching, and not performing any operation;

[0052] Time information; here, the time information specifically refers to, for example, the time when the RS is in operation, the time when the first device reports information related to the inference results of the first AI model, or the time when the first device reports information related to channel estimation; for example, the time information can be represented by at least one of the system frame number, system subframe number, system slot number, and Coordinated Universal Time (UTC); here, "RS" refers to the RS configured by the second device for the first device or the RS received by the first device, etc.

[0053] Supervision failure information, for example, when the second device (specifically, the model performance on the second device side) does not meet the supervision requirements of the first device, the second device feeds back supervision failure information to the first device.

[0054] In this optional implementation, the first device receives first information sent by the second device to determine the performance of the first AI model based on the first information, and then executes corresponding supervision decisions based on the determined supervision performance of the first AI model. In this way, the supervision of the first AI model deployed on the first device is realized through the interaction between the first device and the second device, thereby providing a complete AI model supervision scheme and avoiding the impact of AI model performance differences on the communication process.

[0055] Furthermore, the first information may also include at least one of the following:

[0056] Cell ID: When the first device is a UE, the Cell ID is the ID of the cell where the first device is located. When the first device is a network-side device, such as a base station, the Cell ID is the ID of the cell covered by the first device. Therefore, when the first device reports information to the second device, it will report the Cell ID to the second device so that when the second device performs model supervision, it can select the model to supervise the first AI model based on the Cell ID. Therefore, the first information sent by the second device to the first device can also carry the Cell ID.

[0057] The valid area of ​​the Assistance Data indicates the region / conditions used by the model and is used to determine whether the model is within a reasonable range.

[0058] As a specific example, the following explains the content of the first information when forming different supervision schemes:

[0059] Supervision scheme one: The first information includes information related to RS and / or time information, as well as first location information; in this way, the first device can match the first location information with the second location information obtained by the first AI model inference (obtained by channel estimation based on RS corresponding to the information related to RS and time information) based on the information related to RS / time information, and calculate supervision index information based on the matched first location information and second location information, thereby executing the corresponding supervision decision based on the calculated supervision index information.

[0060] Supervision Scheme 2: The first information includes information related to RS and / or time information, as well as first location information and first indication information; thus, when the method of obtaining the first location information indicated by the first indication information is the same as the method of requesting by the first device, the first device supervises the first AI device according to the implementation process of "Supervision Scheme 1", which will not be elaborated here.

[0061] Supervision Scheme 3: The first information includes information related to RS and / or time information, as well as first location information and second indication information. In this way, the first device can determine whether to use the first location information fed back by the second device to supervise the first AI model based on the second indication information. For example, if the reliability indicated by the second indication information reaches the preset reliability level, then the first location information fed back by the second device is used to supervise the first AI model. The subsequent calculation of supervision index information is the same as "Supervision Scheme 1" and will not be repeated here.

[0062] Supervision Scheme 3: The first information includes supervision indicator information. The first device determines the corresponding supervision decision based on the supervision indicator information and executes the determined supervision decision.

[0063] Supervision Scheme 4: The first information includes a supervision decision, and the first device executes the supervision decision.

[0064] Supervision Scheme 5: The first information includes supervision indicator information and second instruction information. When the reliability indicated by the second instruction information reaches the preset reliability level, the first device determines the corresponding supervision decision based on the supervision indicator information and executes the determined supervision decision.

[0065] Supervision Scheme Six: The first information includes a supervision decision and a second instruction. When the reliability indicated by the second instruction reaches a preset reliability level, the first device executes the supervision decision.

[0066] Supervision Scheme 7: The first information includes the Validity Area of ​​the Assistance Data. The first device determines whether the second location information obtained by reasoning based on the first AI model is located in the Validity Area of ​​the Assistance Data, and executes the corresponding supervision decision (e.g., if it is located, no operation is performed; if it is not located, the model is switched or the non-AI mode is switched).

[0067] Supervision Plan 8: The first message includes supervision failure information, so that the first device does not perform any operation or switches to other supervision methods.

[0068] Supervision Scheme Nine: The first information includes RS-related information and / or time information. The first device can obtain the third location information based on the RS-related information and / or time information using traditional positioning methods. The third location information is then matched with the second location information obtained by the first AI model (obtained by channel estimation based on the RS corresponding to the RS-related information / time information). Supervision index information is calculated based on the matched third and second location information, and corresponding supervision decisions are executed based on the calculated supervision index information.

[0069] Supervision Scheme 10: The first information includes first location information, RS-related information and / or time information. The first device can obtain intermediate location variables based on the RS-related information and / or time information, based on the first AI model deployed on the first device side (the first AI model is specifically an AI-assisted positioning model), and then calculate the fourth location information based on the intermediate location variables. The fourth location information is matched with the first location information inferred from the second AI model (obtained by channel estimation based on RS corresponding to RS-related information / time information), and supervision index information is calculated based on the matched fourth location information and the first location information. Then, the corresponding supervision decision is executed based on the calculated supervision index information.

[0070] Furthermore, as an optional implementation, prior to step 101, the method further includes at least one of the following:

[0071] Send a first request to the second device. The first request is used to request supervision of the first AI model. Here, the first request can be a signaling message used alone to request model supervision, or it can be a location request signaling message. That is, model supervision can be requested by adding a new signaling message, or by enhancing an existing signaling message (such as a location request signaling message / location service signaling message) (such as adding a field for requesting model supervision).

[0072] The system sends third information to the second device, which assists the second device in performing positioning operations and / or in supervising the first AI model. Specifically, the third information assists the second device in performing positioning operations by performing positioning based on the third information (e.g., positioning based on the second AI model or using traditional positioning methods). When the first device is a UE, the second device positions itself based on the third information; when the first device is a network-side device, the second device positions the UE sending RS to the network-side device based on the third information. Furthermore, the third information assists the second device in supervising the first AI model by calculating the supervision target information of the first AI model or determining the supervision decision for the first AI model based on the third information.

[0073] Receive a second request sent by the second device, the second request being used to trigger supervision of the first AI model.

[0074] Based on the above optional implementation methods, it can be seen that the AI ​​model supervision method of the embodiments of this application can be initiated by the first device as needed, or by the second device as needed.

[0075] As a specific implementation, the first request includes at least one of the following: location method request, monitoring duration, monitoring requirements, third indication information, monitoring cycle, monitoring indicator parameters, and monitoring auxiliary parameters.

[0076] The second request includes at least one of the following: location method request, monitoring duration, monitoring requirements, third indication information, monitoring cycle, monitoring indicator parameters, and monitoring auxiliary parameters.

[0077] Specifically:

[0078] The positioning method request is used to request a positioning method that needs to be executed by the first device and / or a positioning method that needs to be executed by the second device; that is, the positioning method request can request one or more positioning methods. Specifically, in this embodiment, the same RS (such as Positioning Reference Signal (PRS)) resource can support multiple positioning methods. The indication that multiple positioning methods can be implemented by making corresponding enhancements to existing protocols, specifically, indicating support for the same process to perform multiple positioning methods includes at least one of the following methods:

[0079] Add a new measurement request IE, such as the parameter in the existing protocol: New Radio(NR)-Downlink(DL)-Time Difference of Arrival(TDOA)-Request Location Information based on NR signal. The added IE requests two measurements at the same time.

[0080] Add a new bit to the existing IE to indicate the reporting of other measurements, such as adding a new bit to the Measurement Characteristic Information Requested by LMF to indicate the measurement required by another positioning method;

[0081] Supervision duration specifically refers to the length of time spent supervising the first AI model;

[0082] Supervision requirements may include, for example, requirements on the reliability (e.g., accuracy) of the second AI model deployed on the second device side, specifically requiring the reliability of the second AI model to be greater than a threshold A;

[0083] The third indication information is used to indicate whether the second device needs to output the positioning results of the second AI model deployed on the second device;

[0084] The specific parameters of the supervision indicators can be in the format of predefined supervision indicators, such as: MSE, similarity or difference, etc.

[0085] The supervision auxiliary parameters are parameters that the second device requests the first device to report, used to assist the second device in model supervision. It should be noted that the second request should include these parameters, such as the type and format of the channel estimation reported by the first device.

[0086] Furthermore, as an optional implementation, the method also includes at least one of the following:

[0087] After sending the first request to the second device, a first response corresponding to the first request is received;

[0088] After receiving the second request sent by the second device, send a second response corresponding to the second request to the second device;

[0089] Wherein, the first response and the second response each include at least one of the following:

[0090] Location method indication; corresponding to the location method request in the first / second request, it may indicate one or more location methods;

[0091] Supervision duration response; corresponding to the supervision duration in the first / second request. For example, if the supervision duration in the first / second request is 2 hours, the supervision duration response here can be either agree or disagree (it can also include the reason for disagreement or the suggested supervision duration);

[0092] The supervision request response corresponds to the supervision request in the first / second request.

[0093] A first instruction response is used to instruct the second device to determine the location result of the second AI model deployed on the second device;

[0094] The second indication response is used to instruct the second device to determine not to output the positioning result of the second AI model;

[0095] Supervision cycle response; corresponds to the supervision cycle in the first / second request;

[0096] The monitoring indicator parameter response corresponds to the monitoring indicator parameter in the first / second request, for example, the determined monitoring indicator parameter is MSE;

[0097] The supervision auxiliary parameter response is a parameter determined by the first device and reported to the second device to assist the second device in model supervision; it corresponds to the supervision auxiliary parameter in the first / second request.

[0098] As a specific implementation, the third information includes at least one of the following;

[0099] The fourth information is related to the inference result of the first AI model. Specifically, the fourth information includes the inference result of the first AI model or information about the transformation of the inference result. The inference result of the first AI model or information about the transformation of the inference result can also be called positioning intermediate information. Specifically, it includes at least one of the following: Time of Arrival (TOA), Time of Flight (TOF), Angle of Arrival (AOA), Direction of Arrival (DOA), Reference Signal Time Difference (RSTD), Relative Time of Arrival (RTOA), Time Difference between Transmitter and Receiver (Rx-Tx Time difference), and Round Trip Time (RTT).

[0100] Positioning intermediate information obtained based on non-AI positioning methods; here, non-AI positioning methods are, for example, New-Radio-Radio Access Technology (RAT) methods; wherein, the positioning intermediate information obtained based on non-AI positioning methods includes at least one of the following: TOA, TOF, AOA, DOA, RSTD, RTOA, Rx-Tx Time Difference, and RTT;

[0101] The second location information is obtained by reasoning from the first AI model.

[0102] The fifth information is related to channel estimation, which corresponds to the RS received by the first device. For example, the fifth information is the channel estimation or channel estimation transformation information corresponding to the RS, such as time information (the time of the RS), power information, and phase information, specifically such as: Channel Impulse Response (CIR), Power Delay Profile (PDP), Delay Profile, Reference Signal Received Power (RSRP), and Reference Signal Received Path Power (RSRPP). The specific parameters included in the fifth information can be indicated by the second device.

[0103] The time information, as mentioned above, specifically includes, for example, the time when the RS is located, the time when the first device reports information related to the inference result of the first AI model (the aforementioned fourth information), or the time when the first device reports information related to channel estimation (the aforementioned fifth information); for example, the time information can be represented by at least one of the system frame number, system subframe number, system slot number, and UTC; here, "RS" refers to the RS configured by the second device for the first device or the RS received by the first device, etc.

[0104] Information related to RS is used to determine a unique reference signal. This information may include, for example, the ID information of the RS, the ID information of the resource carrying the RS, or the ID information of the resource set carrying the RS. Additionally, the information related to RS may also include reference signal configuration information.

[0105] Here, since the third information includes the fourth information or the second location information, the method further includes the following steps before sending the third information to the second device:

[0106] Based on the channel estimate corresponding to the RS received by the first device and the first AI model, model inference is performed to obtain the fourth information or the second location information; in other words, this step is: inputting the channel estimate corresponding to the RS received by the first device into the first AI model to obtain the fourth information (e.g., the first AI model is an assisted positioning model) or the second location information (the first AI model is a direct positioning model).

[0107] Additionally, it should be noted that the first device itself can supervise the first AI model based on the fourth information corresponding to the same RS and the intermediate positioning information obtained based on the non-AI positioning method. Specifically, the fourth information and the intermediate positioning information obtained based on the non-AI positioning method are compared to obtain supervision index information (MSE, similarity, or difference between the two) based on the comparison results. Then, the corresponding supervision decision is executed based on the obtained supervision index information. In this way, it is not necessary to supervise the first AI model through the interaction between the first device and the second device. Therefore, the third information may not include the aforementioned fourth information and the intermediate positioning information obtained based on the non-AI positioning method.

[0108] As an optional implementation, step 102 includes:

[0109] Based on the first information, the supervision indicator information is obtained; for example, when the first information carries the supervision indicator information, this step is to extract the supervision indicator information from the first information.

[0110] Based on the supervision indicator information, a model supervision decision is executed. Here, the supervision decision corresponds to the supervision indicator information. For example, if the supervision indicator information indicates that the performance of the first AI model is poor, the supervision decision to be executed can be: switch the model or switch to non-AI mode. If the supervision indicator information indicates that the performance of the first AI model is good, the supervision decision to be executed is: do not perform any operation. If the supervision indicator information indicates that the performance of the first AI model just meets the requirements or is slightly below the requirements, the supervision decision to be executed is: re-training or fine-tuning.

[0111] As a specific implementation method, based on the first information, supervision indicator information is obtained, including:

[0112] Based on the relevant information and / or time information of the RS received by the first device, the first information obtained is matched with the location information inferred by the first AI model to obtain the target first information that matches the location information inferred by the first AI model; here, the matched target first information and the location information inferred by the first AI model are related to the RS that meet the pre-configuration requirements, wherein the RS that meet the pre-configuration requirements are, for example, the same RS, or two RS that are close in time (for example, the time interval is less than the preset first duration).

[0113] The location information obtained based on the first target information is compared with the location information inferred by the first AI model to obtain the supervision index information; here, the location information obtained based on the first target information is, for example, the first location information carried in the first target information; the supervision index information is the MSE, similarity or difference of the two location information being compared.

[0114] As an optional implementation, step 102 includes:

[0115] Based on the supervision index information obtained from the first information and the sixth information, a model supervision decision is executed, wherein the sixth information is information obtained using a pre-configured model supervision method.

[0116] In other words, when making supervised decisions, one can consider not only the information provided by the second device, but also information obtained through other supervision methods (sixth information), such as the data distribution information of the model input and / or output during the model training and inference phases.

[0117] Embodiments of this application also provide an AI model supervision method, which is applied to a second device, for example, an LMF (Laser-Based Model). Figure 2 As shown, the method includes:

[0118] Step 201: Send first information, which is used to assist in the supervision of the first AI model, which is deployed on the first device.

[0119] In the above steps, the first device is a UE or a network-side device (such as a gNB or TRP), and sending the first information in the above steps specifically means sending the first information to the first device.

[0120] In the embodiments of this application, the second device sends first information to assist in supervising the first AI model, enabling the first device to determine the performance of the first AI model based on the first information, and thus execute corresponding supervision decisions based on the performance of the first AI model. In this way, the embodiments of this application provide a complete and specific model supervision scheme, realize the supervision of the first AI model, and solve the problem of communication process being affected by abnormal performance of the first AI model.

[0121] As an optional implementation, the first information is obtained based on a second AI model, which has the same function as the first AI model. The second AI model is deployed on a second device.

[0122] Specifically, the functions of the first AI model and the second AI model in this optional implementation include: AI positioning (e.g., positioning in complex scenarios), AI beam management, AI channel state information (CSI) feedback or AI CSI prediction, etc., wherein AI positioning includes, for example, direct AI positioning and / or AI-assisted positioning.

[0123] In other words, the AI ​​model supervision method in this application embodiment can be to use a verified AI model with better performance (the second AI model) to supervise the first AI model deployed on the first device, wherein the second AI model is deployed on the second device. In this way, the first AI model deployed on the first device can be supervised based on the interaction between the first device and the second device.

[0124] As an optional implementation, the first information includes at least one of the following:

[0125] The first location information is obtained by the second device based on a second AI model or a non-AI positioning method;

[0126] The first indication information is used to indicate whether the first location information is obtained based on the reasoning of the second AI model;

[0127] The second indication information is used to indicate the reliability of the first location information and / or the second AI model;

[0128] Information related to RS is used to identify a unique RS;

[0129] Monitoring indicator information;

[0130] Oversee decision-making;

[0131] Time information;

[0132] Monitoring failure information.

[0133] The information included in the first information in the above optional implementation methods is the same as the information included in the first information on the first device side, and will not be explained again here.

[0134] As a specific implementation, the second indication information, when used to indicate the reliability of the second AI model, includes at least one of the following:

[0135] The loss value of the second AI model;

[0136] The accuracy of the second AI model supervision; accuracy can also be called accuracy rate, for example: the inference accuracy when the information of the channel estimate or its transformation corresponding to the reference signal sent or received by the Positioning Reference Unit (PRU) is used as the model input; such as MSE, positioning error (e.g., if the PRU's true position is (a1, a2, a3) and the model's inference position is (b1, b2, b3), its positioning error is... wait;

[0137] The confidence level of the second AI model supervision;

[0138] The reliability of the second AI model supervision.

[0139] Furthermore, as an optional implementation, prior to step 201, the method further includes at least one of the following:

[0140] Receive a first request sent by the first device, the first request being used to request supervision of the first AI model;

[0141] The second device receives third information sent by the first device, which is used to assist the second device in locating the first device and / or in supervising the first AI model. Specifically, the third information assists the second device in performing a location operation by performing location based on the third information (e.g., location based on the second AI model or using traditional location calculation methods). When the first device is a UE, the second device locates the first device based on the third information; when the first device is a network-side device, the second device locates the UE sending RS to the network-side device based on the third information. On the other hand, the third information assists the second device in supervising the first AI model by calculating the supervision target information of the first AI model or determining the supervision decision for the first AI model based on the third information.

[0142] A second request is sent, which triggers supervision of the first AI model.

[0143] As a specific implementation, the first request includes at least one of the following: location method request, monitoring duration, monitoring requirements, third indication information, monitoring cycle, monitoring indicator parameters, and monitoring auxiliary parameters.

[0144] The second request includes at least one of the following: location method request, monitoring duration, monitoring requirements, third indication information, monitoring cycle, monitoring indicator parameters, and monitoring auxiliary parameters.

[0145] Specifically:

[0146] A location method request is used to request a location method that the first device needs to execute and / or a location method that the second device needs to execute.

[0147] The third indication information is used to indicate whether the second device needs to output the positioning results of the second AI model deployed on the second device;

[0148] Supervision auxiliary parameters are parameters that the second device requests the first device to report, used to assist the second device in performing model supervision.

[0149] In the specific implementation described above, the first request and the second request include the same content as the first request and the second request in the embodiment on the first device side, and will not be repeated here.

[0150] It should be noted that after receiving the first request sent by the first device, the second device also configures a reference signal, specifically including: configuring periodic reference signal transmission and / or on-demand reference signal transmission, such as On-demand PRS configuration, to send RS to the first device, thereby performing positioning based on information related to RS and information obtained from channel estimation or channel estimation transformation corresponding to RS, and supervising the first AI model based on the positioning results.

[0151] Furthermore, as an optional implementation, the method also includes at least one of the following:

[0152] After receiving the first request sent by the first device, send a first response corresponding to the first request to the first device;

[0153] After sending the second request, receive the second response sent by the first device corresponding to the second request;

[0154] Wherein, the first response and the second response each include at least one of the following:

[0155] Positioning method indication;

[0156] Monitoring duration response;

[0157] Respond to supervisory requirements;

[0158] A first instruction response is used to instruct the second device to determine the location result of the second AI model deployed on the second device;

[0159] The second indication response is used to instruct the second device to determine not to output the positioning result of the second AI model;

[0160] Monitoring cycle response;

[0161] Monitor the response of indicator parameters;

[0162] The supervision auxiliary parameter response is a parameter determined by the first device and reported to the second device to assist the second device in model supervision.

[0163] In the above optional implementation methods, the first response and the second response include the same content as the first response and the second response in the embodiment on the first device side, and will not be repeated here.

[0164] As a specific implementation, the third information includes at least one of the following;

[0165] The fourth piece of information is related to the reasoning result of the first AI model;

[0166] Location intermediate information obtained based on non-AI positioning methods;

[0167] The second location information is obtained by reasoning from the first AI model.

[0168] The fifth piece of information is related to channel estimation, and the channel estimation corresponds to the RS received by the first device;

[0169] Time information;

[0170] Information related to RS is used to determine a unique reference signal.

[0171] In the specific implementation described above, the third information includes the same content as the third information in the embodiment on the first device side, and will not be repeated here.

[0172] Furthermore, as an optional implementation, after receiving the third information sent by the first device, the method further includes at least one of the following:

[0173] Based on the third information and the second AI model deployed on the second device, a positioning operation is performed; specifically, this step may involve inputting the third information (such as the fifth information related to channel estimation) into the second AI model and outputting the first location information inferred by the second AI model.

[0174] Based on the third information, a non-AI positioning operation is performed; specifically, this step may be to use traditional positioning calculation methods to calculate the fourth information related to the reasoning result of the first AI model or the positioning intermediate information obtained based on the non-AI positioning method to obtain the corresponding first location information.

[0175] Based on the relevant information and / or time information of the RS configured for the first device, at least two of the received third information, location information obtained by AI positioning, and location information obtained by non-AI positioning are matched, and supervision index information is calculated based on the two matched location information. For example, this step is: matching the second location information in the third information with the location information obtained by AI positioning, or matching the second location information in the third information with the location information obtained by non-AI positioning, or matching the location information obtained by AI positioning with the location information obtained by non-AI positioning, wherein the matching is based on the fact that these location information are related to RSs that meet the pre-configuration requirements, wherein the RSs that meet the pre-configuration requirements are, for example, the same RS, or two RSs that are close in time (e.g., the time interval is less than a preset first duration).

[0176] Combining the aforementioned optional implementation methods, the implementation steps of the AI ​​model supervision method in the embodiments of this application generally include the following two methods:

[0177] Method 1: Model supervision is initiated by the first device, which includes the following steps:

[0178] 1. The first device sends a first request to the second device, the first request being used to request supervision of the first AI model;

[0179] 2. The second device is configured with RS;

[0180] 3. The second device sends a first response to the first device;

[0181] 4. The first device performs channel estimation on the received RS to obtain the fifth information related to the channel estimation (including information about the channel estimation itself or the channel estimation conversion);

[0182] 5-1. The first device inputs the channel estimation into the first AI model to obtain the second location information (the first AI model is a direct positioning model) or obtains the inference result of the first AI model or the information of the transformation of the inference result (the first AI model is an auxiliary positioning model).

[0183] 5-2. The first device obtains intermediate positioning variables based on non-AI positioning methods and channel estimation;

[0184] 6a. The first device matches the inference result or the information transformed from the inference result of the first AI model with the positioning intermediate variable, compares the matched information to obtain the supervision indicator information, and executes supervision decisions based on the supervision indicator information;

[0185] 6b. The first device sends third information to the second device, the third information being used to assist the second device in performing positioning operations and / or to assist the second device in supervising the first AI model;

[0186] Among them, 6a and 6b are two parallel solutions. 6a is that the first device itself supervises the first AI model, while 6b is that the first device and the second device interact to supervise the first AI model.

[0187] 7a. After executing 6b, the second device inputs the fifth information related to channel estimation from the third information into the second AI model to obtain the first location information for the second AI model inference; and sends the second location information, the ID and / or time information of the RS corresponding to the second location information to the first device so that the first device can match and compare the first location information and the second location information to obtain model supervision information.

[0188] 7b. After executing 6b, the second device calculates the intermediate positioning variables obtained from the third information based on the non-AI positioning method using the traditional positioning method to obtain the first location information; and sends the second location information, the ID of the RS corresponding to the second location information, and / or the time information to the first device so that the first device can match and compare the first location information and the second location information to obtain model supervision information.

[0189] 7c. After executing 6b, the second device uses a traditional method to solve the fourth information in the third information that is related to the inference result of the first AI model to obtain a first position information. The second device also inputs the fifth information related to channel estimation into the second AI model to obtain another first position information. By matching and comparing the two first position information, model supervision information is obtained.

[0190] 7d. After executing 6b, the second device uses a traditional method to solve the fourth information related to the reasoning result of the first AI model in the third information to obtain a first location information. The second device also uses a traditional positioning method to solve the positioning intermediate information obtained based on the non-AI positioning method to obtain a second first location information. By matching and comparing the two first location information, model supervision information is obtained.

[0191] 7e. When the second device determines that the performance of the second AI model deployed on it does not meet the supervision requirements in the first request, it generates supervision failure information;

[0192] The above 7a, 7b, 7c, 7d, and 7e are four parallel implementation schemes;

[0193] 8a. After performing 7a, 7b, 7c, 7d or 7e above, the second device sends the first information to the first device;

[0194] 8b. After executing 7a, 7b, 7c, 7d or 7e above, the second device determines a supervision decision based on the above supervision indicator information, and after determining the supervision decision, sends the first information to the first device;

[0195] It should be noted here that 8a and 8b are two parallel cases;

[0196] 9. The first device executes a supervisory decision based on the first information.

[0197] Method 2: Model supervision is initiated by a second device, which includes the following steps:

[0198] 1. The second device sends a second request to the first device, the second request being used to trigger supervision of the first AI model;

[0199] 2. The first device sends a second response to the second device;

[0200] The subsequent steps are the same as steps 4 to 9 in Method 1, and will not be repeated here.

[0201] Below, with Figure 3 For example, in the AI ​​model supervision method of this application embodiment, the two localization methods are AI direct localization performed on the LMF side and AI direct localization performed on the UE side, and the same set of PRS is used. Of course, depending on the different supervision methods, the two localization methods may also be the combination shown in Table 1, and the measurement time (gap) can be enhanced to a certain extent.

[0202] Table 1

[0203]

[0204] Below, based on the reference signal received at the UE / gNB side and the channel estimation obtained based on the reference signal, examples of different combinations of supervision methods in Table 1 are provided:

[0205] Combination 1: The UE uses the obtained channel estimate as model input to obtain location information A1 using its own AI direct positioning model, and sends the channel estimate to the LMF. The LMF uses the obtained channel estimate as model input to obtain location A2 using its own AI direct positioning model. Finally, based on the comparison results of A1 and A2, a supervision index characterizing the performance of the direct positioning model on the UE side is obtained (which can be executed by the UE or the LMF), so that the UE can make supervision decisions according to the supervision index.

[0206] Combination 2: The UE uses the obtained channel estimate as the model input to obtain the location information A1 by its own AI direct positioning model, and sends the location intermediate variable b1 obtained by measuring the reference signal to the LMF. The LMF solves the obtained location intermediate variable b1 based on the traditional positioning method to obtain the location A3. Finally, based on the comparison result of A1 and A3, a supervision index characterizing the performance of the direct positioning model on the UE side is obtained (which can be executed by the UE or the LMF), so that the UE can make supervision decisions according to the supervision index.

[0207] Combination 3: The UE uses the obtained channel estimate as model input, obtains location information A1 using its own AI direct positioning model, and locates location A4 using a non-AI positioning method (traditional positioning method) based on the received reference signal. Finally, a supervision index characterizing the performance of the UE-side direct positioning model is obtained based on the comparison result of A1 and A4, so that the UE can perform supervision decisions according to the supervision index. In this combination method for supervising the first AI model, the positioning method request in the information used to request model supervision (such as the first request / second request mentioned above) can be used only to request the positioning method that the first device needs to execute, while the other positioning method depends on the UE implementation.

[0208] Combination 4: The UE uses the obtained channel estimate as model input to obtain location information A1 using its own AI direct positioning model. The UE uses the obtained channel estimate as model input to obtain positioning intermediate variable b2 using its own AI-assisted positioning model. The UE then uses traditional positioning solution methods to solve the positioning intermediate variable b2 to obtain location A5. Finally, based on the comparison results of A1 and A5, a supervision index characterizing the performance of the direct positioning model on the UE side is obtained, so that the UE can perform supervision decisions according to the supervision index.

[0209] Combination 5: The UE uses the obtained channel estimate as model input to obtain the intermediate positioning variable b3 using its own AI-assisted positioning model, and sends the intermediate positioning variable b3 and the channel estimate to the LMF. The LMF uses the obtained channel estimate as model input to obtain the location A2 using its own AI direct positioning model, and uses the traditional positioning solution method to solve the intermediate positioning variable b3 to obtain the location A6. Finally, based on the comparison results of A1 and A6, a supervision index characterizing the performance of the assisted positioning model on the UE side is obtained (which can be executed by the UE or the LMF), so that the UE can make supervision decisions according to the supervision index.

[0210] Combination 6: The UE uses the obtained channel estimate as model input to obtain the intermediate positioning variable b2 using its own AI-assisted positioning model, and uses the traditional positioning solution method to solve b2 to obtain the location A5; the UE sends the obtained channel estimate to the LMF, and the LMF uses the obtained channel estimate as model input to obtain the location A2 using its own AI direct positioning model. Finally, based on the comparison result of A2 and A5, a supervision index characterizing the performance of the assisted positioning model on the UE side is obtained (which can be executed by the UE or the LMF), so that the UE can make supervision decisions according to the supervision index.

[0211] Combination 7: The UE uses the obtained channel estimate as the model input to obtain the intermediate positioning variable b3 using its own AI-assisted positioning model, and performs non-AI positioning based on the reference signal to obtain the position A7. Finally, based on the comparison results of A1 and A7, a supervision index characterizing the performance of the assisted positioning model on the UE side is obtained, so that the UE can perform supervision decisions according to the supervision index.

[0212] Combination 8: The gNB uses the obtained channel estimate as model input to obtain the intermediate positioning variable b3 using its own AI-assisted positioning model. The gNB sends the intermediate positioning variable b3 and the channel estimate to the LMF. The LMF uses the obtained channel estimate as model input to obtain the location A2 using its own AI direct positioning model, and uses the traditional positioning solution method to solve b3 to obtain the location A8. Finally, the supervision index characterizing the performance of the gNB-side assisted positioning model is obtained based on the comparison results of A2 and A8.

[0213] Combination 9: The gNB uses the obtained channel estimate as the model input and its own AI-assisted positioning model to obtain the positioning intermediate variable b3. When there are multiple direct paths, it uses the reference signal corresponding to the direct path to obtain the positioning intermediate variable b4 using the traditional positioning method. The gNB sends the positioning intermediate variables b3 and b4 to the LMF. The LMF uses the traditional positioning solution method to solve b3 and b4 respectively to obtain positions A8 and A9. Finally, based on the comparison results of A8 and A9, a supervisory index characterizing the performance of the gNB-side assisted positioning model is obtained.

[0214] It should be noted that the non-AI positioning methods involved in the above combinations can only be executed under certain conditions. For example, the condition may be that there are n direct paths between the devices transmitting and receiving reference signals (such as the multiple direct paths in combination 9).

[0215] It should also be noted that in the above combinations, if there is no interaction between the first and second devices, the location method request in the information used to request model supervision (such as the first request / second request mentioned above) does not need to trigger both location methods. For example, when the first device calculates the supervision index, in combination 1, only the LMF side can be triggered to perform AI location. The LMF will configure the PRS, and the UE's operation after receiving the PRS is its internal implementation. However, if the UE needs to report the results of both location methods to the LMF, then both location methods need to be triggered.

[0216] It should also be noted that the aforementioned non-AI positioning methods can also be referred to as traditional positioning methods. These non-AI positioning methods may include at least one of the following: Network-Assisted Global Navigation Satellite System (GNSS) methods, Observed Time Difference of Arrival (OTDOA) Positioning Based on Long Term Evolution (LTE) signals, Enhanced Cell ID Methods Based on LTE Signals, Wireless Local Area Network (WLAN) Positioning, Bluetooth Positioning, Terrestrial Beacon System (TBS) Positioning, Sensor-Based Methods: Barometric Pressure Sensor or Motion Sensor, and NR Enhanced Cell ID Methods (NR E-CID) based on NR signals. NR signals), Multi-Round Trip Time Positioning (Multi-RTT based on NR signals), Downlink Angle-of-Departure (DL-AoD) based on NR signals, Downlink Time Difference of Arrival (DL-TDOA) based on NR signals, and Uplink Angle-of-Arrival, including Horizontal Angle-of-Arrival (A-AOA) and Vertical Angle-of-Arrival (UL-AoA) based on NR signals.This includes Azimuth Angle-of-Arrival (A-AoA) and Zenith Angle-of-Arrival (Z-AoA) based on NR signals, Sidelink Time Difference of Arrival (SL-TDOA), Sidelink Round Trip Time (SL-RTT), and Sidelink Azimuth Angle-of-Arrival / Zenith Angle-of-Arrival (SL-AoA / ZoA) based on NR signals.

[0217] The following are specific examples illustrating the AI ​​model supervision method described in the embodiments of this application.

[0218] Example 1: This example specifically supervises an AI model deployed on the UE side. The AI ​​model is a direct localization model. The specific supervision process is as follows: Figure 4 As shown, it includes the following steps:

[0219] 1. The UE initiates a Location Service Request to the LMF. The Location Service Request can also be called a Location Request or a Supervision Request. The request indicates a request for an AI model based on the LMF.

[0220] 2. Reference signal configuration, etc.; Specifically, this step can configure the reference signal for the UE through the gNB by the LMF; More specifically, this step includes the transmission of the LMF configuration reference signal (DL-PRS);

[0221] 3. The gNB sends a PRS to the UE;

[0222] 4. UE performs channel measurement + (AI inference); Specifically, this step is as follows: After receiving the DL-PRS, the UE further obtains the channel estimation information corresponding to the DL-PRS. Then, the channel estimation information is used as the model input, and the UE's own direct positioning model (corresponding to the aforementioned first AI model) is used to obtain the position a, where a corresponds to the PRS id information and / or time information; the time information here can be the time configured for the DL-PRS or the time maintained locally by the UE, etc.

[0223] 5. The UE reports channel measurements to the LMF; specifically, this step involves the UE reporting channel estimation information, PRS ID, and time information to the LMF.

[0224] 6. LMF performs AI inference; specifically, this step is as follows: LMF uses the received channel estimation information to perform AI direct positioning to obtain location b;

[0225] 7. The LMF sends the supervision metrics / inference results to the UE. Specifically, this step involves the LMF sending location b and its corresponding PRSid, time information, confidence level, or reliability information to the UE. After this, the UE determines the correspondence between location a and location b through the PRSid and / or time information. Then, the UE compares location a and location b. If a and b are relatively close (e.g., MSE value, or positioning error less than the threshold Δd), the UE's AI model is considered to have good performance. If a and b differ significantly (e.g., MSE value, or positioning error greater than the threshold Δd), the UE's AI model is considered to have poor performance, and a supervision decision can be executed, such as rollback, re-training, fine-tuning, or model switching. Rollback is switching to a non-AI mode, re-training is retraining or updating the model, fine-tuning is fine-tuning the model, and model switching is switching from model A to model B.

[0226] Example 2: The implementation process of this example is similar to that of Example 1 above, both involving the supervision of the AI ​​model on the UE side. Here, the AI ​​model is a direct localization model. Specifically, steps 1 to 4 are identical in the supervision process. Below, only the steps where differences exist will be explained:

[0227] 5. The UE reports channel measurements to the LMF; specifically, this step involves the UE reporting the intermediate positioning variables, PRS ID, and time information based on DL-PRS measurements to the LMF.

[0228] 6. LMF performs AI inference / localization calculation; specifically, in this example, this step is: LMF uses the traditional localization method to calculate the received intermediate localization variables to obtain the location c;

[0229] 7. The LMF sends supervision metrics / inference results to the UE. Specifically, this step involves the LMF sending location c and its corresponding PRSid, time information, confidence level, or reliability information to the UE. After this, the UE determines the correspondence between location a and location c through the PRSid and / or time information. Then, the UE compares location a and location c. If a and c are relatively close (e.g., MSE value, or positioning error less than the threshold Δd), the UE considers the performance of its AI model to be good. If a and c differ significantly (e.g., MSE value, or positioning error greater than the threshold Δd), the UE considers the performance of its AI model to be poor, and supervision decisions such as rollback, re-training, fine-tuning, and model switching can be performed.

[0230] It should be noted here that the model supervision process in Examples 1 and 2 above can be completed within the normal localization process, or there can be additional model supervision requests (such as...). Figure 4 Step 5a: The UE requests model supervision from the LMF.

[0231] Example 3: This example specifically supervises an AI model deployed on the network side (such as a gNB), where the AI ​​model is an auxiliary localization model. The specific supervision process is as follows: Figure 5 As shown, it includes the following steps:

[0232] 1. The UE initiates a Location Service Request to the LMF. The Location Service Request can also be called a Location Request or a Supervision Request. The request indicates that the location method is a New Radio (NR) Radio Access Network (RAN) Node-assisted location method and / or an LMF-based location method.

[0233] 2. Reference signal configuration, etc.; Specifically, this step can configure the reference signal for the UE through the gNB by the LMF; More specifically, this step includes the transmission of the LMF-configured reference signal (SRS-pos);

[0234] 3. The UE sends an SRS to the gNB;

[0235] 4. gNB performs channel measurement + (AI inference); Specifically, this step is as follows: after receiving SRS-pos, gNB further obtains the model input through channel estimation, and performs AI model inference to obtain the positioning intermediate variable information a1;

[0236] 5. The gNB reports channel measurements to the LMF; specifically, this step is as follows: the gNB reports the channel estimation information obtained from SRS-pos to the LMF;

[0237] 5a. gNB reports the inference results to LMF; specifically, this step is: gNB reports the location intermediate variable information a1 to LMF;

[0238] 6. LMF performs AI inference; specifically, this step is: LMF uses the received channel estimation information to perform AI direct positioning model inference to obtain the position f;

[0239] 6a. LMF performs positioning calculation; specifically, this step is: LMF obtains position e using the traditional positioning calculation method based on a1;

[0240] 7. The LMF sends the supervision results to the gNB. Specifically, before this step, the LMF needs to perform any of the following processes: (1) compare the information of position e and position f, and calculate the supervision index (such as MSE value, positioning error or similarity, etc.); (2) compare the information of position e and position f, calculate the supervision index, and give the supervision decision. Among them, after performing process (1), this step (step 7) specifically sends the supervision index to the gNB, and after performing process (2), this step (step 7) specifically sends the supervision decision to the gNB.

[0241] Example 4: This example specifically supervises an AI model deployed on the network side (such as a gNB), where the AI ​​model is an assisted localization model. In the specific supervision process, steps 1 to 4, step 5a, and step 7 are the same, and this example does not include step 6 from Example 3. Below, only the steps where there are differences will be explained:

[0242] 5. gNB reports channel measurements to LMF; specifically, this step is as follows: in the case of multiple direct paths, gNB will report the intermediate positioning variable a2 obtained by the traditional positioning method based on the SRS-pos corresponding to the direct path to LMF.

[0243] 6a. LMF performs positioning calculation; specifically, this step is as follows: LMF obtains position e using the traditional positioning calculation method based on a1, and obtains position f using the traditional positioning calculation method based on a2.

[0244] It should be noted here that the model supervision process in Examples 3 and 4 above can be completed within the normal localization process, or there can be additional model supervision requests (such as...). Figure 5 Step 5c: gNB requests model supervision from LMF.

[0245] Embodiments of this application provide an AI model supervision device, wherein the method is applied to a first device, such as... Figure 6 As shown, the device includes:

[0246] The acquisition module 601 is used to acquire first information, which is used to assist in the supervision of the first AI model, and the first AI model is deployed on the first device;

[0247] The execution module 602 is used to execute model supervision decisions based on the first information.

[0248] Optionally, the first information is obtained based on a second AI model, which has the same function as the first AI model.

[0249] Optionally, the acquisition module 601 is specifically configured to: receive the first information sent by the second device, wherein the first information includes at least one of the following:

[0250] First location information, which is obtained by the second device based on a second AI model or a non-AI positioning method, wherein the second AI model is deployed on the second device;

[0251] The first indication information is used to indicate whether the first location information is obtained based on the reasoning of the second AI model;

[0252] The second indication information is used to indicate the reliability of the first location information and / or the second AI model;

[0253] Information related to the reference signal RS is used to determine the unique RS;

[0254] Monitoring indicator information;

[0255] Oversee decision-making;

[0256] Time information;

[0257] Monitoring failure information.

[0258] Optionally, the device further includes at least one of the following:

[0259] The first sending module is used to send a first request to the second device before the acquisition module 601 acquires the first information. The first request is used to request supervision of the first AI model.

[0260] The second sending module is used to send third information to the second device before the acquisition module 601 acquires the first information. The third information is used to assist the second device in performing positioning operations and / or to assist the second device in supervising the first AI model.

[0261] The first receiving module is configured to receive a second request sent by the second device before the acquisition module 601 acquires the first information, the second request being used to trigger supervision of the first AI model.

[0262] Optionally, the first request and the second request each include at least one of the following:

[0263] A location method request is used to request a location method that the first device needs to execute and / or a location method that the second device needs to execute.

[0264] Duration of supervision;

[0265] Supervision requirements;

[0266] The third instruction information is used to indicate whether the second device needs to output the positioning results of the second AI model deployed on the second device;

[0267] Supervision cycle;

[0268] Monitoring indicator parameters;

[0269] Supervision auxiliary parameters are parameters that the second device requests the first device to report, used to assist the second device in performing model supervision.

[0270] Optionally, the device further includes:

[0271] The second receiving module is used to receive a first response corresponding to the first request after the first sending module sends a first request to the second device;

[0272] The third sending module is used to send a second response corresponding to the second request to the second device after the first receiving module receives the second request sent by the second device;

[0273] Wherein, the first response and the second response each include at least one of the following:

[0274] Positioning method indication;

[0275] Monitoring duration response;

[0276] Respond to supervisory requirements;

[0277] A first instruction response is used to instruct the second device to determine the location result of the second AI model deployed on the second device;

[0278] The second indication response is used to instruct the second device to determine not to output the positioning result of the second AI model;

[0279] Monitoring cycle response;

[0280] Monitor the response of indicator parameters;

[0281] The supervision auxiliary parameter response is a parameter determined by the first device and reported to the second device to assist the second device in model supervision.

[0282] Optionally, the third information includes at least one of the following;

[0283] The fourth piece of information is related to the reasoning result of the first AI model;

[0284] Location intermediate information obtained based on non-AI positioning methods;

[0285] The second location information is obtained by reasoning from the first AI model.

[0286] The fifth piece of information is related to channel estimation, and the channel estimation corresponds to the RS received by the first device;

[0287] Time information;

[0288] Information related to RS is used to determine a unique reference signal.

[0289] Optionally, the execution module 602 includes:

[0290] The acquisition submodule is used to obtain supervision indicator information based on the first information;

[0291] The execution submodule is used to execute model supervision decisions based on the supervision index information.

[0292] Optionally, the acquisition submodule includes:

[0293] The matching unit is configured to match the acquired first information with the location information inferred by the first AI model based on the relevant information and / or time information of the RS received by the first device, so as to obtain the target first information that matches the location information inferred by the first AI model.

[0294] The comparison unit is used to compare the location information obtained based on the first target information with the location information inferred by the first AI model to obtain the supervision index information.

[0295] Optionally, the execution module 602 is specifically used to: execute model supervision decisions based on the supervision index information obtained from the first information and the sixth information, wherein the sixth information is information obtained using a pre-configured model supervision method.

[0296] It should be noted that the AI ​​model supervision device provided in this application embodiment can implement all the method steps implemented in the AI ​​model supervision method embodiment applied to the first device, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0297] Embodiments of this application provide an AI model supervision device, wherein the method is applied to a second device, such as... Figure 7 As shown, the device includes:

[0298] The first sending module 701 is used to send first information, which is used to assist in the supervision of the first AI model, which is deployed on the first device.

[0299] Optionally, the first information is obtained based on a second AI model, which has the same function as the first AI model.

[0300] Optionally, the first information includes at least one of the following:

[0301] The first location information is obtained by the second device based on a second AI model or a non-AI positioning method, wherein the second AI model is deployed on the second device;

[0302] The first indication information is used to indicate whether the first location information is obtained based on the reasoning of the second AI model;

[0303] The second indication information is used to indicate the reliability of the first location information and / or the second AI model;

[0304] Information related to the reference signal RS is used to determine the unique RS;

[0305] Monitoring indicator information;

[0306] Oversee decision-making;

[0307] Time information is used to determine two locational information for calculating monitoring indicator information;

[0308] Monitoring failure information.

[0309] Optionally, the second indication information, when used to indicate the reliability of the second AI model, includes at least one of the following:

[0310] The loss value of the second AI model;

[0311] The accuracy of the second AI model supervision;

[0312] The confidence level of the second AI model supervision;

[0313] The reliability of the second AI model supervision.

[0314] Optionally, the device further includes at least one of the following:

[0315] The first receiving module is configured to receive a first request sent by the first device before the first sending module 701 sends the first information, wherein the first request is used to request supervision of the first AI model.

[0316] The second receiving module is used to receive third information sent by the first device before the first sending module 701 sends the first information. The third information is used to assist the second device in locating the first device and / or to assist the second device in supervising the first AI model.

[0317] The second sending module is used to send a second request before the first sending module 701 sends the first information. The second request is used to trigger supervision of the first AI model.

[0318] Optionally, the first request and the second request each include at least one of the following:

[0319] A location method request is used to request a location method that the first device needs to execute and / or a location method that the second device needs to execute.

[0320] Duration of supervision;

[0321] Supervision requirements;

[0322] The third instruction information is used to indicate whether the second device needs to output the positioning results of the second AI model deployed on the second device;

[0323] Supervision cycle;

[0324] Monitoring indicator parameters;

[0325] Supervision auxiliary parameters are parameters that the second device requests the first device to report, used to assist the second device in performing model supervision.

[0326] Optionally, the device further includes:

[0327] The third sending module is used to send a first response corresponding to the first request to the first device after the first receiving module receives the first request sent by the first device;

[0328] The third receiving module is used to receive the second response corresponding to the second request sent by the first device after the second sending module sends the second request;

[0329] Wherein, the first response and the second response each include at least one of the following:

[0330] Positioning method indication;

[0331] Monitoring duration response;

[0332] Respond to supervisory requirements;

[0333] A first instruction response is used to instruct the second device to determine the location result of the second AI model deployed on the second device;

[0334] The second indication response is used to instruct the second device to determine not to output the positioning result of the second AI model;

[0335] Monitoring cycle response;

[0336] Monitor the response of indicator parameters;

[0337] The supervision auxiliary parameter response is a parameter determined by the first device and reported to the second device to assist the second device in model supervision.

[0338] Optionally, the third information includes at least one of the following;

[0339] The fourth piece of information is related to the reasoning result of the first AI model;

[0340] Location intermediate information obtained based on non-AI positioning methods;

[0341] The second location information is obtained by reasoning from the first AI model.

[0342] The fifth piece of information is related to channel estimation, and the channel estimation corresponds to the RS received by the first device;

[0343] Time information;

[0344] Information related to RS is used to determine a unique reference signal.

[0345] Optionally, the device further includes:

[0346] The first positioning module is used to perform a positioning operation based on the third information sent by the first device and the second AI model deployed on the second device after the second receiving module receives the third information sent by the first device.

[0347] The second positioning module is used to perform a non-AI positioning operation based on the third information sent by the first device after the second receiving module receives the third information.

[0348] The matching module is used to match at least two of the received third information, location information obtained by AI positioning, and location information obtained by non-AI positioning, according to the relevant information and / or time information of the RS configured for the first device after the second receiving module receives the third information sent by the first device, and to calculate supervision index information based on the two matched location information.

[0349] It should be noted that the AI ​​model supervision device provided in this application embodiment can implement all the method steps implemented in the AI ​​model supervision method embodiment applied to the second device, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0350] An embodiment of this application also provides an AI model supervision device, including a transceiver 810, a processor 800, a memory 820, and a program stored in the memory 820 and executable on the processor 800; wherein, when the processor 800 executes the program, it implements the AI ​​model supervision method applied to a first device as described above, or implements the AI ​​model supervision method applied to a second device as described above.

[0351] The transceiver 810 is used to receive and send data under the control of the processor 800.

[0352] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 800) and memory (memory 820). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 810 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium.

[0353] The processor 800 is responsible for managing the bus architecture and general processing, while the memory 820 can store the data used by the processor 800 during operation.

[0354] It should be noted that the AI ​​model supervision device provided in this application embodiment can implement all the method steps implemented in the above AI model supervision method embodiment and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiment and the beneficial effects will not be described again here.

[0355] Embodiments of this application also provide a readable storage medium storing a program. When executed by a processor, this program implements the various processes described in the embodiment of the AI ​​model supervision method applied to the first device, or implements the various processes described in the embodiment of the AI ​​model supervision method applied to the second device, achieving the same technical effect. To avoid repetition, further details are omitted here. The readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0356] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions for executing the methods described in the various embodiments of this application.

[0357] Therefore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps described above in the AI ​​model supervision method applied to a first device, or implement the steps described above in the AI ​​model supervision method applied to a second device, and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0358] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0359] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An artificial intelligence (AI) model supervision method, comprising: The method applied to a first device comprises: obtaining first information used for assisting supervision of a first AI model deployed on the first device; based on the first information, performing a model supervision decision.

2. The method of claim 1, wherein, The first information is obtained based on a second AI model, and the second AI model has the same function as the first AI model.

3. The method of claim 1, wherein, Obtaining first information comprises: receiving the first information sent by a second device, wherein the first information comprises at least one of the following: first position information obtained by the second device based on a second AI model or a non-AI positioning method, wherein the second AI model is deployed on the second device; first indication information indicating whether the first position information is obtained based on inference of the second AI model; second indication information indicating a reliability degree of the first position information and / or the second AI model; information related to a reference signal (RS) for determining a unique RS; supervision index information; supervision decision; time information; supervision failure information.

4. The method according to claim 1 or 3, characterized in that, Before obtaining the first information, the method further comprises at least one of the following: sending a first request to the second device, wherein the first request is used for requesting supervision of the first AI model; sending third information to the second device, wherein the third information is used for assisting the second device in performing a positioning operation and / or assisting the second device in supervising the first AI model; receiving a second request sent by the second device, wherein the second request is used for triggering supervision of the first AI model.

5. The method of claim 4, wherein, The first request and the second request respectively comprise at least one of the following: a positioning method request for requesting a positioning method required to be performed by the first device and / or a positioning method required to be performed by the second device; a supervision duration; a supervision requirement; third indication information indicating whether the second device needs to output a positioning result of a second AI model deployed on the second device; a supervision cycle; a supervision index parameter; a supervision assistance parameter, wherein the supervision assistance parameter is requested by the second device to be reported by the first device, and is a parameter used for assisting the second device in model supervision.

6. The method of claim 4, wherein, The method further comprises at least one of the following: after sending the first request to the second device, receiving a first response corresponding to the first request; after receiving the second request sent by the second device, sending a second response corresponding to the second request to the second device; The first response and the second response respectively comprise at least one of the following: a positioning method indication; a supervision duration response; a supervision requirement response; a first indication response indicating that the second device determines to output a positioning result of a second AI model deployed on the second device; a second indication response indicating that the second device determines not to output the positioning result of the second AI model; a supervision cycle response; a supervision index parameter response; a supervision assistance parameter response, wherein the supervision assistance parameter response is determined by the first device to be reported to the second device, and is a parameter used for assisting the second device in model supervision.

7. The method of claim 4, wherein, The third information comprises at least one of the following: Fourth information, the fourth information is related to inference result of the first AI model; Positioning intermediate information obtained based on a non-AI positioning manner; Second position information, the second position information is obtained by inference of the first AI model; Fifth information, the fifth information is related to channel estimation corresponding to RS received by the first device; Time information; Information related to RS, used for determining unique reference signal.

8. The method of claim 1, wherein, Based on the first information, performing model supervision decision, including: Based on the first information, obtaining supervision index information; According to the supervision index information, performing model supervision decision.

9. The method of claim 8, wherein, Based on the first information, obtaining supervision index information, including: According to the related information of RS received by the first device and / or time information, matching the obtained first information with the position information obtained by inference of the first AI model, to obtain target first information matched with the position information obtained by inference of the first AI model; Comparing the position information obtained according to the target first information with the position information obtained by inference of the first AI model, to obtain the supervision index information.

10. The method of claim 1, wherein, Based on the first information, performing model supervision decision, including: According to the supervision index information obtained from the first information and sixth information, performing model supervision decision, the sixth information is information obtained by using preconfigured model supervision manner.

11. An AI model supervision method, characterized by, Applied to a second device, the method includes: Sending first information, the first information is used for assisting supervision of a first AI model, the first AI model is deployed on a first device.

12. The method of claim 11, wherein, The first information is obtained based on a second AI model, the function of the second AI model is the same as that of the first AI model.

13. The method of claim 11, wherein, The first information includes at least one of the following: First position information, the first position information is obtained by the second device based on a second AI model or a non-AI positioning manner, the second AI model is deployed on the second device; First indication information, used for indicating whether the first position information is obtained by inference of the second AI model; Second indication information, used for indicating reliability of the first position information and / or the second AI model; Information related to reference signal RS, used for determining unique RS; Supervision index information; Supervision decision; Time information, used for determining two position information for calculating supervision index information; Supervision failure information.

14. The method of claim 13, wherein, When the second indication information is used for indicating reliability of the second AI model, the second indication information includes at least one of the following: Loss value of the second AI model; Accuracy of supervision of the second AI model; Confidence of supervision of the second AI model; Reliability of supervision of the second AI model.

15. The method of claim 11, wherein, Before sending the first information, the method further includes at least one of the following: Receiving a first request sent by the first device, the first request is used for requesting to supervise the first AI model; Receiving third information sent by the first device, the third information is used for assisting the second device to position the first device and / or assisting the second device to supervise the first AI model; sending a second request, the second request being used to trigger supervision on the first AI model.

16. The method of claim 15, wherein, The first request and the second request respectively include at least one of the following: a positioning method request, used to request a positioning method required to be performed by the first device and / or a positioning method required to be performed by the second device; a supervision duration; a supervision requirement; third indication information, used to indicate whether the second device needs to output a positioning result of a second AI model deployed on the second device; a supervision cycle; a supervision index parameter; a supervision auxiliary parameter, which is requested by the second device to be reported by the first device, and is used to assist the second device in model supervision.

17. The method of claim 15, wherein, The method further includes at least one of the following: after receiving the first request sent by the first device, sending a first response corresponding to the first request to the first device; after sending the second request, receiving a second response corresponding to the second request sent by the first device; The first response and the second response respectively include at least one of the following: a positioning method indication; a supervision duration response; a supervision requirement response; a first indication response, used to indicate that the second device determines to output a positioning result of a second AI model deployed on the second device; a second indication response, used to indicate that the second device determines not to output the positioning result of the second AI model; a supervision cycle response; a supervision index parameter response; a supervision auxiliary parameter response, which is determined by the first device to be reported to the second device, and is used to assist the second device in model supervision.

18. The method of claim 15, wherein, The third information includes at least one of the following: fourth information, the fourth information being related to an inference result of the first AI model; positioning intermediate information obtained based on a non-AI positioning method; second position information, the second position information being obtained by inference of the first AI model; fifth information, the fifth information being related to channel estimation, the channel estimation corresponding to RS received by the first device; time information; information related to RS, used to determine a unique reference signal.

19. The method of claim 15 or 18, wherein, After receiving the third information sent by the first device, the method further includes at least one of the following: performing positioning operation based on the third information and a second AI model deployed on the second device; performing non-AI positioning operation based on the third information; According to the related information of the RS configured for the first device and / or the time information, match at least two of the received third information, the position information obtained by AI positioning, and the position information obtained by non-AI positioning, and calculate supervision index information based on the two matched position information.

20. An AI model supervising apparatus comprising: Applied to a first device, the apparatus includes: an acquisition module, configured to acquire first information, the first information being used to assist supervision on a first AI model, the first AI model being deployed on the first device; an execution module, configured to perform model supervision decision based on the first information.

21. An AI model supervising apparatus comprising: Applied to a second device, the apparatus includes: A first sending module is configured to send first information, wherein the first information is used for assisting supervision of a first AI model, and the first AI model is deployed on a first device. 22.An AI model supervising device comprising a transceiver, a memory, a processor, and a computer program stored on the memory and running on the processor, wherein, The processor executes the computer program to implement the AI model supervision method in any one of claims 1 to 10, or implement the AI model supervision method in any one of claims 11 to 19.

23. A readable storage medium, having stored thereon a program or instructions, characterized in that, The program or the instruction is executed by the processor to implement the AI model supervision method in any one of claims 1 to 10, or implement the AI model supervision method in any one of claims 11 to 19.

24. A computer program product, characterised in that, The program or the instruction is executed by the processor to implement the AI model supervision method in any one of claims 1 to 10, or implement the AI model supervision method in any one of claims 11 to 19. The program or the instruction is executed by the processor to implement the AI model supervision method in any one of claims 1 to 10, or implement the AI model supervision method in any one of claims 11 to 19.

Images